JeremyEpsteinSoc2006: ImportExportApiModule

Drupal SoC 2006 Application: Generic import / export module

Update (25 May 2006): this proposal has been accepted, so this is what I will be working on!

I propose to develop a generic import / export module for Drupal. The core of this module will be an API that allows data definitions (consisting of database fields and field relationships) to be defined, as well as an engine for importing or exporting data according to these definitions. The module will support XML and CSV as its formats. I'm planning for the definitions to be defined as nested associative arrays in PHP, which will be consistent with the way that form fields are defined in the Drupal 4.7 Forms API (this will also possibly be a third import/export format). The module itself will include definitions for a number of data structures in Drupal core, such as users, taxonomy terms / vocabularies, comments, and a number of the core node types. Additionally, it will support importing and exporting CCK node types.

A number of modules already exist in Drupal contrib to help with importing and exporting data, and I have spent some time researching each of them. These modules include: book import; export DXML; import-export; node_import; taxonomy_xml; and user import. Each of these modules has been written to work with various different data structures in Drupal, and each one has different internals, and supports different formats and operations. The variety of modules in this list is itself an indication that people need many different things to be imported and exported in Drupal, and that there is a strong need for an API that is generic enough to support all such needs in a standard fashion. I plan to draw heavily on the code from these existing modules, so as to re-use code rather than re-inventing the wheel wherever possible.

Proposed roadmap:

MILESTONE: May 23rd, start project

Study the existing import and export solutions in Drupal in greater detail. Find as much inspiration for design approach as possible.
Research existing methods of inputting and outputting XML and CSV data in Drupal. Look for standard tools that may exist in this area.
Find out if other Drupal developers have additional features that they'd like to see in this module / API.
MILESTONE: Jun 11th, written report on research findings
Design data definition API, and method for adding new definitions (hooks and/or .inc files).
Develop data definition API (the core of the module).
MILESTONE: Jun 20th, beta-quality code ready to be delivered
Develop XML and CSV input and output engines (on top of existing code if possible).
MILESTONE: Jun 20th, beta-quality code ready to be delivered
Develop the user interface for performing imports and exports.
MILESTONE: Jun 20th, beta-quality code ready to be delivered
Write data definitions for common data structures in Drupal core.
MILESTONE: Jun 20th, beta-quality code ready to be delivered
VACATION: Jul 16th-23rd.
Develop support for CCK node types.
Develop support for importing and exporting files (such that their relationship to the data fields is preserved).
MILESTONE: Jun 20th, beta-quality code ready to be delivered
Testing, refinement, documentation.

MILESTONE: Aug 20th, project complete, ready for submission

Why I am well suited to complete this project

One of the biggest challenges will be maintaining relationships between data entities during export and import. Existing import / export modules have had to go to great lengths to cater to book nodes, and preserving parent-child relationships. As the author of the category module <http://drupal.org/node/39683>, I feel that I am particularly aware of the issues involved with this.
I'm an experienced Drupal module developer and core contributor, knowledgeable in Drupal design patterns, coding conventions, and the latest trends and developments in the Drupal world (e.g. CCK, FAPI).

Other info

Name: Jeremy Epstein
Email: omitted
IM: omitted

How much time do I expect to have for this project: about 20 hours on any given week (excluding weeks when I will be away). Note: this is updated from my original estimate of '10 or more hours per week'.
Jobs, summer classes, and/or vacations that I'll need to work around:

Jobs: I'm doing a work placement with Optus∞ (Australian phone company) until 7th Jul, which involves working 9-5, Mon-Fri. However, there's no homework, so my weekends and evenings are pretty free during this time.
Classes: I live in Australia, and it's not summer here, so my break is quite short! I start classes on 3rd Aug, and will have about 10 hours of classes per week (taking up 2 days of the week), which I consider to be a fairly light schedule.
Vacations: Going on a ski trip, 16th-23rd Jul. Will also possibly be away for the week before this (TBC).

My technical experience

PHP: Very experienced, through my being a Drupal contributor over the past year.
XHTML/CSS: Very experienced, through approx. 7 years of web designing, starting as an amateur and gradually moving into professional work.
SQL: Very experienced, through writing MySQL and PostgreSQL compatible queries for some of my Drupal contributions. Also experienced in using Sybase, and in using Transact-SQL to write stored procedures and triggers for this database.
JavaScript: moderately experienced, through scripts that I've written for my own site, and through contributions to Drupal.
UNIX: some experience in writing bash scripts, familiar with tools such as grep and awk.
CVS: moderately experienced, through my contributions to Drupal. Also familiar with SVN, as I've used this at university and with clients.
C/C : moderately experienced, through lessons at university, and through writing some simple command-line programs of my own.
Java: moderately experienced, through lessons at university, and through maintenance of Java applications in my work placement.
Visual Basic (v6): moderately experienced, through projects at high school, and through maintaining VB6 apps in my work placement.

My usage experience/familiarity with Drupal: very high. I am an active member of the Drupal developer community, as well as a part-time Drupal consultant.
My development experience with Drupal: extensive. I have written modules for Drupal from scratch (such as the category module), I have produced custom themes, and I have written patches for Drupal core.

Did I participate in a 2005 Summer of Code project: No. I had some very big consulting jobs that I was involved in last year, and I decided that I didn't have adequate time to participate in SoC - but if I'd had the time, I would have applied!
Have I applied (or do I plan to apply) for any other 2006 Summer of Code projects: Yes - I plan to apply for other Drupal projects (still deciding which ones to apply for), from the list being maintained on drupal.org.
School that I attend: University of Technology Sydney∞.
Years of attendance: 3 (including this year).
My specialty/major at the school: Information Technology (no major, since my course is a short 'fast-track' course - but I am considering doing honours).
My resume: http://basement.greenash.net.au/resume/resume2006.pdf∞
My blog: http://www.greenash.net.au/∞

Drupal SoC 2006 Application: Generic import / export module

Deletions:

Drupal SoC 2006 Application: Generic import / export module

Edited on 2007-04-19 23:03:02 by BmoN01

Additions:
- C/C : moderately experienced, through lessons at university, and through writing some simple command-line programs of my own.

Deletions:
- C/C: moderately experienced, through lessons at university, and through writing some simple command-line programs of my own.

Edited on 2006-05-24 19:23:18 by JeremyEpstein [added note that this app has been accepted]

Additions:
Update (25 May 2006): this proposal has been accepted, so this is what I will be working on!

Edited on 2006-05-24 19:22:13 by JeremyEpstein [fixed up line breaks]

Additions:
1) Write data definitions for common data structures in Drupal core.
MILESTONE: Jun 20th, beta-quality code ready to be delivered
VACATION: Jul 16th-23rd.

Deletions:
1) Write data definitions for common data structures in Drupal core.--MILESTONE: Jun 20th, beta-quality code ready to be delivered--VACATION: Jul 16th-23rd.

Edited on 2006-05-07 01:19:00 by JeremyEpstein [Copied in and formatted SoC proposal.]

Additions:

Drupal SoC 2006 Application: Generic import / export module

I propose to develop a generic import / export module for Drupal. The core of this module will be an API that allows data definitions (consisting of database fields and field relationships) to be defined, as well as an engine for importing or exporting data according to these definitions. The module will support XML and CSV as its formats. I'm planning for the definitions to be defined as nested associative arrays in PHP, which will be consistent with the way that form fields are defined in the Drupal 4.7 Forms API (this will also possibly be a third import/export format). The module itself will include definitions for a number of data structures in Drupal core, such as users, taxonomy terms / vocabularies, comments, and a number of the core node types. Additionally, it will support importing and exporting CCK node types.

A number of modules already exist in Drupal contrib to help with importing and exporting data, and I have spent some time researching each of them. These modules include: book import; export DXML; import-export; node_import; taxonomy_xml; and user import. Each of these modules has been written to work with various different data structures in Drupal, and each one has different internals, and supports different formats and operations. The variety of modules in this list is itself an indication that people need many different things to be imported and exported in Drupal, and that there is a strong need for an API that is generic enough to support all such needs in a standard fashion. I plan to draw heavily on the code from these existing modules, so as to re-use code rather than re-inventing the wheel wherever possible.

Proposed roadmap:

MILESTONE: May 23rd, start project

Study the existing import and export solutions in Drupal in greater detail. Find as much inspiration for design approach as possible.
Research existing methods of inputting and outputting XML and CSV data in Drupal. Look for standard tools that may exist in this area.
Find out if other Drupal developers have additional features that they'd like to see in this module / API.
MILESTONE: Jun 11th, written report on research findings
Design data definition API, and method for adding new definitions (hooks and/or .inc files).
Develop data definition API (the core of the module).
MILESTONE: Jun 20th, beta-quality code ready to be delivered
Develop XML and CSV input and output engines (on top of existing code if possible).
MILESTONE: Jun 20th, beta-quality code ready to be delivered
Develop the user interface for performing imports and exports.
MILESTONE: Jun 20th, beta-quality code ready to be delivered
Write data definitions for common data structures in Drupal core.--MILESTONE: Jun 20th, beta-quality code ready to be delivered--VACATION: Jul 16th-23rd.
Develop support for CCK node types.
Develop support for importing and exporting files (such that their relationship to the data fields is preserved).
MILESTONE: Jun 20th, beta-quality code ready to be delivered
Testing, refinement, documentation.

MILESTONE: Aug 20th, project complete, ready for submission

Why I am well suited to complete this project

One of the biggest challenges will be maintaining relationships between data entities during export and import. Existing import / export modules have had to go to great lengths to cater to book nodes, and preserving parent-child relationships. As the author of the category module <http://drupal.org/node/39683>, I feel that I am particularly aware of the issues involved with this.
I'm an experienced Drupal module developer and core contributor, knowledgeable in Drupal design patterns, coding conventions, and the latest trends and developments in the Drupal world (e.g. CCK, FAPI).

Other info

Jobs: I'm doing a work placement with Optus∞ (Australian phone company) until 7th Jul, which involves working 9-5, Mon-Fri. However, there's no homework, so my weekends and evenings are pretty free during this time.
Classes: I live in Australia, and it's not summer here, so my break is quite short! I start classes on 3rd Aug, and will have about 10 hours of classes per week (taking up 2 days of the week), which I consider to be a fairly light schedule.
Vacations: Going on a ski trip, 16th-23rd Jul. Will also possibly be away for the week before this (TBC).

My technical experience

PHP: Very experienced, through my being a Drupal contributor over the past year.
XHTML/CSS: Very experienced, through approx. 7 years of web designing, starting as an amateur and gradually moving into professional work.
SQL: Very experienced, through writing MySQL and PostgreSQL compatible queries for some of my Drupal contributions. Also experienced in using Sybase, and in using Transact-SQL to write stored procedures and triggers for this database.
JavaScript: moderately experienced, through scripts that I've written for my own site, and through contributions to Drupal.
UNIX: some experience in writing bash scripts, familiar with tools such as grep and awk.
CVS: moderately experienced, through my contributions to Drupal. Also familiar with SVN, as I've used this at university and with clients.
C/C: moderately experienced, through lessons at university, and through writing some simple command-line programs of my own.
Java: moderately experienced, through lessons at university, and through maintenance of Java applications in my work placement.
Visual Basic (v6): moderately experienced, through projects at high school, and through maintaining VB6 apps in my work placement.

Oldest known version of this page was edited on 2006-05-06 11:13:30 by JeremyEpstein [Initial stub.]

Page view:

This page will have info about the import / export API module proposal that I've submitted as an application to the Google SoC program.