Thursday, July 06, 2006

Eclipse Building Blocks

During last week's Eclipse Council meetings, we discussed evolution of the "base" eclipse build system to include additional features and encourage projects not already using it (such as DTP) to jump on board. I agree that DTP should be using as much standard eclipse.org infrastructure and best practices as possible, both since there usually are very good reasons (often based on hard-won experience) for the choices made and to provide a consistent context for those extending eclipse.

One thing I haven't convinced myself about yet is the complexity of builds. At eclipse.org we talk a lot about "transparency" (public discussions and decisions, etc.), and in general I think we do a good job with it. But builds tend to be a different creature. It goes something like this:

  1. Get code
  2. Magic configuration
  3. Compile
  4. More magic
  5. Build done!

In many projects I've worked on, only the release engineering team could do a build. No, I don't mean could == allowed, but could == capable. Any even then, if the stars were not aligned just right and the magic words were not spoken with enough gravity, the build wouldn't finish.

I don't think this is very transparent ("now you see this code, now you don't, here's the build!"), and that worries me for an open source project. Aside from the principle, there are practical reasons why tricky builds are bad: developers, whether committers on the project or not, should be able to get a code snapshot from source control, make minimal configuration settings, and have a build. Now, this might take a while to run, and sometimes you can't help that. But you should at least have the option. If I am changing/experimenting with some core functionality, I'd like to verify how it all hangs together in the build. (Doing so in a dev environment is good too, but there are many errors that can creep in between dev and "binary" environments too. Ever forget to update your build.properties file to include new resources?)

The build system for DTP started out with the main goal of "run for anyone," but with time and pressure, cruft slipped in. Going forward, we'd like to migrate to the eclipse "base" builder, but would also like to re-capture the "run for anyone" goal if possible too.

Any thoughts/experience out there? Anyone want to help DTP with this (I think it would benefit us all....)?

4 Comments:

Blogger Le ScaL said...

Before going for the full automated build with automatic checkout, etc. I recommend you to start simple with only PDE Build (the eclipse base builder supersets pde build) and build the smallest feature you have.
To do so please try the instructions in the PDE Build section of the help system (http://help.eclipse.org/help32/topic/org.eclipse.pde.doc.user/guide/tasks/pde_feature_build.htm)

Once you are done with that, add automatic checkout and additional magic and finally move to the releng base builder (note that what base builder is mostly glue specific to the eclipse platform team with respect to build publishing and testing).

9:48 PM

 
Blogger Kim Moir said...

If you are interested in learning more about pde build and the platform build process these documents are good starting points

http://www.eclipse.org/articles/Article-PDE-Automation/automation.html

http://wiki.eclipse.org/index.php/Platform-releng-faq#Eclipsecon_2006_presentation

For us, the releng team is the only one that runs the builds. The reason is that our builds are very long (two hours for build, four hours for junit tests and 14 hours for performance tests). If everyone ran builds there would be scheduling conflicts with the test machines.
This probably isn't an issue for other projects that produce a smaller number of zips and fewer tests. Several of our teams run their junit tests before submitting their content to an integration build which allows them to validate their submission without running a full build themselves.

If you have questions regarding the eclipse basebuilder, please ask questions on platform-releng-dev@eclipse.org or eclipse.platform.

9:35 AM

 
Blogger John Graham said...

Kim, le scal:

Thanks for the links and advice: DTP will certainly be looking into the base PDE builder going forward, and will use the forums mentioned to post questions.

-- John

10:10 AM

 
Blogger Dave said...

I have to admit that I miss the Linux autoconf experience of being able to download a tarball, unzip it, open a command line and:

./configure && make && \
su -c 'make install'

to reliably produce a build configured for just my machine and installed in the right places.

Although it's unlikely that I'll do this for all of Eclipse unless I'm also an Eclipse distributor, I'd sometimes like to do this for individual plugins. This would make it easy for me to customize my Eclipse experience and would lower the barrier of entry for folks becoming contributors to Eclipse rather than just consumers of Eclipse.

10:17 AM

 

Post a Comment

<< Home