>Eclipse Development Process: Technical Practice – Continuous Integration

>The Eclipse Development Process encourages projects to adopt an agile development process. It recommends that projects include regular milestones (about 6 weeks), between each milestone. However, it does not say what the technical practice for achieving this should be.

The Eclipse Project Development Process (EPDP) developed by the main eclipse project, evolved over time to take on a very OpenRUP like process. The EPDP like OpenRUP specifies that Continuous Integration is a technical practice for this process. However, what is actually required when we are talking about continuous integration? How is it supposed to be implemented? The EPDP only goes part way with continuous integration, and thus runs into problems because all of the practices are not being followed.

Before I get into what these practices are, here are some of the Anti-Patterns on Continuous Integration:

  • Broken builds – builds that are consistently broken indicate a problem with some technical practices by the team.
  • Minimal build feedback – if the team is not getting just enough feeback on build state and test state, this is a problem.
  • Spam build feedback – too much feedback (i.e. notifications that the build is consistently broken) is also a problem. People will start ignorning the build notifications.
  • Bloated builds – builds that take a long time reduce the amount of feedback to the team about problems. If you have long running builds you have a problem with how Continuous Integration is implemented.

These Anti-Patterns come from a good set of articles here, here, and here. I highly recommend reading them. Following the technical practices outlined by Martin Fowler’s Continuous Integration article will improve your overall build process and help improve the overall quality of your eclipse project. Doing builds alone will not help, but doing them wrong actually causes more developer time wasted than it may seem to save. More time means more features and more bugs fixed.

Continuous Integration Key Practices:

The continuous integration key practices are the required steps to avoid the anti-patterns identified above. Skipping these will eventually start causing the team to fall into the anti-patterns which starts to affect the overall quality, reliablility, and speed in which the developers can deliver the code. Following these practices is something that the team as a whole has to enforce. If the development team does not make a clean build a priority, then why are you continuously building? As specified by the article, the key practices are:

  • Maintain a Single Source Repository – this means that their should be one master source repository that is built from. If you are using GIT or a distributed version control system, there still should be a master repository that all changes are eventually migrated too.
  • Automate the Build – builds should be automated. They need to run with out human intervention. The should also be able to run on any system with out modification. Anybody should be able to run a build locally on their own machine. No machine specific dependencies.
  • Make Your Builds Self-Testing – unit tests, functional tests, and acceptance tests need to be incorporated into the build. Including build reports.

So far the EPDP which many eclipse projects follows, seems to be doing alright in regards to the key practices. However, at this point it devates and starts following the anti-patterns outlined above. Adding the following key practices can help eliminate them.

  • Everyone Committs Every Day – it’s important for every developer on the team to synch with the main repository at least once a day. Ideally more frequently. According to Martin Fowler, “By doing this frequently, developers quickly find out if there’s a conflict between two developers. The key to fixing problems quickly is finding them quickly. With developers committing every few hours a conflict can be detected within a few hours of it occurring, at that point not much has happened and it’s easy to resolve. Conflicts that stay undetected for weeks can be very hard to resolve.”
  • Every Committ Should Build the Mainline on an Integration Machine – MAP file only builds are not sufficient for continuous integration. They are a valid overall build step, but code should also be built from the head as often as possible.
  • Keep the Builds Fast – A common reason that is used for not building from Head, is that the builds take too long. The longer the build takes the less feedback to the developers and the long potential integration problems go un-addressed. Extreme Programming recommends a build should take no longer than 10 minutes. Martin Fowler suggests a Staged Build, “The idea behind a staged build (also known as build pipeline) is that there are in fact multiple builds done in sequence. The commit to the mainline triggers the first build – what I call the commit build. The commit build is the build that’s needed when someone commits to the mainline. The commit build is the one that has to be done quickly, as a result it will take a number of shortcuts that will reduce the ability to detect bugs. The trick is to balance the needs of bug finding and speed so that a good commit build is stable enough for other people to work on.” Maybe not every unit test needs to be run. Maybe building the particular components and bring in dependencies is enough. Other longer running builds can be run, i.e. MAP file, or the current integration builds. The point is to provide feedback early and often to catch problems while they are easy to find and fix.
  • Test in a Clone of the Production Environment – Ideally, you have a build that runs on all of the platforms that you will eventually deploy too. If you are deploying to Windows, Linux, and Mac. You should have builds that run on all of those environments.
  • Make it Easy for Anyone to Get the Latest Executable – This is where the Eclipse Development Process causes an anti-pattern. Section 6.4 is the culprit. It specifically makes it difficult to find the latest builds. It says they should not be visible to the the general public. However this makes it much more difficult for adopters and potential early users to help find bugs. Nightly, and I-Builds should be made available to everyone.
  • Everyone can See what is Happening – Builds Status should be viewable by anyone. Continuous Integration is all about communication. Communication to the developers. Communication to the adopters. Communication to the users.
  • Automate Deployment – Deployement to the end users machine should be as simple as possible. If you are building a product to be distrubuted, then the installation process should be simple. This may mean creating the necessary installer scripts.

Eclipse projects in general have too many anti-patterns in their builds processes. Many are simple to address and the developers can handle it themselves. Others will require the designated release engineer to help address. Builds should happen from head. They should be fast, and developers need to make build failures a higher priority than they are. Not addressing these issues is harming you more than you may realize.

This entry was posted in build, eclipse, testing, xml. Bookmark the permalink.

4 Responses to >Eclipse Development Process: Technical Practice – Continuous Integration

  1. >Dave, The Eclipse Foundation development processes don’t get to this level of detail. On purpose. We don’t want to tell projects how to run themselves. What we call the “Eclipse Development Process” is really more of a project lifecycle.So I think that what you are saying might well be best practices. But I can’t (at this time at least) imagine the Eclipse Foundation mandate that projects must follow them.Perhaps a better approach would be to see if the Architecture Council would like to start documenting some recommended practices such as these?

  2. David Carver says:

    >Mike, the purpose of these blog postings is to not change the Eclipse Development Process. It’s to show that other processes besides the Eclipse Project Development Process (notice the word Project in there). The technical practice article before this was devoted to the XP practice of Unit Testing, this is continuing the XP methodology requirement of Continuous Integration. It’s part of series of posts:Tweaking the Eclipse Development ProcessEclipse Development Process: Addressing BugsEclipse Development Process: Technical Practice – Unit TestingThe EDP should be an overall guide for moving a project through the stages it has to go. However, the Eclipe Projects development process isn’t the only one that can be used within the Eclipse Development Process. These entries are a way to point out some of the problems, and how some other technical practices and management methodologies can help. While still working within the current overall Eclipse Development Process.I still say the EDP is wrongly titled. It should be a Eclipse Release Process as it manages the release of the projects, not necessarily their development.

  3. Wayne says:

    >As we have discussed, David, I’d like to see this series of articles be used as a foundation on which the Architecture Council can build a template for consideration by projects looking to formalize a development process of their own. I believe that this has already been a topic of discussion at previous Architecture Council meetings.

  4. >Dave, Thanks for the explanation.I certainly won’t disagree that the Eclipse Development Process isn’t necessarily the best name. Unfortunately, that label is enshrined in our governance documents (e.g. Bylaws, etc.) which make it a major deal to rename.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s