>Note: This entry is a little long, and is primarily here to help clarify what I mean about continuous integration. The opinions and views expressed are my own. Some will agree with them, others will not. Hopefully at the minimum it makes one at least think about how builds are done, and if there are ways to improve existing processes.
Agile development practices like Extreme Programming (XP) favor building the code as often as possible. They shun away from code freezes. The purpose of building and testing often is to catch integration errors with other people’s code sooner rather than later. This entry will discuss some aspects on how one might apply these principles to the various projects at eclipse.
Some are already implementing these practices, others are not.
What are the benefits of Continuous Integration?
Wikipedia summarizes as the following:
- When unit tests fail, or a bug is discovered, developers might revert the codebase back to a bug-free state, without wasting time debugging.
- Integration problems are detected and fixed continuously – no last minute hiatus before release dates;
- Early warning of broken/incompatible code;
- Early warning of conflicting changes;
- Immediate unit testing of all changes;
- Constant availability of a “current” build for testing, demo, or release purposes;
- The immediate impact of checking in incomplete or broken code acts as an incentive to developers to learn to work more incrementally with shorter feedback cycles.
What is Continuous Integration?
In order to set the expectation of what I mean by continuous integration it helps to define it. Depending on your view point continuous integration can mean multiple things and can be different at different stages of a build. I happen to like the Wikipedia definition:
Continuous Integration is the practice of integrating early and often, so as to avoid the pitfalls of “integration hell”. The ultimate goal is to reduce timely rework and thus reduce cost and time. When done well, continuous integration has been shown to achieve these goals.
A blog entry titled “Agile 102: Continuous Integration Explained”, has a pretty good summary and a good graphic depicting continuous integration as shown below.
Okay what do you mean by “Integration Hell”?
Basically this is when the amount of code that has to be integrated or the number of changes that have occurred in the source stream causes the developer to curl up in the fetal position, and cry. It takes longer to integrate the changes than it did to originally write the code. The less often code is integrated, the more time it takes and the greater the possibility of integrating wrong.
So what are the practices?
There are several recommended practices.
- 2.1 Maintain a code repository
- 2.2 Automate the build
- 2.3 Make your build self-testing
- 2.4 Everyone commits every day
- 2.5 Every commit (to mainline) should be built
- 2.6 Keep the build fast
- 2.7 Test in a clone of the production environment
- 2.8 Make it easy to get the latest deliverables
- 2.9 Everyone can see the results of the latest build
- 2.10 Automate Deployment
Each of these must be implemented to really be doing continuous integration. Leaving any of these out starts to run into the Anti-Patterns of Continuous Integration.
What about integration with other Eclipse Components, where does that come in?
This is an important aspect. Some may feel that continuous integration means rebuilding the entire system everytime code is released. Opinions differ at times in the agile community about this. I personally view this long running process as a Nightly job. However, from what I’ve seen it appears that most eclipse projects that this is the only type of build they run. If it is the only type of build you run then it isn’t meeting several of the recommended practices. Primarily “Keep the Build Fast” and “Every Commit to Mainline (Head) should be built” are not being met. A complete cross project integration build can take a couple of hours to run. This should occur once or twice a day. Often enough to provide valuable feedback. It could occur as little as once a day, but it shouldn’t be a weekly or biweekly occurrence. Fast builds allow developers to identify issues sooner rather than later.
We can’t build that often!
This is probably because of the way the builds are setup. If they are not designed for this type of testing and are only doing complete top level project to top level project integration then yes you currently can not do it. You have to rework your build process.
There are numerous issues and reason that people come up with for why they can not do it. I’ve heard most of them repeated in some for or another with other projects I’ve been involved with, and even where I work.
I’ve seen continuous integration work really well, and I’ve seen it fail miserably. Where it has failed is because not all the recommended practices were followed by the team. Building from head takes a different approach to development than some are used to. It means working in smaller chunks, writing tests, and at least integrating with the main line at least once a day (i.e. check out the latest code). Before checking in the code, you have to do at least a minimum of testing of the critical systems. Ideally running your whole test suite for your project to make sure you are not broken. It does not necessarily mean running every test you have, but it does mean running the critical ones.
Martin Fowler explains ways to do Continuous Project level integration and then do a whole Product Integration test separately. At work, we have hourly builds from head. However, we have a complete integration build with all projects and components that make up the product, at night. This is the long running build and if it fails, there is an email sent to the team to fix in the morning.
A broken build should be treated with the highest priority. Getting the build working again, should come before everything else. It’s your heartbeat for your project. A red broken build means the project just had a heart-attack.
Okay, so how can you fix it if the code was built from head and head constantly changes?
Currently most eclipse based projects tag all code for a build and then build it using the release manager. Code is then pulled out according to the map files, and a build is done using those map files. An alternative to this approach is to check out from head, and then after building, tag all the files that were checked out for that build. This tags everything for a particular build, and the necessary files that may need to be fixed, can be checked out and fixed. Changes are released back to the mainline. If there really is a major problem, a branch could always be done. Changes worked on in the branch, and then merged back into the system. Projects could still manually tag code to be released if necessary.
To implement this type of tagging, it means that the build process needs to have a login that has at least write access to the repository so that files can be tagged. Currently all eclipse builds I’ve seen pull code using the pserver protocol if they are using CVS. They assume code has been tagged to build, and the build itself does not actually do the tagging. Flip this around, build from head, and let the product integration builds tag the code. You could tag all your builds, but this might be overkill.
Personally, I’ve been on both sides of the fence. I thought that this was a lot of work for very little gain. I was not convinced of agile development techniques, and did not think this could work. In fact, my first experience with an agile project that tried what they called continuous integration, had in fact fallen into the Anti-Patterns of continuous integration. The teams did not follow the practices outlined, so they did not gain the complete benefits. Soon they blamed each other, and they were always missing their target dates.
Since then, I’ve learned that one has to at the minimum integrate often, build often, test often…mix, stir and repeat. So some will say there are not enough time or resources to get everything done. The question really comes down to, how much extra time and resources are being spent fixing builds and missing target dates due to not trying or sticking the practices. How much more time could be spent working on other bug fixes, and features that help improve the state of the code and the overall product for the end user.
Okay so have you done this with an eclipse project?
I’ve got a proof of concept that is at least showing that the continuous integration builds can happen. It’s not a complete build yet, and once I have it fully worked out I’ll post more information. It implements everything in the recommended practices, except running the tests.
I’ve used XSL Tools as the proof of concept project. If you are interested, drop me a note and I can provide more information, otherwise, wait and I’ll have an updated entry when all pieces are in order.