Are Projects Outpacing the ability to Adapt and Respond?

After reading Ed Merk’s, post about trying to be fair and balanced, and the corresponding bug, in which Scott Lewis has a valid concern, I’m wondering if the number of projects that the Foundation has under it’s umberalla is outpacing it’s ability to adapt to the changing environment. To be honest, the Friends of Eclipse program sounds like a good thing; however, as soon as you put a committe in place to determine how and when that money is going to be used, it be comes a contentious issue.

I think in general the issue at hand is more than the distribution of funds from Friends of Eclipse, but a larger issue that has been growing, the Foundation’s ability to adapt to existing project requests and needs, while also having to adapt to the slowly growing economic recovery. The fact of the matter is that as resources get constrained, money gets diverted to other projects. Non profits and membership dues are either canceled or cut back. The less funds there are, the more the Foundation itself has to manage it’s own budget. Staff cuts occur, and positions are not filled after people leave. It’s just a fundamental fact when you have to depend mostly on membership dues to fund development.

Scott’s request for money to pay some students to do releng work is interesting, however, my personal opinion is that releng should be the responsibility of all committers. There is in general a fundamental problem with releng work, nobody really wants to do it. How complicated doing releng really depends on your project, and more particularly the process and tools you use. Sometimes we need to take a step back from our desire to keep writing new features and new functionality, and just look at what we have created. Are we doing things the most efficiently? Are we doing the simpliest thing that can possibly work, or have we over engineered things? Have our processes become a big giant ball of tangled spaghetti?

I work on several smaller projects and sub-projects, and maintain both code and releng scripts for both. My experience is that unless the committers themselves take the time to learn the release process, and to manage it themselves, it will always be this big black whole. Some of the newer technologies for doing eclipse releases really do take the pain out of things. If you are doing a PDE build based system, because that is what you started the project with. Maybe it’s time to take a step back and look at the other items out there? Will a B3, Buckminster, or Tycho build prove to be something easier to maintain, with the limited resources the team has?

The other issue comes down to the fact that many of the current jobs that are running on the Hudson build system at the Foundation are not playing well with other jobs. Jobs that take 6 to 8 hours will lock up Executors for that entire time. Maybe a staged build approach would be better? Break the build into smaller stages, or maybe look at refactoring the build to reduce the time it takes. Is a full build required every time? Do you need to run every unit test every time? Can the test be broken up so that you only run the most critical ones during a Continuous Integration build, and run longer running tests on another slave?  Why do projects have to jump through multiple hoops in order to do a release to downloads.eclipse.org? All projects need to keep in mind that their build is not the only one running on the build servers.

Foundation time and resources do the best they can at responding to projects needs, but my feeling is that maybe the Foundation needs to look at what is the ultimate size and number of projects it can safely and fairly administer. Are the number of projects it has now, putting a bigger burden on the foundation than it’s current resources can support? How can we address this fundamental issue?   I don’t have an answer, but willing to chat about it over a beer at EclipseCon.

Advertisements
This entry was posted in build, eclipse, release engineering. Bookmark the permalink.

5 Responses to Are Projects Outpacing the ability to Adapt and Respond?

  1. Kim Moir says:

    Good points Dave. I think there is really an issue with resources at the foundation. If there are 10 new projects, are there funds to support x% more utilization in build engines and storage space etc?

  2. kingargyle says:

    It’s not only build resources, but man power resources. The Foundation has over 90+ active projects. Recent reductions in IP staff and specified project timelines will take a toll. As the release train comes closer and closer to a reality, those projects that don’t play nice on the system will monopolize the Executors we have. We have 6 executors on the system. Imagine six, eight hour jobs taking all the executors at once. Imagine the backlog and panic. People will start randomly switching jobs to Master, Fastlane, Windows Test, Mac test just to get things running.

  3. Ed Merks says:

    You’re right in general that any time somebody is in a position to make decisions that affect somebody else, there will be a point of contention. So let’s just get the community to vote you say? Elections are never contentious! Hey, wait a minute… I think contention is unavoidable…

    Kim raises important points too. We have to consider whether an organization that comes to host a project but doesn’t even contribute enough resource to cover the basic cost of IP reviews and other burdens to the foundation staff and infrastructure should be allowed to proceed. Of course it’s not simply an issue of the number of project; some projects are higher maintenance than others…

    • Miles Parker says:

      “We have to consider whether an organization that comes to host a project but doesn’t even contribute enough resource to cover the basic cost of IP reviews and other burdens to the foundation staff and infrastructure should be allowed to proceed. Of course it’s not simply an issue of the number of project; some projects are higher maintenance than others…”

      I’m biased here.. but I’d be really bummed if something like that — I guess a kind of “pay to play” rule — ever happened. It seems like it would strike at the heart of what makes Eclipse unique — the ability to have corporate sponsored and independent projects work on equal footing. I think many of the most innovative Eclipse projects and efforts have come from small organizations, universities and (ahem) independent consultants.

  4. rch says:

    Reading this it is clear that the foundation needs a system for tracking resource usage. Bandwidth, disk space, compute time, etc…. pretty much all of the metrics you have in AWS. If such a beast exists already, please post a link to the data (if it isn’t public, it doesn’t exist, and a one-off report doesn’t count).

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s