noodling towards a functional brain

Thursday, July 15, 2010

An unfortunate situation.

I've been working a bit with Hadoop lately. As many will know, there are essentially two versions of Hadoop that coexist within the Hadoop API; a deprecated legacy API and a newer one that is not nearly so feature-complete. It's a source of confusion to say the least.

Earlier today, I asked in the #hadoop channel on freenode, why were the breaking changes not simply made on a separate branch, the major version number incremented, and so forth such that everyone could move forward? The answers that I got back illuminated a troublesome trend that I think is worth considering for those projects (like my beloved Scala) that are moving towards professional support models of funding.

During the discussion, I commented that there is an unfortunate reality that the largest companies, who may provide the greatest support any given open-source project are also those who will likely be most conservative with respect to change. So as soon as open-source projects grow large enough that companies selling commercial support for those projects survive, there is an immediate disincentive for the support companies to make breaking changes. And stagnation ensues.

The response was startling. People who had been involved with numerous prominent businesses and projects immediately replied with stories of their own projects suffering such a fate, sometimes to the point that the project died entirely.

I think that this "success effect" can be traced to a single, simple cause, with a single, simple solution. Companies (and the individuals that make them up) need to stop thinking of mission-critical software as something that is ever complete. Marketing is never complete; sales is never done with their job, purchasing and HR are never done, nor is accounting. Software is no different - as engineers, we do not (and should not purport to) produce solutions to the problems that companies have; instead, we automate and support the operation of businesses so that businesses can do more, serve more customers, and in the end create more wealth. A project is finished when nobody is using it; never before then.

1 comment:

  1. Change has cost, and one strategy is to try to avoid it altogether. The agile strategy is to make it as cheap as possible. Hence, the agile motto, "embrace change."

    Unfortunately, making technology changes cheap lies outside the power of most users. If the syntax of Scala suddenly changed next year (and older versions were no longer supported), then companies would have to spend a huge amount of money rewriting their Scala code bases (for only marginal long-term gain) -- this is money that could have been used to produce more value for end-users, so it is pure waste (from a lean perspective).

    Now had the Scala community simultaneously written a tool to automatically upgrade source code (as was actually done at some point in the past), then the cost of change would be minimized.

    However, given the reality that most technology developers do not produce tools that ease upgrading, the choice is generally between "don't change the technology," and "waste lots of money." Large companies push and opt for the former.

    Neither of these options is particularly attractive. But it's easy to see there are two sides to this story and reasons for falling on either side depending on the particulars of the situation.


About Me

My photo
aspiring to elegant simplicity