How open source is changing the face of business

No, what I’m talking about is a totally different mindset – namely, that if your company creates something new and useful, it goes ahead and just gives it away for free, so that anyone, including its closest competitors, can benefit from the work it’s done.

This is exactly what companies such as Google, Facebook, Yahoo, Twitter and many, many others have been doing for several years now. They’ve created amazing pieces of software, invested tens of thousands of developer-hours in doing so, and have then made that software available to anyone who wants it.

The example I want to focus on for the purposes of this discussion is a suite of software that goes under the umbrella name of Hadoop. Regular readers may remember that a couple of months ago in this column I wrote an introduction to Hadoop, which I’ve been experimenting with for some while and was impressed with its capabilities.

I’m now much closer to the product since I’ve started working for a company that exists solely to provide support and training in Hadoop, and that’s given me a far greater insight into the whole open-source development process.

Hadoop’s history

Hadoop was originally based on a technical presentation by some Google engineers who explained how that company goes about indexing and storing the massive amounts of data it has to deal with in order to provide its central web search service.

That paper spurred a small group of developers into emulating the Google MapReduce system and the filesystem it employed for storage – then Yahoo hired the chief architect of the project, Doug Cutting, and within a year Hadoop was being used as a crucial part of Yahoo’s infrastructure.

Now at this point, in the old way of doing business, Google might have sent its lawyers after everyone involved in the project, since after all it holds a number of patents on key aspects of this technology. But it didn’t do that.

In fact, Google made it known that it was okay about the Hadoop project, and it has officially granted a free, perpetual licence to the patents that lie behind the technologies to the Apache Foundation, which now nurtures Hadoop.

Google made it known that it was okay about the Hadoop project, and it has officially granted a free, perpetual licence to the patents

Note carefully what I wrote just there: that the Apache Foundation, a non-profit organisation that manages several other open-source projects, now controls Hadoop. Although Yahoo was paying the people who developed this software, it went out of its way to make that software open source.

From these beginnings Hadoop is now used by a large number of companies that have to deal with truly huge amounts of data: Facebook, Twitter, Netflix, LinkedIn; the list just goes on and on (and includes a lot more traditional companies that won’t allow me to name them in print).

But hold on a minute, Facebook and Twitter? Facebook and Yahoo? Aren’t these companies at least nominally in competition with each other? Indeed they are, and yet they’re all using the same basic software infrastructure and, most crucially, all contributing to its growth and success.

So why on earth would any company do this? Why would it take something it’s invested vast amounts of money in and give it away for free? And why would some other company start using such software developed by its competitor? It turns out there are several very compelling reasons.