How open source is changing the face of business

I’ve been thinking deeply about the world of open-source software lately, partly because I’ve recently changed jobs and am now working for a “pure play” open-source company – a company that makes its revenue solely by supporting an open-source project.

To be sure, I used to work for Sun, teaching students about MySQL, but that was slightly different because Sun actually sold an enterprise version of MySQL and that’s the direction from which a lot of my students came.

Now, my new employer, Cloudera, sells nothing but support and training: although we do have our own version of the product involved, we give that away and contribute any improvements we make to it back to the open-source community.

What I’ve come to realise is that becoming involved with open source really will represent a complete shift of paradigm for many businesses

Now I realise that you’re not reading this column just to find out about my latest employment status, but you might find some of my recent thoughts on the whole open-source business that have been triggered by my change of job interesting.

That’s because I think that many people are stuck in the same mindset that I used to be in – namely, that open-source software is great because it’s free and convenient… but that’s all you can say about it. What I’ve come to realise is that becoming involved with open source really will represent a complete shift of paradigm for many businesses.

The old way of business

To understand what I’m talking about, you need to think about the way companies used to do business. If you were working for MegaCorp Inc and you came up with a revolutionary piece of software to expedite some part of its production process, which saved your company huge amounts of money – say, by achieving something in 20 minutes that used to take 20 hours – then your company would go to the ends of the earth to keep your new method secret.

Very often it wouldn’t even attempt to patent it, because the mere filing of a patent requires that you explain the novel way you’re doing the job, and that could give away the idea to other people simply by reading your application.

Or if the product did get patented, MegaCorp’s legal department would ferociously pursue anyone who produced anything the company deemed to be even remotely similar (a phenomenon we’ve seen a lot of recently over various user interfaces, including the iPhone).

Of course, that’s a perfectly defensible attitude to take: your revolutionary new software product is giving MegaCorp a competitive edge, and if the people at UltraCorp get wind of how you’re doing things they could replicate it and then you’d be competing on a level playing field again.

This is the way that business has been conducted pretty well ever since the Industrial Revolution – build a better mousetrap and sell the heck out of it until someone else trumps you by building an even better mousetrap. Rinse and repeat, ad infinitum.

Over the past few years, however, the open-source movement has radically changed the way at least some companies do their business. These companies tend to be the newer ones, less mired in the old mindset, although plenty of venerable institutions have changed tack too.

I’m not just talking here about businesses using open-source software products rather than proprietary ones – although of course that’s nice because it saves them money (who knows how many millions of dollars MySQL has saved companies that use it rather than Oracle, Microsoft SQL Server or whatever).
No, what I’m talking about is a totally different mindset – namely, that if your company creates something new and useful, it goes ahead and just gives it away for free, so that anyone, including its closest competitors, can benefit from the work it’s done.

This is exactly what companies such as Google, Facebook, Yahoo, Twitter and many, many others have been doing for several years now. They’ve created amazing pieces of software, invested tens of thousands of developer-hours in doing so, and have then made that software available to anyone who wants it.

The example I want to focus on for the purposes of this discussion is a suite of software that goes under the umbrella name of Hadoop. Regular readers may remember that a couple of months ago in this column I wrote an introduction to Hadoop, which I’ve been experimenting with for some while and was impressed with its capabilities.

I’m now much closer to the product since I’ve started working for a company that exists solely to provide support and training in Hadoop, and that’s given me a far greater insight into the whole open-source development process.

Hadoop’s history

Hadoop was originally based on a technical presentation by some Google engineers who explained how that company goes about indexing and storing the massive amounts of data it has to deal with in order to provide its central web search service.

That paper spurred a small group of developers into emulating the Google MapReduce system and the filesystem it employed for storage – then Yahoo hired the chief architect of the project, Doug Cutting, and within a year Hadoop was being used as a crucial part of Yahoo’s infrastructure.

Now at this point, in the old way of doing business, Google might have sent its lawyers after everyone involved in the project, since after all it holds a number of patents on key aspects of this technology. But it didn’t do that.

In fact, Google made it known that it was okay about the Hadoop project, and it has officially granted a free, perpetual licence to the patents that lie behind the technologies to the Apache Foundation, which now nurtures Hadoop.

Google made it known that it was okay about the Hadoop project, and it has officially granted a free, perpetual licence to the patents

Note carefully what I wrote just there: that the Apache Foundation, a non-profit organisation that manages several other open-source projects, now controls Hadoop. Although Yahoo was paying the people who developed this software, it went out of its way to make that software open source.

From these beginnings Hadoop is now used by a large number of companies that have to deal with truly huge amounts of data: Facebook, Twitter, Netflix, LinkedIn; the list just goes on and on (and includes a lot more traditional companies that won’t allow me to name them in print).

But hold on a minute, Facebook and Twitter? Facebook and Yahoo? Aren’t these companies at least nominally in competition with each other? Indeed they are, and yet they’re all using the same basic software infrastructure and, most crucially, all contributing to its growth and success.

So why on earth would any company do this? Why would it take something it’s invested vast amounts of money in and give it away for free? And why would some other company start using such software developed by its competitor? It turns out there are several very compelling reasons.
The first of these compelling reasons is, quite simply, the fact that there’s some incredibly innovative open-source software out there. New techniques and approaches to problem solving are being developed all the time, and it may take years before a commercial company picks up on some novel idea that’s available for free right now.

Then there’s the openness factor: if you don’t like the way a commercial software product behaves, you have to submit a feature request to its developers and then wait. And wait. And wait, and wait. Sometimes forever.

New techniques and approaches to problem solving are being developed all the time, and it may take years before a commercial company picks up on some novel idea that’s available for free right now

Whereas because you have access to all the source code of an open-source product, you can modify it to your heart’s content, until it works the way you want it to. Just throw a few developers at the problem and you’ll get exactly the feature set you need, when you need it.

Obviously, it’s a little more complicated than that when you’re dealing with a large project whose sheer complexity means that modifying it may not be something you can do in days, or even weeks – but the potential is always there. And of course, if you’re using a closed-source software product from a company that goes bust, is taken over, or ceases active development of that product, you’re out of luck. With open-source software you always keep options.

Other open source benefits

Those are two well-known and widely advocated points in favour of using open-source software, but there are a couple of other benefits that might be helpful if you’re trying to push your old, mired-in-1960s-thinking management toward using a new open-source project.

Consider this, for example: rather than trying to develop something in-house, you can take advantage of the fact that there are many people already working on that project. By becoming involved in it and putting resources behind it you’ll get a powerful voice in the direction that software takes, and this effectively translates into a large, free, extra workforce of developers at your disposal.

Of course, you can’t demand that a project moves in a certain direction, but if you become a significant presence in that project’s community you can certainly influence it heavily in your desired direction.

Put a couple of the project’s key figures on your payroll and you’ll have even more influence. Apple has learned to play this game wonderfully well, with WebKit the core software that underlies the company’s Safari web browser, and which is now used in the browser of virtually every major smartphone out there: WebKit is an open-source project that the company embraced several years ago.

Next – and this is a point that was explicitly articulated by Yahoo’s executives at a recent technical conference I attended – there’s the fact that your future workforce is already trained.

This is almost certainly a large part of Google’s reasoning behind supporting Hadoop by granting the Apache Foundation a licence to the relevant patents, and also behind the company’s support of universities that teach Hadoop and other distributed programming techniques.

Google needs lots of graduates who are already familiar with its core technologies and experienced in writing software that uses them – that way, its costs for training new developers drops dramatically and it’s far easier to spot the candidates with the strongest abilities.
This same reasoning, of course, also works for Yahoo, Facebook and the rest of that list – basically anyone who uses Hadoop. Once again, though, don’t run away with the idea that I’m only talking about a single project here, because everything that I’m saying about Hadoop is just as relevant to a large number of other major open-source projects.

And by the way, if you’re a developer this is a factor you shouldn’t underestimate: contributing to a major open-source project and getting your name known in its community will do wonders for your employability at any number of companies, both new and old.

Contributing to a major open-source project and getting your name known in its community will do wonders for your employability

There are downsides to this approach, too. If you’re working for a smaller organisation you may not have the resources to contribute a huge amount to a project, which means your voice will carry less weight, and so it may go off in a direction you’re less interested in.

Sure, you’re free to do your own thing with the software, but you’re losing the built-in community of developers, and because of the way many open-source projects are “managed”, disputes over overall direction, new features to add and so on can arise. The largest projects, such as those managed by an organisation like the Apache Foundation, are less likely to suffer from this, but even they’re not wholly immune.

Checkbox mentality

Then there’s what I like to call the “checkbox mentality”. It never ceases to amaze me when I come across firms that aren’t allowed to employ any software that doesn’t come with a support contract, yet in the traditional corporate world this is almost the norm: “We have to be able to go to someone for help if there’s a problem,” is their argument, despite the fact that often the only response they get from the vendor is “Yes, that’s a known bug, and we’ll fix it in a future release (when we feel like it)”.

So before you can use any program, you have to tick the box that says “Yes, we’ve purchased support for this”. Again, though, if you’re a developer you should see this as an opportunity. Get involved in the project. Contribute patches. Get your name known. Then start your own business offering precisely that support.

You’d be surprised how many people make a living doing just that, and how many of those people then get courted for very highly paid senior jobs at the kind of interesting, cutting-edge companies that don’t have the old-school mentality.

What I’m saying is that you shouldn’t think of the open-source world as simply a place to pick up free alternatives to existing commercial products: if you start to look at some of the more cutting-edge, forward-thinking projects, both you and your company could benefit hugely – and have a lot of fun at the same time.