As Apache Hadoop 2.0 was released, Doug Cutting, creator of the Apache Hadoop project and chief architect at Cloudera, was preparing to give the keynote at the Big Data Technology Conference in San Francisco yesterday. We caught up to him before he spoke and quizzed him about Hadoop 2.0, Hadoop 3.0, and the state of the Hadoop community.
Four years ago, you said that if Hadoop was the only Big Data processing platform in the market in four years, that it would have won, and would remain the de facto standard for many years to come. Hadoop 2.0 shipped this morning from Apache, and there’s really no sign of a competing product still. Does that mean Hadoop has won?
It looks like nothing else has appeared. It’s really become the de facto standard. I honestly expected Microsoft or Oracle or IBM to come out with something competing.
I think that’s a credit to the open-source methodology. It’s doing something everyone can get on board with. Apache tries to make sure we have projects everyone can support.
(More on Hadoop 2.0: Hadoop 2.0 comes, bringing YARN)
Hadoop 2.0 includes support for batch jobs outside of the Map/Reduce model. Was this something you ever considered while writing Hadoop?
We talked about it fairly early on. Arun [Murthy of Hortonworks] had this proposal to refactor Map/Reduce into a more general platform, and make Map/Reduce an application-level logic on top of that. At the time, it was like, “That’d be nice some day, but we need to get Map/Reduce working well first.” Arun held onto that dream, and eventually when Map/Reduce was stable and out in wide use, he went back and started pushing that agenda again, and now we’re there.
HDFS becomes highly available in Hadoop 2.0, yet for years HDFS has been ridiculed as a less-than-stellar file system, as compared to modern, more functional file systems. Do you think the HDFS updates in Hadoop 2.0, combined with the continued popularity of HDFS, is a vindication of your original designs?
I think it’s a vindication of the original Google design being a great starting point. As the strategy we went with—which was to get something that was working that demonstrated the scalability and utility, and not worry about having all the features from day one—I think people often get distracted by not realizing what is a critical thing. Triage is the term they use in medicine. A lot of projects don’t have enough triage.
Now we’re back, filling in the gaps. We’ve had to do a lot of work on security, and the same with the single point-of-failure problem. Rolling upgrades, snapshots and disaster recovery are all things we’ve been able to add after the fact. I think it’s roughly the order of features Google added. You need to make sure you’ve got the scalability right before any of it matters, and basic usability. If you haven’t proven that, then it doesn’t matter if it is secure or not.
But that’s been your strategy all along: Get it working, not fancy. Has this been one of your reasons for success? Ruthless focus on getting the important stuff done while ignoring some portions of functionality?
I think it helps if your strategy can be a grassroots adoption strategy, which I am very fond of. At the outset, there were no grand claims made [about Hadoop], there were no sales pitches. It was, “Here’s some stuff, try it out. If it does something useful, use it, if it almost does something useful, help us get at that last percentage.” People came to it with reasonable expectations. Now, we have the risk of hyping it.
Cloudera was originally supposed to be a product company, but you’ve been forced by the market to offer lots of services and training. Has this shift been difficult?
I think we never expected training to be as big a part of our business as it has been. We think it’s a real complement to our business. We need to train people in order to get them to be able to use the technology, but it’s not something we do begrudgingly. It’s not our main business. It’s not the business we intend to be our growth engine.
But we’ve got a great training team that has taught 15,000 people so far. It’s something we’re happy to do. It also helps us have great documentation that goes hand-in-hand with people who can really explain the stuff. We also send our new employees through training.