It’s the best software platform named for a toy elephant, and it’s hard to imagine designing a modern data-intensive application without seriously considering Apache’s Hadoop ecosystem as a way to distribute the workload.
Hadoop has the same ability to revolutionize data handling that the Apache HTTP server did for websites nearly two decades ago, or what Tomcat did for Web applications, or what Subversion and Git are doing for collaborative software development.
Don’t take my word for it. Let me quote from a new study from Transparency Market Research:
The demand for Hadoop is increasing globally due to its capability to access data faster and at cheaper cost as compared to RDBMS. Moreover, the exponential increase in the amount of data generated across different application sectors such as retail, BFSI [banking, financial services and insurance], government and healthcare among others is also fueling the growth of Hadoop solutions.
Certainly you hear all about Hadoop, and its many affiliated projects, in every Big Data-related conversation, and of course in sessions at events like Big Data TechCon, coming to San Francisco on Oct. 15-17. Indeed, Doug Cutting, founder of the Hadoop project, is a keynote speaker at Big Data TechCon.
It is inconceivable to begin any new serious Big Data initiative, or any heavy-duty data application that requires scalability across a network or cluster, that doesn’t somehow leverage Hadoop.
Yes, Map/Reduce, the paradigm behind Hadoop, is not ideal for every task. And yes, there are valid criticisms of Hadoop’s implementation of Map/Reduce, particularly for structured transactional data living in a conventional RDBMS. Still, Hadoop does a great job most of the time, especially in a greenfield environment.
While you consider Hadoop for your business applications, you might also consider Hadoop for yourself. No, I’m not talking about launching a huge metadata mining initiative for your personal music library (though that would be fun, wouldn’t it?). No, I mean as a career move. If you are looking to add to your own skills, let me suggest learning to develop for Hadoop. Get it. Install it. Master it. Maybe even get certified by a company like Cloudera.
The Transparency study suggests opportunities for those with Hadoop skills; if you like data, this may be a ticket for a wonderful professional future:
Currently, a major challenge affecting the market growth is unavailability of experienced and qualified work professionals who could handle Hadoop efficiently. Additionally, lower adoption of Hadoop architecture is also restricting the market, primarily in developing regions such as Asia Pacific and RoW [rest of world].
Hadoop. It’s big. It’s getting bigger. Don’t ignore the toy elephant.
Alan Zeichick, founding editor of SD Times, is principal analyst of Camden Associates.