In open-source parlance, the year 2010 can be summed up in one word: Hadoop. But what are the open-source projects that will be most important in 2011? Will NoSQLs be hanging around, or will a newfangled dynamic language framework like Rails be the hip new thing? We’ve taken out our crystal balls and, with the help of some open-source experts, picked five projects that we feel will be the most important for developers in 2011.
With so many shops now playing with Hadoop, the architectural questions tend to revolve around, “Why do we need two giant data stores; one for live data and one for Hadoop’s after-the-fact number crunching?” And indeed, this is a question asked by StumbleUpon, the popular website-rating browser plug-in. Rather than live with two data stores, StumbleUpon has been contributing caching capabilities to HBase, the database built for Hadoop.
With HBase’s new caching capabilities, entire sites can now be built with Hadoop as the live back end. This saves time and money as all the information remains in one place, without the need to load information off of production servers and onto the Hadoop cluster after the fact. If HBase continues to evolve in this way in 2011, it might become a viable alternative to MySQL for simpler database-backed applications. And while the conventional wisdom has said that Hadoop is ill-suited to hosting live data, HBase users are starting to find that this isn’t necessarily the case.
Eclipse has long been an essential tool for developers. With the arrival of the RCP project for Eclipse, the platform moved from just being an IDE into the new category of application platform.
But the transformations don’t end there. Mike Milinkovich, director of the Eclipse Foundation, said that Project Virgo offers an even further expansion of the Eclipse platform into the realm of the runtime.
“We’ve been talking for a number of years now around the idea of Eclipse getting more into runtimes,” he said. “With Virgo, Gemini and Jetty coming to Eclipse in the last 18 months or so, it has been a watershed moment for Eclipse and runtimes. We’ve now hit critical mass, and you can get a pretty cool runtime server from Eclipse. I can’t think of anything I can talk about specifically, but I am already seeing projects and products looking at Virgo as something to move to.”
And as previous years have shown, never bet against Eclipse expanding into new areas of development usage; developers will always find ways to use more Eclipse.
The flirtation with NoSQLs is over; the databases will actually have to perform in the real world to remain relevant in 2011. One NoSQL that isn’t receiving heaps of scorn in the trough of disillusionment is CouchDB.
Mu Dynamics’ CTO Kowsik Guruswamy was so moved by the use of CouchDB in one of his company’s products that he was driven to blog, tongue-in-cheek, about how the project makes building a startup too easy. “CouchDB written in Erlang is honestly spooky,” he wrote.
“With an uptime of eight months and memory utilization of 0.1%, we think it’s actually survived a few power outages too! The joy of running an online community site is the debugging, thinking and solving locking and concurrency issues, outages, and how best to scale the site as the traffic continues to increase.