Building applications that run map/reduce jobs shouldn’t be rocket science, says the founder of a startup company that has developed an application development framework for Apache Hadoop.
Hadoop is a Java-based framework that brings methods to large volumes of data on disk for map/reduce data analysis jobs. It is used by Internet search providers, social networking sites such as Facebook and Twitter, financial service providers, and in Amazon’s Elastic Compute Cloud to process large data sets.
Karmasphere, a company founded by alumni of Aventail and Raytheon Systems, today released three new products based on its own framework. The products are Karmasphere Studio Analyst, Karmasphere Studio Professional Edition and Karmasphere Client.
The client sits at the lowest level of the Karmasphere framework, and handles provider interfaces and clusters created with the bigdata open-source computing fabric solution. Above that layer, infrastructure services manage clusters and file systems as well as debugging applications and reporting.
The framework also provides language support for Cascading, Hive QL, Java and Pig, in addition to an integrated graphical user environment.
Hadoop, an open-source map/reduce framework, was originally used by computer scientists and other “rocket scientist” developers, said Karmasphere founder and CEO Martin Hall. “Hadoop is great but not easy to use.”
Hadoop requires developers to understand its file system, and there is no API compatibility between versions and distributions, Hall explained. Business users require an abstraction from that complexity, and analysts want something that looks like SQL, he said.
On the other hand, developers want to be able to see what is going on inside of a cluster, he said.