IBM shifts data science into high gear today with the announcement of new In-Hadoop analytics technologies to accelerate the conversion of data into valuable insight for the business. IBM is delivering machine learning, R, and many new features that can run over large-scale data in the new IBM BigInsights for Apache Hadoop.

A growing number of organizations today recognize Apache Hadoop as a powerful technology for collecting and economically storing a very large set of highly variable data, and yet they struggle to realize its full potential in every part of their business. For example, a business analyst needs to quickly find relevant information, the data scientist needs to make sense of the data with statistical modeling, and the whole environment needs to be easy for IT to manage and deploy for everyone to use.

As the size and complexity of Hadoop applications continues to grow, data science has become a critical function for unlocking meaningful insights and identifying actions that could optimize outcomes across the entire business. IBM BigInsights for Apache Hadoop includes a broad data science toolset to query data, visualize, explore and conduct distributed machine learning at scale.

“There’s so much data out there that it’s often difficult for companies to find the information that really matters,” said Anand Mahurkar, CEO of Findability Sciences. “IBM BigInsights helps connect data elements that often go undiscovered to bring context to the customer relationship. Some of our clients have improved customer retention by up to 25 percent with this enhanced understanding. They can now focus efforts and resources on where they’ll have the greatest effect, helping win and retain more business and drive more profitable operations.”

IBM BigInsights for Apache Hadoop will introduce three new modules:

  • IBM BigInsights Analyst will include IBM’s SQL engine and IBM’s intuitive spreadsheet and visualizations to find data quickly and easily. On average, millions of SQL queries are run each year. With BigInsights Analyst, the efficiency of these queries has been shown in some cases to improve by approximately 2x to 4x on Apache Hadoop depending on the shuffle size. The ANSI compliant SQL means queries can run unchanged against Hive, HBase and relational databases.
  • IBM BigInsights Data Scientist will deliver a new machine-learning engine that automatically tunes its performance over large-scale data to find interesting patterns– plus over a dozen industry-specific algorithms such as Decision Trees, PageRank and Clustering to help tackle complex problems out of the box. It will also provide native support for open source R statistical computing helping clients leverage their existing R algorithms, or gain from the more than 4,500 freely available statistics packages from the R community.
  • IBM BigInsights Enterprise Management will introduce new management tools for clients to realize faster time to results. Designed to help allocate resources and optimize workflows, these tools will allow deployments that can scale to large numbers of users and clusters, and will help satisfy high workload demand. These tools will provide multi-tenancy and multi-instance support in a cluster.

Also announced today is IBM Open Platform with Apache Hadoop. Based on open source software, the platform will provide the necessary data access controls and authentication for an enterprise. We are also adding support for Apache Spark to enable new computing engines to facilitate interactive analytic applications.

IBM is also sponsoring Big Data University and its new courses for Programming for Analytics and Data Science that cover machine learning. Big Data University addresses today’s lack of big data skills by delivering free online courses to a community of more than 230,000 registered participants around the world.

These Hadoop solutions fit cohesively into IBM’s broader analytics platform to help deliver insight from data, when, where and how it is needed. The  platform provides a full range of analytics and integrates Hadoop solutions as part of a warehousing and data architecture. In particular, the predictive capabilities of SPSS can build the predictive models, exercise machine learning and R in Hadoop, apply business optimization, and deploy these models into real-time business processes.

“In our fast paced world, the ability to turn data into insight is the difference between success and failure,” said Beth Smith GM of IBM Analytics Platform. “With the announcements today, our ability to address the sophisticated needs of data scientists, to improve access for a broader community of analysts, and the surety of production scale will set in motion the next wave of value our clients expect from big data.”

The new capabilities in IBM BigInsights for Apache Hadoop expand what is already the industry’s deepest portfolio of Big Data solutions, spanning software, services, research and hardware. IBM Analytics combines traditional data warehouse technologies with new Big Data techniques, such as Apache Hadoop, stream computing, data exploration, advanced analytics, enterprise integration and IBM Watson cognitive computing to create an integrated solution to embed the opportunity of Big Data & Analytics into every organization.

IBM is also a founding member of the Open Data Platform (ODP) Initiative, a new industry association announced today to help drive collaboration, innovation, and standardization across Hadoop and big data technologies.

For more information on IBM BigInsights for Apache Hadoop see http://www.ibm.com/hadoop.