IBM is looking to accelerate innovation for the Spark ecosystem with its latest commitments to the open-source project. According to the company, the Big Data processing engine is potentially the most significant open-source project of the decade.
“Spark is undoubtedly a force to be reckoned with in the Big Data ecosystem,” said Beth Smith, general manager of the analytics platform at IBM. “Collaborating with Databricks is the next logical step toward delivering next-generation applications for our customers.”
(Related: IBM opens Bluemix to developers for NASA International Space Apps Challenge)
The company plans to incorporate Spark into its core platforms and devote resources and developers to help development efforts. Spark will be built into IBM’s analytics and commerce platform, utilized in the company’s Watson Health Cloud and offered as a Cloud service on IBM Bluemix.
IBM also announced it has open-sourced its machine-learning technology IBM SystemML to help Databricks advance Spark’s machine learning capabilities.
“IBM has been a decades-long leader in open-source innovation,” said Smith. “We believe strongly in the power of open source as the basis to build value for clients, and are fully committed to Spark as a foundational technology platform for accelerating innovation and driving analytics across every business in a fundamental way. Our clients will benefit as we help them embrace Spark to advance their own data strategies to drive business transformation and competitive differentiation.”
In addition, IBM will devote more than 3,500 IBM researchers and developers to work on Spark-related projects; open a Spark Technology Center for the data science and developer community; and educate more than 1 million data scientists and engineers on Spark.
“The size and scale of companies that are partnering with Databricks to support the Spark movement is both inspiring and validating,” said Ion Stoica, CEO of Databricks. “We are looking forward to IBM becoming a key member of the Spark community, as seen by their investment in a Spark Technology Center in San Francisco. This collaboration will help Spark continue to gain mainstream adoption and deliver next-generation Big Data analytics and applications.”
