Datameer, the only end-to-end big data analytics application for the Hadoop ecosystem, today introduced Datameer 5.0 with Smart Execution, a new patent-pending technology that intelligently and dynamically selects the best-of-breed compute framework at each step in the big data analytics process. Datameer is transforming the analytics process with its new Smart Execution engine by examining dataset characteristics, analytics tasks and available system resources to determine the most efficient compute framework for the task at hand.

A new series of computation frameworks complementary to MapReduce have joined the Hadoop ecosystem to address the full range of data analytics needs. This proliferation of tools creates complexity, as most frameworks are optimized for either small or large data, or for particular analytics tasks, meaning end-to-end data pipelines require switching between multiple solutions and computation frameworks. Often, organizations try to integrate these solutions themselves, resulting in homegrown analytics systems that are difficult to maintain and require extensive IT resources.

Intelligent Orchestration of Compute Frameworks
Datameer Smart Execution leverages and simplifies the rapidly evolving Hadoop ecosystem by intelligently and dynamically selecting the best compute framework for each step of an analytics job, orchestrating multiple compute frameworks as necessary. Once that selection is made, Smart Execution uses a snapshot of total and currently available system resources to intelligently allocate and run each concurrent workload for maximum performance.

“Datameer is addressing an emerging bottleneck with big data computing,” said Tony Baer, principle analyst at Ovum. “As Hadoop becomes multi-purpose, it is drawing a wider variety of workloads. Some utilize large clusters more efficiently than others. With Smart Execution, Datameer is introducing optimization that caches and directs the right workloads to the right compute framework.”

Smart Execution will execute analyses on large data in a Hadoop cluster using Apache Tez, an optimized form of MapReduce, while small data analysis will be executed on a single Hadoop node or using in-memory technology. This selection is completely transparent to the end user and does not require IT assistance or extra hardware or software. Most importantly, Smart Execution can easily incorporate new advances in the Hadoop ecosystem as they become available, including technologies such as Spark, as they become enterprise ready.

“We are thrilled that Smart Execution has broadened access to new Hadoop ecosystem technologies like Apache Tez and YARN,” said John Kreisa, vice president of strategic marketing at Hortonworks. “Datameer’s solution will be the first in its class to fully leverage Tez’s unique capability of supporting mixed workloads with enterprise grade performance, and will free users from having to determine what compute framework is best for their big data analytics jobs.”

The Results
Significant Performance Improvement: For environments with disparate workloads, especially with smaller data sets, performance improvements are significant. Smart Execution also dramatically improves performance for analyses that include iterative machine learning, where complex multi-step analyses leverage both big and small data, and environments where jobs of both long and short duration are always running. This enables an expanded set of use cases with Datameer, including any analytics use case where predictable and consistent low latency is a requirement.

Improved Hadoop Hardware Utilization: Smart Execution reduces network I/O, increases CPU utilization and provides more jobs per node so that users can maximize the hardware already in use. As the Hadoop system gains increased adoption, it can handle the development of new use cases and the addition of new concurrent users.

Enhanced End User Productivity: With dramatically reduced analytics cycle times, end users now have the luxury to focus on analytics rather than infrastructure technology.

“Smart Execution takes our vision of making big data analytics simple one step further by creating a new standard for data analysis,” said Stefan Groschupf, CEO of Datameer. “Now users don’t have to think about the best technology to analyze their data because it’s intelligently done for them, and they can instead spend their time focusing on discovering insights that matter.”

Learn More
Datameer 5.0 with Smart Execution will be available in Q4 2014. For more information please visit the Datameer 5.0 product page and register for the upcoming Datameer 5.0 demo webinar, which will be held Thursday, Oct. 23 at 10 a.m. PT/ 1 p.m. ET.