Big Data software and tools provider Revelytix today announced early access availability of Loom Dataset Management for Hadoop which makes it easier for data scientists to work with Hadoop and easier for their organizations to manage the huge challenges of big data files created with Hadoop.

Loom Dataset Management for Hadoop tracks the lineage and provenance of all registered HDFS data and offers query execution using SQL, SPARQL or HiveQL, as well as integration with R.

Revelytix CEO Mike Lang stated, “Loom makes it easy for data scientists and IT to build more analytics faster with easy-to-use interfaces that simplify getting the right data for the job quickly and managing datasets efficiently over time with proper tracking and data auditing.

“Hadoop is a breakthrough technology whose benefits can be harnessed for many enterprises – and we are just at the beginning of this story.  While some enterprises are using Hadoop as an important resource, for the most part enterprises are only now starting to move from concept evaluations to pilot programs to production – first for single point applications and ultimately, for most organizations, to enterprise platforms. One of the key challenges is becoming clear —  Managing all this data is a huge challenge – both for organizations and their data scientists. Loom makes all of that much, much easier.  All large enterprises that support multiple stakeholders and datasets need this capability as part of their Hadoop program.  Hadoop is hard.  You’re going to need Loom.”

Data scientists need tools to describe data, automatically track their activities and make the hard work of creating the finished product from the raw material easier.  Finding the right data sets and proving they meet criteria is usually a complex and time-consuming problem.  Metadata management is a critical piece of every information management system and Loom is a new kind of product for Hadoop – a new generation of easy to use metadata management that is tightly integrated with Hadoop.

Loom automatically captures detailed, timestamped data lineage and provenance information Hadoop datasets.

Loom Dataset Management for Hadoop includes:
Lineage – Loom Dataset Lineage ensures that you always know where a dataset came from. This is vital for knowing which datasets are suitable for which tasks and for supporting data audits.

Extensible Registry – Loom Extensible Registry enables users to enter customized metadata about datasets and other entities so they are easier to find and manage.
Active Scan – Loom Active Scan dynamically profiles datasets, collecting valuable metadata, including table, column, and partition statistics.

Lab Bench – Loom Lab Bench provides data scientists with a lightweight interface for finding, transforming, and analyzing data in Hadoop and Hive.

Data Suitability – Finding the right data for a task is one of the most difficult challenges confronting data scientists.  Loom ensures that every dataset is described well-enough that data scientists can easily find the right data for the job. 

Open APIs – All Loom metadata and functionality is exposed through published, RESTful APIs. Loom comes with a package for R that is built on these APIs.

For data scientists who may spend as much as 75% or more of their time munging or wrangling datasets,  Loom boosts productivity by letting the data scientist get easier access to the data they need using the tools they use today – Hive and R.

Based on nearly a decade of designing and building big data fabrics and solutions for the United States Department of Defense, the leading intelligence organizations in the United States and major pharmaceutical, financial services and life sciences companies, Loom is the product of deep big data experience.  Because Hadoop makes practical so many new analytics and datasets, Loom’s tracking and management capabilities are fundamental to managing datasets in Hadoop.

Loom Early Access program announced today will engage a group of enterprises with Loom Dataset Management for Hadoop.