Hadoop has, for the most part, moved beyond the proof-of-concept phase and the initial chasm of adoption. More and more organizations are putting the open-source framework to work on mountains of complex Big Data. The next step in Hadoop’s evolution is getting a handle on governance.

To that end, Hortonworks—the enterprise data platform provider and open-source Apache Hadoop contributor—announced a new Data Governance Initiative (DGI) to design and implement a comprehensive, centralized approach to data governance. The initiative’s goals range from defining governance standards and protocols within Hadoop, to recreating real-time auditable and traceable data landscapes.

Ultimately, though, the DGI aims to tie together the rapidly growing ecosystem of Hadoop projects, from HBase, Hive and Pig to Drill, Mahout and ZooKeeper, with a flexible standardized system that serves as a foundation and a common conduit for Hadoop’s growth.

“Everybody has Big Data problems,” said Andrew Ahn, director of governance on Hortonworks’ product management team. “We really needed to tag data well. Just having a metadata tag or a business glossary isn’t that difficult to do in and of itself, but there really wasn’t a product that fit all needs. There is no one-size-fits-all solution; it just doesn’t exist. Oftentimes Hadoop is considered sort of black box. You have a lot of workflows, but once it enters this governance area, it kind of goes dark. There’s not a lot of visibility in there.”

A new governance paradigm
The initiative is also indicative of the new industries exploring the possibilities afforded by Hadoop and Big Data processing. The DGI’s contributors are not traditional technology partners; aside from Hortonworks, the initiative’s founding members include healthcare giant Aetna, pharmaceutical corporation Merck, and Target, one of the largest retail chains in the U.S.

“From a business standpoint, [governance] is a pain point that a lot of customers have, hence this consortium we’re building,” Ahn said. “We have a lot of industry players. You see one from retail, a multinational pharmaceutical company, a healthcare provider and SAS [Hortonworks’ technology partner, a business analytics software provider]. It really is a testament to where a lot of folks are in their Hadoop journey and the realization that governance isn’t something you can just tack on. You have to plan for it, and it has to be integral to your overall strategy.”

Hortonworks designed the DGI based on the use cases of its partners, building governance policies based on business taxonomies in what Ahn called a “happy meal” of data tags for organizations to deploy code immediately into production with the knowledge that they’re always in compliance.

In this way, Hortonworks and its partners see the DGI as a new paradigm for software development; with the open-source community ultimately building governance capabilities based on individual enterprise needs. Ahn pointed to a retail application as an example of creating a culture around governance.