Just a decade ago, the idea of the United States government being a Big Data warehouse for the public was silly. Today, however, the country has a chief data scientist for the first time in its history, and he showed up in Silicon Valley today to talk Big Data with the attendees of Strata.
A day after being named chief data scientist, DJ Patil told the attendees that Big Data and Hadoop have completely changed the way the United States is governed.
“The president has a data dashboard to track the spending of IT projects. We’ve got an executive order that ensures all data created by the U.S. government is open and machine-readable. That’s a massive change,” he said.
Patil added that the president is able to use data to make policy decisions, something that has been relatively unheard of in the government’s past. He said that the Data.gov site is also indicative of the change in tenor of the White House since President Barack Obama came into office. Patil said that the data sets hosted on Data.gov represent a great opportunity for developers who want to learn about Big Data.
Using these data sets, said Patil, developers can build a store of data that is complex and interesting enough to teach them about building Big Data applications. He hoped this idea would spark interest in attendees in joining the U.S. Digital Service to help the country revolutionize its IT and data infrastructures.
Patil wasn’t the only one trying to “establish a data-driven culture” in Washington and in government. Solomon Hsiang, professor of public policy at the University of California, Berkeley, spoke at the Strata keynote. “The work you are doing on Big Data analytics is having a fundamental impact on how we govern today,” he said.
“Innovations in the technology of governance have transformed our ability to deploy efficient government policies, and to make the world a better place. We are now in the midst of another revolution allowing us to better understand ourselves, and that helps us design better policies.”
Hsiang said that his work for the past few years has involved the quantifying of costs around climate change. “We’re causing the climate to change. We have the data and analytics to understand what is happening. The thing I call the fundamental climate-management problem is trying to determine if the costs of adopting a low emission scenario out-weigh the costs of climate change,” he said.
Hsiang said that data analysis tools helped him to tweeze out the hidden costs of climate change. While a typhoon or a hurricane is obviously a destructive force that does a set amount of damage in dollars, there are other repercussions of these weather events that go untabulated.
“We’ve reconstructed the exposure of individuals on the ground as hurricanes and typhoons pass over them,” he said, showing his Big Data work. “We used data on the over 6,000 hurricanes and typhoons that have taken place since 1960. These have really striking impacts on the environment and the individual. Twenty years after the storm, we can still see the signals in the macro economy. There is a US$10 trillion cost we haven’t even been considering yet.”
SQL, SQL everywhere
Two big themes at Strata from a product perspective were SQL access on Hadoop, and automated storage management and optimization tools. Many companies, like Tuplejump, offered solutions to ease developer woes around query optimization and running. Others, like Actian, Pivotal and Splice Machine, demonstrated their SQL-on-Hadoop solutions.
Pivotal open-sourced its SQL-on-Hadoop solution, HAWQ, just as Strata began, while Actian touted the performance of its solution. Hortonworks’ Hive has been updated to be ACID compliant, while Splice Machine has long offered ACID compliance on its own.
Process monitoring was another hot issue for busy Hadoop clusters. Concurrent’s Driven allows developers to monitor and observe Hadoop cluster activities and job performance, provided users are building their applications with Cascading. Pepperdata, on the other hand, offers a view into Hadoop cluster jobs without the need for code changes.
Jack Norris, CMO of MapR, said that the themes of this year’s Strata conference validated a number of new markets in the Hadoop ecosystem. He said that at the show he saw an “acknowledgement of the importance of real-time and of enterprise-grade features. In the early days of this show, it was very Hadoop-centric. Now, what we’re seeing is a lot of issues and questions and tips about integrating Hadoop with the rest of your environment, myriad project.”
Doug Cutting, creator of Apache Hadoop and a Cloudera engineer, said that he was impressed with how much the conference has grown in the past few years. His coworker, Matt Brandwein, director of product marketing at Cloudera, said that the future is bright for the company.
When asked about the recent appearance of Hortonworks on public stock exchanges, Brandwein said that Cloudera is similarly ready for an IPO, but the firm will not go public until the time is right. He said that all the needed paperwork and information has already been finished, and that an actual IPO from Cloudera could take as little as 21 days. He added, however, that Cloudera plans on staying private thanks to a huge war chest of investment capital, and revenues that have passed $100 million.
Hortonworks, on the other hand, is planning its future alongside Pivotal. Hortonworks recently took over as the Hadoop provider for existing Pivotal customers that had already purchased and installed Pivotal’s Hadoop distribution.
