Hsiang said that data analysis tools helped him to tweeze out the hidden costs of climate change. While a typhoon or a hurricane is obviously a destructive force that does a set amount of damage in dollars, there are other repercussions of these weather events that go untabulated.
“We’ve reconstructed the exposure of individuals on the ground as hurricanes and typhoons pass over them,” he said, showing his Big Data work. “We used data on the over 6,000 hurricanes and typhoons that have taken place since 1960. These have really striking impacts on the environment and the individual. Twenty years after the storm, we can still see the signals in the macro economy. There is a US$10 trillion cost we haven’t even been considering yet.”
SQL, SQL everywhere
Two big themes at Strata from a product perspective were SQL access on Hadoop, and automated storage management and optimization tools. Many companies, like Tuplejump, offered solutions to ease developer woes around query optimization and running. Others, like Actian, Pivotal and Splice Machine, demonstrated their SQL-on-Hadoop solutions.
Pivotal open-sourced its SQL-on-Hadoop solution, HAWQ, just as Strata began, while Actian touted the performance of its solution. Hortonworks’ Hive has been updated to be ACID compliant, while Splice Machine has long offered ACID compliance on its own.
Process monitoring was another hot issue for busy Hadoop clusters. Concurrent’s Driven allows developers to monitor and observe Hadoop cluster activities and job performance, provided users are building their applications with Cascading. Pepperdata, on the other hand, offers a view into Hadoop cluster jobs without the need for code changes.
Jack Norris, CMO of MapR, said that the themes of this year’s Strata conference validated a number of new markets in the Hadoop ecosystem. He said that at the show he saw an “acknowledgement of the importance of real-time and of enterprise-grade features. In the early days of this show, it was very Hadoop-centric. Now, what we’re seeing is a lot of issues and questions and tips about integrating Hadoop with the rest of your environment, myriad project.”
Doug Cutting, creator of Apache Hadoop and a Cloudera engineer, said that he was impressed with how much the conference has grown in the past few years. His coworker, Matt Brandwein, director of product marketing at Cloudera, said that the future is bright for the company.
When asked about the recent appearance of Hortonworks on public stock exchanges, Brandwein said that Cloudera is similarly ready for an IPO, but the firm will not go public until the time is right. He said that all the needed paperwork and information has already been finished, and that an actual IPO from Cloudera could take as little as 21 days. He added, however, that Cloudera plans on staying private thanks to a huge war chest of investment capital, and revenues that have passed $100 million.
Hortonworks, on the other hand, is planning its future alongside Pivotal. Hortonworks recently took over as the Hadoop provider for existing Pivotal customers that had already purchased and installed Pivotal’s Hadoop distribution.