Organizations today get that they have to collect data to stay competitive. They understand how to store it, retrieve it and slice it.
The idea now is to understand the data itself, to detect patterns and trends that will help the organization get new customers or members, service them more personally and engage with them more deeply.

“The idea of using data is ubiquitous,” Mike Olson, co-founder of Hadoop platform company Cloudera, said during a panel at the recent Strata + Hadoop World conference in New York City. “Now, we’re starting to see applications that leverage data. We’re starting to see what data enables beyond the technology. It’s less about the technology stack and about how we decide better, as people and as organizations. All this is in the service of making better decisions.”

Alistair Croll, the chairman of the conference produced by O’Reilly Media, added that Big Data today “is a double-edged sword. We need the belief that we will use data for good and not for bad—but be diligent about the bad and get the word out about it.”

Trust plays an important role in the acceptance and use of data, according to Tim O’Reilly, founder of O’Reilly Media. “We have to reframe the dialogue away from privacy and into trust. If Google tells me to leave 15 minutes earlier because of traffic, I trust it,” he said. “But what do we do when people break our trust?”

The collection and analysis of data has led to tradeoffs for a long time. For instance, life insurance companies charge higher rates for smokers, and auto insurance companies charge higher rates for young, male drivers.

Yet Croll cautioned about Big Data from a societal standpoint. “There needs to be a renegotiation. By getting so much data on individuals and putting them in buckets of risk, for example, leads us away from the idea of all of us being in this together. And individuals will be less likely to be placed with others at a higher risk to amortize that risk.”

* * *

In a move not lacking in irony, Continuuity has changed its name, to Cask. At the same time, about a month ago, it has also open-sourced its data application platform-as-a-service. I sat down with founder and CEO Jonathan Gray at Strata + Hadoop World to talk about the platform, squarely aimed at developers building data-intensive applications. He described Hadoop itself as the next-generation data management platform, but pointed out, “In the end, it’s all about apps.” Large companies, he said, are becoming large data companies, trying to monetize their data. “Data will become their lifeblood,” he said. “They’ll be building apps themselves, but it’s a challenge if you’re a bank, not Google or Facebook.”