Getting in a Big Data state of mind

One of the most hyped terms in information management today is Big Data. Everyone seems excited by the concept and related possibilities, but is the “Big” in Big Data mere a state of mind? Yes, it is.

Organizations are realizing the potential challenges resulting from the explosion of unstructured information: e-mails, images, log files, cables, user-generated content, documents, videos, blogs, contracts, wikis, Web content… the list goes on. Traditional technologies and information management practices may no longer prove sufficient given today’s information environment. Therefore, forward-looking organizations and individuals are evaluating new technologies and concepts like Big Data in an attempt to address the unstructured information overload that is currently underway. Some forward-thinking organizations even view these challenges as opportunities to derive new insight and gain competitive advantage.

On the other side of the equation, some technology vendors are equally excited by the potential of Big Data as a Big Market Opportunity. Storage vendors focus on the sheer volume of Big Data (petabytes of information giving way to exabytes and eventually zettabytes) and the need for organizations to have efficient and comprehensive storage for all of that information. Data warehousing and business intelligence vendors emphasize the need for advanced statistical and predictive analytical capabilities to sift through the vast volume of information to make sense of it, and to find the proverbial needle in the haystack, typically using newer technologies such as MapReduce and Hadoop, which are cheaper than previous technologies. And so on.

While advanced organizations see tremendous opportunity for harnessing Big Data, how do they start addressing both the challenges and the potential opportunities given the numerous definitions and confusion that exists in the marketplace today? What if I belong to an organization that doesn’t have information in the petabytes (going to exabytes) scale? Does Big Data apply to me—or you?

This is why Big Data can be considered a state of mind rather than something specific. It becomes a concept that is applicable to any organization that feels that current tools, technologies, and processes are no longer sufficient for managing and taking advantage of their information management needs, regardless of whether its data is measured in gigabytes, terabytes, exabytes or zettabytes.

But let’s be clear: Size matters here. If you have information measured in the hundreds of terabytes to tens of petabytes, you have a Big Data problem. However, outside of Internet titans such as Google and Facebook, large global financial service institutions such as JP Morgan Chase and Morgan Stanley; healthcare institutions dealing with patient records; government agencies such as the National Archives and Records Administration; and those capturing a flood of sensor data and images, many organizations (perhaps the majority) are not operating anywhere close to that scale, aspirations aside.

So what does the low-volume majority have in common with the massive-scale minority when it comes to Big Data? It is making sense and understanding the value of the information that they have regardless of size. Ask yourself: “If I do not have information measured in the hundreds of terabytes, why aren’t my current tools and techniques up for the task?” The answer is that the fundamental nature, as well as the consumption, of information has evolved. In other words, even if you don’t have petabytes, you may have a Big Data problem.

What are you going to do about it?

While traditional relational databases have been around for almost 40 years, they did not approach ubiquity until the 1980s and early 1990s. This correlated with the popularity of enterprise software applications such as ERP or CRM, with database technologies as their foundation for storing and managing information.

Although technologies enabled tremendous productivity gains and better insight, almost everything about these applications and the underlying information was structured. Users would input the “first name” for a contact only in a specified field in the UI (not to exceed a predefined number of characters) while they had to enter the contact’s “last name” in a different field (once again, not to exceed a predefined number of characters). They could only query predefined information from a given screen based on predefined criteria determined by the underlying relational tables and optimized indexes.

Similarly, traditional business intelligence and analytics tools require organizations to integrate various systems and applications with time-consuming ETL tools to map the fields from one table to another. Unstructured information such as proposals, contracts and other documents was a second-class citizen, either stored separately in a file system or as “clumps” within the database with limited capabilities to search or analyze details within.

Fast-forward to today. Huge amounts of business information lives “in the wild” outside of the purview of traditional enterprise software systems. Anyone can blog using the tool of their choice without restrictions on how the content is entered. End users typically drive organic growth and evolution of knowledge systems such as wikis and SharePoint instances. External audiences can share both positive and negative information about a product or an organization via social media tools such as Facebook, Twitter and Yelp. Data marketplaces and exchanges are beginning to facilitate the consumption of potentially valuable public sources of information.

In other words, we have moved away from an environment where information is typically internal to the organization and highly “structured,” to a situation where valuable information, typically unstructured in nature, exists both internally as well as externally. It originates from a variety of sources in a variety of formats and mediums.

Not only is the information today highly variable, but the way in which it’s consumed and leveraged now and how it will be in the future is evolving. With new devices such as the iPad and the proliferation of app stores, we are seeing the appification of content and information based on specific context and task at hand.

We are also observing the consumerization of enterprise software applications with business users expecting similar Web 2.0 capabilities to those they use in their personal lives (Saleforce.com’s Chatter application is an example of this).

Organizations are seeing tremendous value in sharing data and information externally as a means to engage with broader audiences in a controlled way.

So, if the origins and formats of today’s information are both more variable and numerous, and the ways in which information may be consumed are equally variable and numerous, then a mismatch exists between today’s information and traditional tools and processes, which are predicated on “structured” input and consumption. If your organization is struggling to manage and leverage today’s information (more sources, more variability, more potential applications) with your existing processes and tools, then you have a Big Data problem.

Given this definition of Big Data, if your organization has a Big Data problem your traditional processes and technologies cannot properly address, what should you do about it?

To start with, understand your organization’s current challenges and what you want to do about them. If you are worried about the volume of information growing within your organization, you look at creating a comprehensive and efficient storage strategy. Then again, does it make sense to grow storage capacity linearly with information volume? Perhaps it makes sense to first analyze and understand the information you have, identify how much of that information is valuable, and establish processes, and leverage tools that enable you to effectively keep the information that is valuable and discard the information that is not.

If information volume is not the primary issue, the focus may instead be on managing and leveraging information that is highly variable in nature, changing rapidly, or being consumed in ways that are unforeseen and unknown ahead of time. Your organization may then consider technologies and strategies that enable better agility both in terms of the types of information as well as consumption and delivery.

In other words, focus on aspects of Big Data in a way that makes sense for your organization. Don’t get caught up in the hype. After all, Big Data is merely a state of mind.

Kenneth Chestnut is vice president of product marketing at MarkLogic, which sells Big Data solutions.