Bringing together Big Data and big business is much more work for developers than they may anticipate. At the Big Data Technology Conference in San Francisco yesterday, representatives of both the Big Data world and the big business world mingled to discuss their successes and failures in the field.

One company on hand was MetaScale, a wholly owned subsidiary of Sears Holdings. The company began as Sears’ internal Hadoop development and deployment team, but has now been unshackled from the retailer in order to help other businesses implement Big Data infrastructure and processes.

(Big Data means big money: Big Data spending to reach $114 billion in 2018)

Ankur Gupta, who heads sales and marketing at MetaScale, said that Sears began experimenting with Hadoop four years ago, giving the company a leg up on many other enterprises when it comes to Big Data processing maturity.

“We saw a business opportunity, and we thought we could provide an enterprise-based overview that’s vendor neutral and platform agnostic,” he said. “So we formed MetaScale to help other companies accelerate their Big Data initiatives, so they don’t make the same mistakes we made.

“We help companies get to production faster than they would on their own. We provide a vendor-neutral perspective. No matter if you’re an IBM shop, and HP shop or a Teradata shop, we can provide from our experiences what may and may not work for you. Similarly, with Hadoop, we have experiences in all the different distribution providers.”

Andy McNalis, Hadoop infrastructure manager at MetaScale, explained some of the customizations to its Hadoop cluster architecture during his talk, titled “Running, Managing, and Operating Hadoop at Sears.”

He said that typically, within the Sears Hadoop cluster, each machine is a simple non-redundant machine, with a single power supply and a single 4TB hard drive. The cluster’s Name Node, however, is a more robust and redundantly equipped box, capable of handling heavier workloads and remaining intact.

About Alex Handy

Alex Handy is the Senior Editor of Software Development Times.