Big Data. The concept has grown from what we in the industry used to refer to as data management to become the darling of mainstream media. What has caught the media’s eye is the benefits companies can gain from mining customer data, and of course the privacy implications of data retention, mining and sharing.

“Look at how your business can know what your customers bought, how much they spend on average, where they live, the months in which they do the most shopping, and more, all to better serve them,” the pundits coo.

In fact, as you can read in David Rubinstein’s article on page 13, the company is building out the platform for what it’s calling “the intelligent integrated enterprise.” The example the company uses is of an online shopper looking to buy a basketball, and how an intelligent integrated enterprise can know if the customer wants it shipped or to pick it up at a partner location, or if he or she also might want to buy an air pump or a hoop, as well as all the other customer data the system collects. But further, this vision enables suppliers to talk to retailers, and for data to be exchanged between mobile devices as well as a back end.

Internet companies and retailers, despite the hype, are not the only organizations that use Big Data. Medical researchers, meteorologists, national security systems, insurers and so many more rely on the insights they can glean from the mountains of data they collect to keep moving toward their goals of curing disease, or predicting weather patterns, or tracking the movements of state enemies, or understanding the driving patterns of the young and old to prevent crashes.

This is what is attracting industry giants such as IBM, Microsoft and Oracle, along with the new, emerging NoSQL players, to create offerings that can store, mine, manipulate and present information from the constant stream of data flowing into these data repositories in a way that is actionable for an organization.

New tools for creating front-end business intelligence dashboards are cropping up, as are tools and languages that facilitate the Map/Reduce technique critical to responding to queries to massive amounts of data.

If you’re reading this and thinking, “That’s great, but I work at a small company, and we can’t be bothered with this,” then you’ve already doomed yourselves to a secondary position in your market. Whether you want to believe it or not, Big Data is here now, and many of the tools needed to “do” Big Data are battle-tested and ready to use today.

So it behooves you to learn Hadoop, and Map/Reduce, and how to glean business intelligence you can put to use. That’s the only way to stay one step ahead and remain competitive in a world where your customer’s every move can be tracked, stored and mined for information that might help you reach him or her in ways that others cannot.
#!
Crowd funding: Not only for consumers
The enterprise software industry has seen numerous licensing, funding, pricing and distribution models come and go. There was a time when all compilers were expensive, programming languages cost money, and databases could cost more than the gross national product of some African nations.

While database pricing hasn’t changed much, open-source business models have turned the software development process on its head. Instead of requiring two years and millions of dollars to make a product, one could now be pulled from the open-source shelf and repurposed into a commercial software offering.

Well, another shift, similar to that of open source, is coming. This shift is into the new realm of crowd funding. Sites like Kickstarter, Indiegogo and Petridish have given the teeming masses the role of venture capitalist. The model works particularly well for software, where investors (outside the Valley, anyway) can be mystified by the prospect of funding a software-development process.

Would Samsung, HP or NVIDIA have funded a VR headset in 2012? Didn’t that technology die in 1995?

Not according to the Oculus Rift, a VR headset funded by a Kickstarter campaign. What about an Android-based videogame console? Such a console in the Ouya was funded with millions of dollars on Kickstarter.

But crowd funding isn’t just for toys anymore. Last year, Chris Granger’s Light Table IDE was funded at more than US$316,000, all because developers loved his prototype for a slick, minimal integrated development environment for user interfaces. Elsewhere, Kickstarter has been a boon to embedded developers.

As of mid-April, the UDOO project was 52 days from completing its crowd-funding campaign on Kickstarter to build a combo single-board Linux computer/Arduino platform. Such developer-focused tools aren’t the norm yet on Kickstarter or other crowd-funding platforms, but they do appear frequently enough for us to give crowd funding the thumbs up.

This is a sustainable model, not a passing fad.