Enterprises have been leveraging Big Data solutions to capture, store, analyze, organize and transform their data for years; but despite the advancements made in this space, they have still not reached a transformational level of maturity, a recently released survey revealed. AtScale announced the results from its Big Data maturity survey, and found only 12 percent of respondents have reached a high-level of maturity when it comes to Big Data. This is up from 8 percent in 2016.

“The companies that achieve a high-level and transformational stage of maturity tend to consider Big Data for revenue generation. Big Data is a game changer for their organization and they are building specific applications that will allow them to monetize Big Data.  They have a roadmap for replacing their enterprise data warehouse (EDW) with a Big Data Lake. They might have one or multiple platforms across their company and are looking to provide self-serve and secure access to their applications broadly, internally and externally,” AtScale wrote. 

The 2018 Big Data Maturity survey was a three-year effort that consisted of 5,593 responses from all industries across the globe. The responses were collected in collaboration with a number of partners including  Cloudera, Hortonworks, MapR, Tableau, ODPi (a Linux Foundation Project and Apache Software Foundation.

The survey looked at how companies around the global are using Big Data and the cloud, and where these technologies are causing enterprises to struggle.

RELATED CONTENT: Where does Big Data go from here?

“Every enterprise needs to understand this report as their leaders will need to consider the impact their evolving Big Data Analytics environment has on their ability to deliver self-service analytics and manage governance,” said John Mertic, director for ODPi.

According to the report, it is clear that respondents see the benefits of utilizing Big Data. Ninety-five percent reported they plan to do as much or more with Big Data over the next three months. About 65 percent are using Big Data in a “strategic” or “game changing” way while 34 percent say they are using it in an experimental or tactical way.

However, the biggest challenges responses are experiencing with Big Data include lack of skill set, governance, performance, security and management. In addition, the report finds an overconfidence among respondents with 78 percent ranking their Big Data maturity as medium or high when in fact only a small amount of reached high-level maturity.

“CIOs are very focused internally on their success and they might be missing external context about how their peers are doing. Companies typically set targets by looking at the year before and they put a percentage factor of increase based on what they’ve done. But that’s an internal view and they don’t have complete information to make the right judgment as it relates to other companies in their industry and in this space,”AtScale wrote in an email to SD Times.

To tackle management issues, many respondents have taken their Big Data efforts to the cloud. More than half of the respondents have some amount of their current Big Data deployment in the cloud and 77 percent project they would use the cloud for Big Data. Although the report finds respondents believe the cloud is making it harder to access data. Forty-two percent of respondents have self-service access, which is down from last year, and 58 percent reported a lack of self-service to Big Data.

This is resulting in a typical enterprise data analytics portfolio to utilize hybrid data environments that combine traditional data warehousing solutions. Only 20 percent of respondents are looking to use Big Data as a placement strategy for earlier data platforms. According to AtScale, to accommodate this, chief data officers need to build out agile data environments that can accommodate traditional Business Intelligence (BI) platforms.

AtScale finds the cloud eliminates the need to manage an environment because it is managed by the cloud provider, and it enables users to use their favorite BI tool to run live reports and dashboards on Big Data.

Other findings included Tableau maintains the #1 spot on Big Data, Microsoft Excel maintains the #1 spot on small data, 52 percent of respondents are dealing with siloed decentralized analytics teams, and central business analytics units are not the norm yet.