Database market consolidation is inevitable – how to pick the winners?

Databases comprise one of the largest and most foundational segments of the IT landscape—in some cases, comprising over 50 percent of the cost of the total application stack. The size and prominence of this segment continues to expand unfettered. Recent estimates from Gartner peg the database market at $36.4 billion, having grown 8.6 percent in 2016. Investment in new database technologies and platforms continues to flourish. CBInsights reports that the database market segment has garnered an average of three funding rounds adding up to over $44 million in each of the last twenty quarters. Open source projects have further contributed to the proliferation of database management systems (DBMS), comprising over 7 percent of the overall market.

This flurry of investment and innovation may suggest that this segment is an emerging segment; that is certainly not the case. Rather, the database market is rapidly evolving and one may define the arc of its evolution in two distinct waves. The first wave was the growth and dominance of relational database offerings from the likes of Oracle, Microsoft, IBM, and SAP. The second wave, which began to pick up momentum about 10 years ago, includes non-relational offerings such as NoSQL and Hadoop and has ushered in a surge of new vendors and innovation. The database popularity ranking compiled monthly by DB-Engines now covers more than 300 DBMSs, but this market is likely to morph to a more sustainable dynamic with significant consolidation and a clear definition of the winners.

There’s no doubt that the database market is experiencing a feverish rate of change and growth. But what’s driving this change? And how are database vendors responding to these drivers in order to remain relevant and survive the coming consolidation? To answer these questions, let’s take a closer look at the core dynamics currently at play—and how these forces are shaping the future of the database market.

• Data explosion is unstoppable. IDC forecasts that by 2025 the global datasphere will grow to 163 zettabytes, 10x the number in 2016. And while this data explosion is delivering unprecedented user experiences and business opportunities, it is also delivering unprecedented challenges in data storage and access that must be overcome with new database technology.
• Real-time performance is a business imperative. Analysts estimate that, in less than 10 years, more than a quarter of data created will be real-time in nature. Additionally, in a world where everything is connected, it is estimated that our interactions with real-time data will grow, with the average rate per capita of data-driven interactions increasing 20-fold over the next 10 years. Enterprises will not be able to survive without the ability to harness this real-time data; not only to deliver highly personalized experiences to end users, but also to extract up-to-the-moment business insights for internal decision-making processes.
• NoSQL database platforms define the path forward. Cloud native applications drove demand for scale-out architectures (versus traditional scale-up models). Development overhead associated with maintaining relational schemas became too rigid and inefficient, hence the shift to the schema on read approach, which doesn’t require that a schema be defined upfront. This need for scalability and flexibility in data schemas, along with overall ease of data management, are the primary drivers for the growth of the NoSQL DBMS segment. Unstructured data has already surpassed total structured data generated globally and this trajectory will not change; NoSQL will be the primary DBMS platform, solving increasing numbers of data management challenges. A report by Sumologic published in November of 2016 showed that two of the top three databases on AWS are NoSQL.
• In-memory databases take mainstage. High throughput (reaching and sometime exceeding a million ops/sec) and low latency (sub-milliseconds in many cases) demands on databases are becoming commonplace. So too are the increasing number of hybrid use cases incorporating transactions and analytics in conjunction with one another. The only way to tackle this level of complexity while still maintaining a high level of performance is to manage data in RAM. With the continued price drop in memory and innovations, in-memory databases are penetrating the market at an unprecedented pace—recent projections put this segment at $7B by 2020. Companies like Intel, Samsung and IBM have introduced new persistent memory technologies which are accelerating the cost reduction curve, making in-memory databases even more attractive.
• DBaaS is not a fad. Gartner projects that, by 2018, more than 70% of new DBMS deployments will leverage the cloud for at least one use case. Given the acceleration of workloads shifting to the cloud and the fact that the vast majority of new applications are cloud native, this is clearly the new normal. Early trepidations related to security, data governance, and other perceived cloud vulnerabilities have been superseded by the tremendous value proposition of the cloud which brings the benefits of pay-as-you-go pricing, elimination of talent related constraints, reduction in resource requirements, scalability, and ultimate flexibility. As a result, almost all DBMS vendors are now moving to offer their platforms in the DBaaS (Database as a Service) deployment model.
• Open source is mandatory. The BOSS Index, created to track the growth of open-source software, credits five of its chosen top 15 OSS projects to the DBMS market. Since 2014, the collective revenue of these open-source DBMS vendors has nearly doubled. Having a large community of developers to drive innovation, cultivate a talent pool, continuously expand the ecosystem and its use cases, advocate for the value proposition, and spread the knowledge base through formal training and apprenticeship, is critical for the long term sustainability of any database platform.

Understanding the above dynamics can help DBMS platform providers define a successful roadmap. One that should be sure to include open source support, developer affinity, seamless scalability, high performance, schema flexibility, multi-model support, native cloud architecture that additionally supports on-premises and hybrid deployments, storage extensibility, and skills availability.

As the mega vendors, particularly public cloud providers such as AWS, GCP, and Azure, broaden their offerings and strengthen their investments in the dbPaaS approach, we should expect to see several vendors drop out of the market. Others will remain afloat with venture funding, but will be unable to capture critical mass and cross the chasm. Still others will be acquired by mega vendors or execute mergers of equals. And a tiny number of ventures, likely around 5, will experience the adequate market success to support profitable stand-alone existences. It is clear that no matter how large a market segment may be, eventual maturity, customer acceptance, self-selection, and execution excellence in the face of ever-evolving challenges will lead to stratification between the winners and everyone else.