Data can be viewed to be as valuable in the digital business world as oil is to the economy. Oil fuels homes, cars, railroads, and even electricity. Data powers the business and revenue. Businesses that want to continue to evolve, transform and improve need are being fueled by their data to tell them where and when to make changes.

It’s not just about obtaining data though. Businesses that want to remain competitive need to gain intelligent and valuable insights fast and in real-time. Yesterday’s data will be worthless tomorrow when every second to deliver high-quality experiences matters today. If a service is slow, if a system goes down, or if a customer can’t find what they are looking for, they won’t waste too much time with your business before moving on.

The challenge, however, is that there is an influx of data coming in from every direction, and that is making it difficult to collect, store, and manage data. The traditional approach to storing data is to put it on a disk-based system for later integration and analysis. In addition to being slow and not conducive to today’s fast-paced way of working, disk requires a lot of overhead involved with querying, finding and accessing data.

“Customer expectations around latency and responsiveness have skyrocketed; today everyone expects constant availability but latency is now considered the new downtime. In other words, brownout is the new blackout in today’s world. In order for businesses to meet and exceed these customer expectations, businesses and IT leaders can no longer rely on old traditional ways of storing and managing data,” said Madhukar Kumar, vice president of product marketing for database provider Redis Labs.

The alternative approach is to store data in memory, but that has not been an option to many businesses because of the cost expense compared to disk until just recently. Recent advances in the memory space are making storing data in memory and in-memory databases more cost effective, according to Mat Keep, senior director of products and solutions for the database platform provider MongoDB.Keep explained that over the last couple of years, memory technology has improved with capacity going up and costs going down. “We are seeing real growth and commoditization of in-memory technology,” he said.

This gives you a competitive advantage because with in-memory computing, businesses can feed their compute capability at a much higher rate and get a much faster access path to data, Keep explained.

“The reality is that disk-based systems are just too slow. You can do all these optimizations to your data, but in the end what a company needs is 10 times to 100 times more improvement and in-memory computing is the only way to get that scale and performance,” Abe Kleinfeld, president and CEO of GridGain Systems, an in-memory computing platform provider, added. “The good news is that computer hardware doesn’t cost that much anymore. It continues to get cheaper and cheaper. You can fit a lot of memory in modern computers today.”

According to the research firm Forrester, an in-memory database refers to “a database that stores all or most critical data in DRAM, flash, and SSD on either a single or distributed server to support various types of workloads, including transactional, operational, and/ or analytical workloads, running on-premises or in the cloud.”

While in-memory systems have been around for years, the costs associated with it placed it in very niche applications for very specialized tasks such as critical systems, explained Keep. Thanks to lower memory prices, Forrester principal analyst Noel Yuhanna sees in-memory technologies moving more towards complex mobile, web and interactive workloads such as real-time analytics, fraud detection, mobile ads and IoT apps.

“We find that everyone wants to run operational reporting and insights quickly to make better business decisions. This is where in-memory comes in. In-memory allows you to process data more quickly than traditional disk-drives. Customers are telling us that they are getting five times or even 10 times faster results with in-memory,” said Yuhanna.

How in-memory databases can strengthen your data strategy
The fact is that the increasing amount of data is putting extreme pressure on existing infrastructure, according to Yuhanna. The main benefit of applying in-memory data technology to your strategy is the amount of speed you get. “The more data you have in-memory the quicker it takes to do the analytics such as fraud detection, patient monitoring, customer-360 and other use cases. Many organizations struggle with performance of critical apps, an issue that’s further exacerbated by growing data volumes and velocities. Business users want fast, real-time data, while IT wants to lower cost and improve operational efficiencies,” Yuhanna said.

With in-memory databases, there is no disk input/output, so that means applications can read and update data in milliseconds, according to Redis Labs’ Kumar. “An in-memory database stores the data in the memory which eliminates seek time when querying the data, thereby making data access exponentially faster,” Kumar said.

This is because data is being stored closest to where the data processing is happening, according to Ravi Mayuram, CTO for Couchbase, a NoSQL database provider. Mayuram explained that processing happens in the CPU, and the closest place for the CPU to consume data is in the main memory of a system. “Memory is the place where you keep the data and where computing can happen the fastest,” he said. “That is where you can get those fast response time. In-memory computing is becoming more and more ubiquitous.”

It also enables businesses to develop applications that are “population scale,” which refers to providing a service or solution that anyone in the world can access, said Mayuram.

“It is really being able to provide much lower latency to your users, much more predictable latency, and fast responsiveness of your applications. If you are running analytics, being able to do that in-memory enables you to pull that data out much more faster,” said MongoDB’s Keep. “Speed is the competitive advantage today. How quickly can you act on data, how quickly can you use it, and how quickly can you extract insights from it.”

The in-memory speed also makes it easier to work with data. “In-memory is great at bringing in a lot of data, having it constantly be updated, and then being able to look at all the data and generate it in real-time,” said Nima Negahban, CTO of analytics database provider Kinetica. “It just makes things a whole lot easier because it is faster.”

In addition, Negahban mentioned the desire to generate maximum ROI from an enterprise’s data lake or data stream is fueling the move toward in-memory technology. In 2010, the Apache Hadoop movement sparked an investment in terms of data generation and storage. What Hadoop has accomplished is the ability to collect all this data, but then the next step was to derive insight and value from that. “That part of the problem is what the Hadoop movement didn’t solve very well, and that is where in-memory data comes in,” he said.

The old days of running a batch report nightly and generate a static report are over. When you are going through a digital transformation and trying to create a business around data, you need real-time visibility into that data, and disk-based systems aren’t going to get you there, Negahban explained.

“Everyone is turning into a technology company, and everyone is turning into a company where customer experience is the number one priority. It is real-time customer experience based on real-time information, and in that world, traditional computing is just too slow. No matter what type of industry you are in, you are trying to collect information about your customers in real time and act in real time to take advantage of opportunities that present themselves from moment to moment. In-memory computing is the only way to achieve that success,” said GridGain’s Kleinfeld.

How in-memory databases can weaken your data strategy
Reduced cost, speed and real-time insights… what could go wrong? In-memory technology is not a silver bullet, and still comes with its own set of limitations and disadvantages.

For instance, while there have been huge advancements to reduce the costs of memory, it is still expensive. The price of RAM has become static, and databases as well as businesses are constantly working toward compressing the cost curve, Kinetica’s Negahban explained.

“You cannot put all your data completely in-memory,” said Forrester’s Yuhanna. “You must be selective on what data and what apps would benefits.”

Businesses now need to consider the cost of memory and figure out how they can get the most benefit with the least amount of memory, such as compressing data, storing data on disk or only keeping a certain amount of data in memory, according to Couchbase’s Mayuram.

“In memory gives you a big performance boost, but comes with the [barriers] of being more expensive and complex, especially as the datasets grow or exceed a system,” said MongoDB’s Keep. If you have only a certain amount of in-memory data you can use, once your data exceeds the available memory, the system will either stop accepting data, crash or crash with an out-of-memory message, he added. From there, the business has to figure out whether to increase the available memory in their system, or add more services or instances.

“Storing data in memory means that when the data volume becomes very large—for example hundreds of terabytes—it becomes expensive to store all of that data in memory since RAM is more expensive than disk storage,” said Redis Labs’ Kumar

In addition, in-memory is not persistent, meaning it has to be backed up with persistent storage to ensure data reliability, according to Forrester’s Yuhanna.

Unless you are checkpointing your data from in-memory to disk — which is going to slow down the system — if your system dies, you will have lost that data, Keep added. There are efforts to improve the reliability of in-memory. Keep says as memory continues to get cheaper and capacity increases, we will start to see the adoption of non-volatile RAM such as Intel and Micron’s 3D XPoint technology, which can provide a persistent form of memory so if you lose power, your data is still persisted. Keep does warn about the wear rate of this type of technology though. “The more you write data to this device, the more it actually wears out, so you can actually start to lose data again,” Keep said.

“Regardless, in-memory technology is here to stay, and its usage is likely to expand to all apps and insights in the coming years,” Yuhanna said.

As time goes on, Kinetica’s Negahban agrees in-memory technology will continue to expand and evolve and move toward what some call the data gravity theory. “Wherever the data lives is where the rest of the solution stack will move toward because data is expensive and the need to be able to have fast access and do mutations of that data is what drives all computing,” he said. “So as data continues to explode and as in-memory databases get more creative, we are going to see databases take on a lot more roles where they become places where you house app containers, visualization engines, machine learning moderns and more. The data gravity phenomenon is going to take effect.”

Overcoming the challenges
To get the most out of in-memory data, Yuhanna sees organizations leveraging a tiered storage strategy. “Organizations are putting hot data in DRAM, warm data in SSD/Flash and cloud data on traditional disk drives. Many apps, databases and cloud solutions are starting to leverage tiered storage in a more intelligent and automated manner that’s helping accelerate insights,” Yuhanna said.

MongoDB’s Keep does not think we will ever get to a place where everything is completely stored in-memory. “The reality is it is never going to happen because the amount of data we are collecting is growing faster than in-memory technology,” he said. Instead, what Keep expects to see is in-memory and disk-based systems merge into a single intelligent system where the “freshest,” most important data will be held close to the processor such as in-memory and the slightly more “aged” data that is accessed less frequently will be stored into disk. “There will never be a winner-takes-all. What will you see is a combination of these technologies to give the best latency to the hottest part of data,” he said.

“For data that requires instantaneous processing, the best practice is to store and process in-memory,” Redis Labs’ Kumar added. “For data that doesn’t require frequent reads/writes and can be processed with batch jobs, the general acceptable practice is to store them on a disk.”

In order to properly do this, Keep suggested finding a database provider that offers both in-memory and disk so developers don’t have to worry about a different set of APIs to master and a different set of technology to operationalize.

Tools should also be built to a standard, according to Couchbase’s Mayuram. This way enterprises can easily adopt multiple tools such as data curation, data migration and data integration that work together in one ecosystem without having to learn or change their underlying workflow, he said.

“In-memory database technology should be part of your data management strategy going forward. Organizations who utilize in-memory are more likely to succeed in delivering a flexible, scalable, and next-generation platform. Focus initially on the apps and insights that are more frequently accessed and require low-latency access. Supplement your in-memory deployment with SSD, Flash, and hard disk for persistence to support data recoverability and lower cost,” said Yuhanna.

Top use cases by industry
In-memory technology is still in its early days, with Forrester estimating that about 35 percent of enterprises leverage in-memory databases for critical business apps and insights. Forrester expects that number to double over the next three years. The biggest use cases will be around real-time apps with low latency, customer analytics, IoT apps and mobile apps that need integrated data, according to Forrester.

In addition, Forrester’s Yuhanna also sees in-memory technology benefiting the following industries. He wrote:

Financial services: Most financial servers have relied on in-memory for a decade to support their key applications and are now starting to take advantage of in-memory databases to support applications that mostly exist on relational models.

Retailers: Customer data stored and processed in-memory help create an opportunity for businesses to upsell and cross-sell new products to a customer based on his or her likes, dislikes, circle of friends, buying patterns, and past orders.

Manufacturer: Today, most manufacturers deal with highly sophisticated machinery to support their plants, whether building a car, airplane, or tire, or bottling wine or soda. With in-memory and Big Data technologies, manufacturers can track machines every minute or even every second to determine if any machine is likely to fail, as well as what parts might be needed to be repaired if a breakdown does occur. As a result, this is allowing large manufacturers to be more proactive, saving them millions of dollars from downtime.

Online gaming: Online gaming has changed dramatically over the years to deliver rich graphics, a global user base, and real-time interactive experiences. In-memory delivers a real-time data platform that can track individuals, groups, game scores, personalized user preferences, and interactions as well as manage millions of connected users, all in real-time.