Developers have always known that storing objects in memory is much faster than storing them on disks. But over the past five years, we’ve all been told that flash memory would be replacing traditional hard disks, and thus, developers could begin to see disks as a faster storage medium.
But somewhere along the line, things have gotten tangled up and confused. While flash storage makers pump their capabilities, developers are running back to RAM for just about any type of problem you can encounter. This is made even more obvious by the trend towards NoSQL usage, and the recent marketing boom around in-memory data grids.
Hammering this point home even further has been the Apache Spark project, where Hadoop jobs go to become in-memory native. Spark has been gathering a large amount of buzz for the past year, and not just because it can handle streams and output Hadoop results in near real-time.
At its core, Spark represents the overall return to RAM, which was never really much of a return. After all, hard drives haven’t been keeping pace with Moore’s Law over the past five years, thanks in large part to various tsunamis taking out South Pacific havens of drive manufacture.
Combine that with a vicious market for flash storage that can easily get you fired for picking an unreliable solution, and you’ve got a recipe that makes developers’ mouths salivate for RAM.
While Spark may not be an in-memory data grid, or a NoSQL solution, it does show that even the largest of projects with the most important of business applications can still be improved by an architectural change that favors the more volatile, but much faster, RAM.
