As the world of Hadoop grows larger, and distributed Big Data applications begin to require hundreds if not thousands of servers, the very act of managing your clusters can become a discipline unto itself. For this reason, I’ve been working on a bigger story for SD Times about Riemann, a network monitoring construct for distributed systems.

At its heart, Riemann is a language and environment for monitoring streams of data. You can use the Riemann language, which is built on top of Clojure, to easily define actions to be taken upon events. In this way, Riemann is a bit like a complex event-processing domain-specific language, or a CEP DSL.

But because Riemann is built from Clojure, those triggered events can be quite complex, even in Java. This entire party happens on the JVM, so if you’re willing to face the trouble that can come from building on a stack three languages deep, you can actually build some exceptionally powerful application-monitoring tools with Riemann.

I first heard about Riemann from Sam Ramji at Apigee. He said that a team there had been evaluating Storm for system monitoring. After some work, the team came across Riemann, which they are still evaluating.

Paul Kim, an engineer at Apigee (the company has no job titles), said that “Storm is not a monitoring solution, and it’s not advertised as that. We are looking at Storm, but by itself, it is not going to cut it. You have to set it up for metrics.”

Riemann, on the other hand, is based on Clojure, which is a Lisp dialect and a functional programming language. As a result, it has many functional advantages that are simply a factor of how functional languages deal with state and variables.

“Since it was built into the language, the original author (Kyle Kingsbury) put a lot of primitives in place that give you a library of functions for these streams. You can handle them, combine them, split them, create a fixed window, track them, do thresholds. There are math primitives for standard deviations, averages, etc. A key part, and I assume the reason he chose Clojure, is that if you didn’t have something you need, being a lisp language you can create any function you’re missing,” said Kim.

We’ll have more on Riemann in an upcoming issue.