Hortonworks announced today that it has updated its Hortonworks DataFlow (HDF) platform with new features and updated Apache projects. HDF 2.0 includes a drastically redesigned Apache NiFi project, better handling of cluster-wide security policies, and a more portable version of its dataflow management layer.

Apache NiFi is at the top of the list of changes. This project was originally created at the NSA and picked up by Hortonworks last year. For HDF 2.0, Apache NiFi has been updated to version 1.0. The big change for this version is the complete rewrite of the NiFi GUI.

NiFi is used to lay out data flows for processing and control. The UI is similar to something like a Microsoft Visio, or other modeling tools, and allows users to develop data flows by pointing and clicking through their design process.

NiFi also received a little brother in the form of MiNiFi, a miniature client for Apache NiFi. The idea is to lay out data flows in the cloud, then run MiNiFi on edge devices, such as Internet of Things devices, so the data flow controls are  propagated to the edges of the network.

Jamie Engesser, vice president of product management for Hortonworks, said that HDF 2.0 is all about handling Hadoop use cases beyond just data lakes and static data in place. “Traditional Hadoopers have done an impressive job of learning about data at rest, but the next generation is looking at net new use cases where they want to deal with data in the stream. The notion of data end-to-end is a key driver of what’s going on. I need the data at rest, and I need it in motion too,” said Engesser.

To this end, HDF 2.0 also includes Apache Storm 1.01 and Apache Kafka .1.0. HDF 2.0 is also compatible with Apache Ranger, and can use this project to manage security policies throughout a cluster. This will make things easier to manage, as in HDF 1.0, each Apache project was managed independently.

“We’ve converge that under Apache Ambari, which is a management console. It can deploy, monitor and manage. It also has the ability to define security policies. You can define security all the way from the edge to the core landing in Hadoop in one security policy flow through to the end,” said Engesser.

Hortonworks has not announced an exact release date for HDF 2.0, but Engesser said that it will ship sometime in the next 10 days.