Hadoop provider Hortonworks announced the addition of a product called DataFlow to its Open Enterprise Hadoop platform, result of the acquisition of Onyara, a specialist in analytics on data in motion.
To help its customers address the velocity of streaming data coming from the Internet of Things Hortonworks hopes these new capabilities will help automation and securing of data flows and to collect, conduct, and curate real-time business insights and actions. In a Hortonworks blog post CTO Scott Gnau said that big data implementations need a Rosetta Stone in order to enable the integration of data from different sensors created at different times will define true success at an enterprise level.
Washington, D.C.,-based Onyara was only founded five months ago, but its engineers have worked for the National Security Agency over the past decade, and developed the technology that would eventually evolve into the Apache NiFi project. The NiFi technology aims to make the remote acquisition of data in motion easier to a robust enterprise-class dataflow solution for managing massive scale streams of data. The Hortonworks Data Platform will add DataFlow functionality, powered by Apache NiFi.
"Nearly a decade ago when IoAT began to emerge, we saw an opportunity to harness the massive new data types from people, places and things, and deliver it to businesses in a uniquely secure and simple way," said Joe Witt, chief technology officer at Onyara. "We look forward to joining the Hortonworks team and continuing to work with the Apache community to advance NiFi."
In an Onyara blog post Witt explains that a few distinct features differentiate NiFi from a growing field of data flow solutions. Witt says that NiFi's HTML5 browser-based interactive drag-and-drop interface for dataflow management and testing. The second distinction is in metadata-based chain-of-custody that provides users with insight into the ‘who, what, when, and where’ for the discrete bits of data in a NiFi dataflow.
NiFi technology was tried and tested in the intelligence community, but Hortoworks will look to build new products and services based on it, and broaden its appeal to other industries. Witt adds that resources for NiFi within Hortonworks will grow, as it looks to accelerate the pace and depth in which Onyara had already been working with Hadoop, Apache Spark, Storm, Kafka and others.
