Integrating Hortonworks DataFlow (powered by Apache NiFi) with SAS Event Stream Processing

[#1]

Hortonworks and SAS have partnered to create two new Apache NiFi processors. These processors allow data/events to be streamed between Hortonworks DataFlow (HDF) and SAS Event Stream Processing.

Why does this matter?

HDF powered by Apache NiFi, Kafka and Storm, is an integrated system for real-time dataflow management and streaming analytics on-premise or in the cloud. SAS Event Stream Processing is a real-time, low-latency, high-throughput event processing solution that can deploy SAS machine learning models. By integrating these technologies, organizations now have the option of deploying their SAS models in real-time within the Hortonworks platform. This offers flexible deployment options for your streaming analytics projects, while providing powerful analytics from SAS.

How does this integration work?

There are two new processors that can be added to NiFi:

• ListenESP: This processor initiates a listener within NiFi that receives events from the SAS Event Stream Processing data stream.
• PutESP: This processor sends events from NiFi to the SAS Event Stream Processing data stream.

Setup and configuration:

1. Download and install Hortonworks DataFlow
2. Copy the SAS .nar file to $NIFI_HOME/lib (This .nar file is provided by SAS when SAS Event Stream Processing is purchased.)
3. Edit $NIFI_HOME/conf/nifi.properties and change the web HTTP port to 31005 (nifi.web.http.port=31005) or another available port of your choice.
4. Start NiFi by running $NIFI_HOME/bin/nifi.sh run
5. Open a browser and go to http://$HOST:31005/nifi

NOTE: For this to work, SAS Event Stream Processing must be purchased and have a valid license.

Once the .nar file has been added, you will have access to the two processors within NiFi. Data events are shared using an Avro schema. Below is a basic example of a NiFi dataflow using both a ListenESP and PutESP (Shown in Figure 1).

Within the PutESP processor, you'll notice a few parameters (shown below in Figure 2😞

• Pub/Sub Host: Hostname or IP of the server running SAS Event Stream Processing.
• Pub/Sub Port: Pubsub port of the SAS Event Stream Processing engine.
• Project: SAS Event Stream Processing project name.
• Continuous Query: Name of the continuous query within the SAS Event Stream Processing project.
• Source Window: Source window within SAS Event Stream Processing where events from NiFi can be injected.

The ListenESP processor has similar parameters (shown below in Figure 3😞

For more information, check out Hortonworks DataFlow (HDF) powered by Apache NiFi and SAS Event Stream Processing.