nifi - spark streaming

[#1]

Hi,

I saw some notes on that we can use site-to-site to trigger spark streaming, also saw a note on nifi-spark-receiver jar

can someone pls suggest whats the best way to connect nifi and spark streaming.

Thanks,

Avijeet

[#2 · Accepted solution]

The recommended setup for production is to use Kafka. NiFi publishes to Kafka, Spark Streaming consumes from the topic (or the reverse). Spark Receiver in NiFi works, but wasn't tested at production scale.

[#3]

I am using Spark receiver for Nifi.. its not working..

# Site to Site properties nifi.remote.input.host=localhost nifi.remote.input.secure=false nifi.remote.input.socket.port=10000 nifi.remote.input.http.enabled=true nifi.remote.input.http.transaction.ttl=30 sec