Support Questions

Find answers, ask questions, and share your expertise

nifi - spark streaming

Expert Contributor

Hi,

I saw some notes on that we can use site-to-site to trigger spark streaming, also saw a note on nifi-spark-receiver jar

can someone pls suggest whats the best way to connect nifi and spark streaming.

Thanks,

Avijeet

1 ACCEPTED SOLUTION

The recommended setup for production is to use Kafka. NiFi publishes to Kafka, Spark Streaming consumes from the topic (or the reverse). Spark Receiver in NiFi works, but wasn't tested at production scale.

View solution in original post

2 REPLIES 2

The recommended setup for production is to use Kafka. NiFi publishes to Kafka, Spark Streaming consumes from the topic (or the reverse). Spark Receiver in NiFi works, but wasn't tested at production scale.

I am using Spark receiver for Nifi.. its not working..

# Site to Site properties nifi.remote.input.host=localhost nifi.remote.input.secure=false nifi.remote.input.socket.port=10000 nifi.remote.input.http.enabled=true nifi.remote.input.http.transaction.ttl=30 sec

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.