Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

nifi - spark streaming

avatar
Super Collaborator

Hi,

I saw some notes on that we can use site-to-site to trigger spark streaming, also saw a note on nifi-spark-receiver jar

can someone pls suggest whats the best way to connect nifi and spark streaming.

Thanks,

Avijeet

1 ACCEPTED SOLUTION

avatar

The recommended setup for production is to use Kafka. NiFi publishes to Kafka, Spark Streaming consumes from the topic (or the reverse). Spark Receiver in NiFi works, but wasn't tested at production scale.

View solution in original post

2 REPLIES 2

avatar

The recommended setup for production is to use Kafka. NiFi publishes to Kafka, Spark Streaming consumes from the topic (or the reverse). Spark Receiver in NiFi works, but wasn't tested at production scale.

avatar

I am using Spark receiver for Nifi.. its not working..

# Site to Site properties nifi.remote.input.host=localhost nifi.remote.input.secure=false nifi.remote.input.socket.port=10000 nifi.remote.input.http.enabled=true nifi.remote.input.http.transaction.ttl=30 sec