Support Questions
Find answers, ask questions, and share your expertise

nifi - spark streaming

Solved Go to solution
Highlighted

nifi - spark streaming

Expert Contributor

Hi,

I saw some notes on that we can use site-to-site to trigger spark streaming, also saw a note on nifi-spark-receiver jar

can someone pls suggest whats the best way to connect nifi and spark streaming.

Thanks,

Avijeet

1 ACCEPTED SOLUTION

Accepted Solutions

Re: nifi - spark streaming

The recommended setup for production is to use Kafka. NiFi publishes to Kafka, Spark Streaming consumes from the topic (or the reverse). Spark Receiver in NiFi works, but wasn't tested at production scale.

View solution in original post

2 REPLIES 2

Re: nifi - spark streaming

The recommended setup for production is to use Kafka. NiFi publishes to Kafka, Spark Streaming consumes from the topic (or the reverse). Spark Receiver in NiFi works, but wasn't tested at production scale.

View solution in original post

Highlighted

Re: nifi - spark streaming

I am using Spark receiver for Nifi.. its not working..

# Site to Site properties nifi.remote.input.host=localhost nifi.remote.input.secure=false nifi.remote.input.socket.port=10000 nifi.remote.input.http.enabled=true nifi.remote.input.http.transaction.ttl=30 sec