Support Questions
Find answers, ask questions, and share your expertise

Apache NiFI configuration with SPARK

Can we configure spark processor in NIFI?

3 REPLIES 3

Re: Apache NiFI configuration with SPARK

Expert Contributor

Hi @hema m,

You can use the "execute process" processor to start a spark job.

Michel

Re: Apache NiFI configuration with SPARK

Super Guru

if you use execute process make sure it's YARN mode so that it doesn't hang and wait.

Run it in YARN Cluster mode from the client spark-submit.

https://spark.apache.org/docs/latest/running-on-yarn.html