Running Spark Application on Spark Standalone cluster via intellij with external jars

[#1]

Hi

I was working with running spark locally in Intellij (sending data from Nifi to spark streaming via site2site). Now I have setup spark standalone cluster and want to run my application on it. I simply changed the master URL from local [*] to

.setMaster("spark://localhost:7077")

it seems to be fine but obviously it throws the class not found error as it does not have the NiFi jars at the executors.

One possible way is to create a standalone jar and then use the spark submit script along with the fat jar to run the application.

Is it still possible to run the application via Intellij some how?

Can I set any of the following properties in the SparkConf().set to make it work?

1. SparkConf().SetJars
2. SparkConf().set (any of the properties mentioned below)
3. spark.driver.extraClassPath
4. spark.jars
5. spark.jars.packages

Can i create a fat jar and pass it to spark.driver.extraClassPath?

Thanks

[#2]

In IDE, it is better to run local mode. For other modes, please try spark-submit script. spark-submit will do some extra configuration things for you to make it work in distribuged mode.

[#3]

I dont think you can submit the code in standalone mode from IDE. I also tried and failed the same.