Support Questions

Find answers, ask questions, and share your expertise
Check out our newest addition to the community, the Cloudera Data Analytics (CDA) group hub.

Running Spark Application on Spark Standalone cluster via intellij with external jars

Expert Contributor


I was working with running spark locally in Intellij (sending data from Nifi to spark streaming via site2site). Now I have setup spark standalone cluster and want to run my application on it. I simply changed the master URL from local [*] to


it seems to be fine but obviously it throws the class not found error as it does not have the NiFi jars at the executors.

One possible way is to create a standalone jar and then use the spark submit script along with the fat jar to run the application.

Is it still possible to run the application via Intellij some how?

Can I set any of the following properties in the SparkConf().set to make it work?

  1. SparkConf().SetJars
  2. SparkConf().set (any of the properties mentioned below)
  3. spark.driver.extraClassPath
  4. spark.jars
  5. spark.jars.packages

Can i create a fat jar and pass it to spark.driver.extraClassPath?



Expert Contributor

In IDE, it is better to run local mode. For other modes, please try spark-submit script. spark-submit will do some extra configuration things for you to make it work in distribuged mode.

New Contributor

I dont think you can submit the code in standalone mode from IDE. I also tried and failed the same.

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.