Support Questions

Find answers, ask questions, and share your expertise
Celebrating as our community reaches 100,000 members! Thank you!

Where is the Jar folder for Spark in Cloudera?


Hi. I want to know Where is the Jar folder for Spark in Cloudera? In my previous company, we just put all the needed jars inside $SPARK_HOME/jar folder (on every node), so that we don't have to worry much about the --jars, --packages,... when running the spark-submit job. Also, it saves lots of disk space and time since we don't need to include every package when building a jar.

But in my new company, which uses Cloudera, I don't know where is this jar folder. I found 2 places (maybe not the right one):

- /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars


- /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/spark/jars


Where should I put the needed jars file on? It seems like all the jars file on the second one are linked to the first one. Also, I found dozens of lib/jar folders everywhere in Cloudera.

Or is there any other way to do this with Cloudera? I read some guides about modified Spark configs on the Cloudera manager on the internet.



Master Collaborator

@quangbilly79 It should be