Where is the Jar folder for Spark in Cloudera?

quangbilly79 — Wed, 21 Dec 2022 02:08:10 GMT

Hi. I want to know Where is the Jar folder for Spark in Cloudera? In my previous company, we just put all the needed jars inside $SPARK_HOME/jar folder (on every node), so that we don't have to worry much about the --jars, --packages,... when running the spark-submit job. Also, it saves lots of disk space and time since we don't need to include every package when building a jar.

But in my new company, which uses Cloudera, I don't know where is this jar folder. I found 2 places (maybe not the right one):

- /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars

- /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/spark/jars

Where should I put the needed jars file on? It seems like all the jars file on the second one are linked to the first one. Also, I found dozens of lib/jar folders everywhere in Cloudera.

Or is there any other way to do this with Cloudera? I read some guides about modified Spark configs on the Cloudera manager on the internet.

Re: Where is the Jar folder for Spark in Cloudera?

Kartik_Agarwal — Sat, 24 Dec 2022 05:19:20 GMT

@quangbilly79 It should be

/opt/cloudera/parcels/SPARK2/lib/

question Re: Where is the Jar folder for Spark in Cloudera? in Support Questions

Where is the Jar folder for Spark in Cloudera?

Re: Where is the Jar folder for Spark in Cloudera?