Hi. I want to know Where is the Jar folder for Spark in Cloudera? In my previous company, we just put all the needed jars inside $SPARK_HOME/jar folder (on every node), so that we don't have to worry much about the --jars, --packages,... when running the spark-submit job. Also, it saves lots of disk space and time since we don't need to include every package when building a jar.
But in my new company, which uses Cloudera, I don't know where is this jar folder. I found 2 places (maybe not the right one):
- /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars
- /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/spark/jars
Where should I put the needed jars file on? It seems like all the jars file on the second one are linked to the first one. Also, I found dozens of lib/jar folders everywhere in Cloudera.
Or is there any other way to do this with Cloudera? I read some guides about modified Spark configs on the Cloudera manager on the internet.
