Member since
01-23-2020
3
Posts
0
Kudos Received
0
Solutions
01-24-2020
07:26 AM
I tried shading and also I tried to create uber jar. But both cases, did not work. It was taking the older version which is there in the classpath.txt
... View more
01-23-2020
08:06 PM
Hi Team,
I see below JSCH jar in the /etc/spark/conf.cloudera.spark_on_yarn/classpath.txt
/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/jsch-0.1.42.jar
I want to use latest jar (jsch-0.1.54.jar) of JSCH for my project. Could you please let me know how I can achieve that?
I have tried below in my spark-submit command. But still it is using the older (jsch-0.1.42.jar).
--conf 'spark.{driver,executor}.userClassPathFirst=true'
--conf 'spark.{driver,executor}.extraClassPath=../lib/jsch.jar'
--jars ../lib/jsch.jar
Also, please let me know if there is a possibility to have the latest jsch jar in the /etc/spark/conf.cloudera.spark_on_yarn/classpath.txt
... View more
Labels:
- Labels:
-
Apache Spark
-
Apache YARN