Override JSCH libraries for spark

[#1]

Hi Team,

I see below JSCH jar in the /etc/spark/conf.cloudera.spark_on_yarn/classpath.txt

/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/jsch-0.1.42.jar

I want to use latest jar (jsch-0.1.54.jar) of JSCH for my project. Could you please let me know how I can achieve that?

I have tried below in my spark-submit command. But still it is using the older (jsch-0.1.42.jar).

--conf 'spark.{driver,executor}.userClassPathFirst=true'

--conf 'spark.{driver,executor}.extraClassPath=../lib/jsch.jar'

--jars ../lib/jsch.jar

Also, please let me know if there is a possibility to have the latest jsch jar in the /etc/spark/conf.cloudera.spark_on_yarn/classpath.txt

[#2]

Given that JSCH is apparently used internally in Spark, I think you'd probably have to use shading to use a different version in your project.

[#3]

I have tried shading also but did NOT work.

[#4]

What happened?

[#5]

I tried shading and also I tried to create uber jar. But both cases, did not work. It was taking the older version which is there in the classpath.txt