Support Questions
Find answers, ask questions, and share your expertise

Override JSCH libraries for spark

Highlighted

Override JSCH libraries for spark

New Contributor

Hi Team,

 

I see below JSCH jar in the /etc/spark/conf.cloudera.spark_on_yarn/classpath.txt

/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/jsch-0.1.42.jar

 

I want to use latest jar (jsch-0.1.54.jar) of JSCH for my project. Could you please let me know how I can achieve that?

 

I have tried below in my spark-submit command. But still it is using the older (jsch-0.1.42.jar).

--conf 'spark.{driver,executor}.userClassPathFirst=true'

--conf 'spark.{driver,executor}.extraClassPath=../lib/jsch.jar'

--jars ../lib/jsch.jar

 

Also, please let me know if there is a possibility to have the latest jsch jar in the /etc/spark/conf.cloudera.spark_on_yarn/classpath.txt

4 REPLIES 4
Highlighted

Re: Override JSCH libraries for spark

Cloudera Employee

Given that JSCH is apparently used internally in Spark, I think you'd probably have to use shading to use a different version in your project.

Highlighted

Re: Override JSCH libraries for spark

New Contributor

I have tried shading also but did NOT work.

Highlighted

Re: Override JSCH libraries for spark

Cloudera Employee

What happened?

Re: Override JSCH libraries for spark

New Contributor

I tried shading and also I tried to create uber jar. But both cases, did not work. It was taking the older version which is there in the classpath.txt

Don't have an account?