Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Override JSCH libraries for spark

Override JSCH libraries for spark

New Contributor

Hi Team,

 

I see below JSCH jar in the /etc/spark/conf.cloudera.spark_on_yarn/classpath.txt

/opt/cloudera/parcels/CDH-5.16.2-1.cdh5.16.2.p0.8/jars/jsch-0.1.42.jar

 

I want to use latest jar (jsch-0.1.54.jar) of JSCH for my project. Could you please let me know how I can achieve that?

 

I have tried below in my spark-submit command. But still it is using the older (jsch-0.1.42.jar).

--conf 'spark.{driver,executor}.userClassPathFirst=true'

--conf 'spark.{driver,executor}.extraClassPath=../lib/jsch.jar'

--jars ../lib/jsch.jar

 

Also, please let me know if there is a possibility to have the latest jsch jar in the /etc/spark/conf.cloudera.spark_on_yarn/classpath.txt

4 REPLIES 4
Highlighted

Re: Override JSCH libraries for spark

Cloudera Employee

Given that JSCH is apparently used internally in Spark, I think you'd probably have to use shading to use a different version in your project.

Highlighted

Re: Override JSCH libraries for spark

New Contributor

I have tried shading also but did NOT work.

Highlighted

Re: Override JSCH libraries for spark

Cloudera Employee

What happened?

Highlighted

Re: Override JSCH libraries for spark

New Contributor

I tried shading and also I tried to create uber jar. But both cases, did not work. It was taking the older version which is there in the classpath.txt

Don't have an account?
Coming from Hortonworks? Activate your account here