Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Spark2 jars issue

Highlighted

Spark2 jars issue

New Contributor

We have 2 versions of Spark, Spark1.6 and Spark2.0. 

 

The jobs in test cluster seem to be working fine but the jobs in production has issues with jars. It is somehow not taking the Spark2 jars in class path.

 

I have checked all the configs which are exactly same in both test and prod.

 

The current issue I'm facing is 

java.io.InvalidClassException: org.apache.commons.lang3.time.FastDateFormat; local class incompatible: stream classdesc serialVersionUID = 1, local class serialVersionUID = 2
	at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:616)

However I tried to overwrite the specific jar and I face another issue which also seems to be regarding the jar mismatch.

 

Please let me know how this can be resolved. 

3 REPLIES 3
Highlighted

Re: Spark2 jars issue

New Contributor

I'm facing the same issues.

 

With this code I can reproduce the problem:

 

 

classOf[org.apache.commons.lang3.time.FastDateFormat].getProtectionDomain().getCodeSource()
val par = sc.parallelize(Array("1","2","3"))
par.foreach{x: String => println(x + classOf[org.apache.commons.lang3.time.FastDateFormat].getProtectionDomain().getCodeSource())}

This returns on the driver:

java.security.CodeSource = (file:/opt/cloudera/parcels/SPARK2-2.0.0.cloudera1-1.cdh5.7.0.p0.113931/lib/spark2/jars/commons-lang3-3.3.2.jar <no signer certificates>)

On the excecutors:

2(file:/opt/cloudera/parcels/SPARK2-2.0.0.cloudera1-1.cdh5.7.0.p0.113931/lib/spark2/jars/hive-exec-1.1.0-cdh5.7.0.jar <no signer certificates>)

 

Highlighted

Re: Spark2 jars issue

New Contributor

I just updated to the new version of spark: 2.0.0.cloudera2-1.cdh5.7.0.p0.118100

 

And cannot reproduce the issue. So I'm a happy camper ;)

Highlighted

Re: Spark2 jars issue

New Contributor

Yeah, moving to new parcel solved things for me too!

Don't have an account?
Coming from Hortonworks? Activate your account here