Reply
Explorer
Posts: 16
Registered: ‎02-17-2015

Re: Spark (Standalone) error local class incompatible: stream classdesc serialVersionUID

I checked my network cofig, everything seems to be alright. Every node can communicate with every other node in my cluster. My entire firewall has been disabled, so this may not  be a 'port-not-open' issue. I was looking at this other post which I think is discussing the same connectivity error: http://community.cloudera.com/t5/Advanced-Analytics-Apache-Spark/Akka-Error-while-running-Spark-Jobs...

 

Could you pls let me know if there is any spark cofig files or any spark specific setting that I need to look into?

 

Thank you

Explorer
Posts: 16
Registered: ‎02-17-2015

Re: Spark (Standalone) error local class incompatible: stream classdesc serialVersionUID

[ Edited ]

Going back to the earlier versionUID conflict error(java.io.InvalidClassException: org.apache.spark.rdd.PairRDDFunctions; local class incompatible: stream classdesc serialVersionUID = 8789839749593513237, local class serialVersionUID = -4145741279224749316), I've found out that my application used the spark jar file called: spark-core_2.10-1.2.0-cdh5.3.0.jar, this .jar file contains the path org.apache.spark.rdd.PairRDDFunctions as shown in the error, how do i check the serialVersionUID in this jar? and could you pls tell me what other spark jar (from cloudera manager/cdh) could this jar be possibily conflicting with? is it with the spark-assembly.jar?

 

 

Explorer
Posts: 6
Registered: ‎04-01-2015

Re: Spark (Standalone) error local class incompatible: stream classdesc serialVersionUID

Hi,

 

I have exactly the same issue and and trying to do a BDD install too, did you find a solution to the problem?

Explorer
Posts: 16
Registered: ‎02-17-2015

Re: Spark (Standalone) error local class incompatible: stream classdesc serialVersionUID

Actually yes, i figured out what the problem was. The CDH jars that are shipped with BDD are of version 5.3.0 whereas the cloudera cdh that I have installed on my cluster is of version 5.3.2. Due to this version mismatch i was getting this error. I removed CDH 5.3.2 and replaced it with cloudera parcels of version 5.3.0 (basically a fresh installation of CM and other hadoop components) and then this error doesn't appear.

 

However, once that error is cleared I'm facing another issue which i have on this post: http://community.cloudera.com/t5/Advanced-Analytics-Apache-Spark/Spark-ERROR-CoarseGrainedExecutorBa...

 

Could you pls let me know(after you implement the version change) if you're getting this same error or if BDD works all the way? Thanks

 

 


Bob

Explorer
Posts: 6
Registered: ‎04-01-2015

Re: Spark (Standalone) error local class incompatible: stream classdesc serialVersionUID

Will do for sure, I have had my head burried in log files this afternoon and I think I am going crazy!

Explorer
Posts: 16
Registered: ‎02-17-2015

Re: Spark (Standalone) error local class incompatible: stream classdesc serialVersionUID

There's very few people out there working on BDD, pls do keep me posted on how things work once you make this fix.

New Contributor
Posts: 3
Registered: ‎04-22-2015

Re: Spark (Standalone) error local class incompatible: stream classdesc serialVersionUID

Hi, I am also facing the similar error/issue. Could you please let me know whether you are able to fix this issue. I am using pseudo cluster with latest cloudera software and Oracle Big Data 1.0
Explorer
Posts: 16
Registered: ‎02-17-2015

Re: Spark (Standalone) error local class incompatible: stream classdesc serialVersionUID

The CDH jars that are shipped with BDD are of version 5.3.0 whereas the cloudera cdh that I have installed on my cluster is of version 5.3.2. Due to this version mismatch i was getting this error. I removed CDH 5.3.2 and replaced it with cloudera parcels of version 5.3.0 (basically a fresh installation of CM and other hadoop components) and then this error doesn't appear.

New Contributor
Posts: 3
Registered: ‎04-22-2015

Re: Spark (Standalone) error local class incompatible: stream classdesc serialVersionUID

Hi, Dataminer, many thanks for quick post. I have got two exceptions 1. java.io.InvalidClassException: org.apache.spark.rdd.PairRDDFunctions; local class incompatible: stream classdesc serialVersionUID = 8789839749593513237, local class serialVersionUID = -4145741279224749316 and the second one ERROR CoarseGrainedExecutorBackend: Driver Disassociated . Are you able to fix both of these by installing CDH 5.3.0, How to find what version of CDH jars were shipped with BDD as i have downloaded BDD software last week, I am not sure whether still BDD has CDH 5.3.0 jars.
Explorer
Posts: 16
Registered: ‎02-17-2015

Re: Spark (Standalone) error local class incompatible: stream classdesc serialVersionUID

For me both these issues were resolved once i installed cdh 5.3.0. The cdh version of the bdd jars can be found under the folder ${oracle_home}/Middleware/BDD1.0/dataprocessing/edp_cli/libs

Announcements