Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here. Want to know more about what has changed? Check out the Community News blog.

[Serialization Error] InvalidClassException: scala.Option local class incompatible

[Serialization Error] InvalidClassException: scala.Option local class incompatible

New Contributor

I wrote an application and compiled as a jar, the application use akka as a channel to communicate with others outside of the cluster.

 

My build sbt is:

scalaVersion := "2.11.8"

libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % "2.1.0" % "provided",
"org.apache.spark" %% "spark-streaming" % "2.1.0" % "provided",
"com.typesafe.akka" %% "akka-actor" % "2.3.16",
"com.typesafe.akka" %% "akka-remote" % "2.3.16"
)

assemblyOption in assembly := (assemblyOption in assembly).value.copy(includeScala = false)

 

But when I submit to CDH-Spark cluster, and receive a AKKA message from the outside, I got a Serialization error.

[INFO] [09/12/2017 17:44:19.443] [PredictSystem-akka.actor.default-dispatcher-34] [akka.tcp://PredictSystem@10.1.4.166:48675/user/predictor] [PredictActor] preStart()
[INFO] [09/12/2017 17:44:19.443] [PredictSystem-akka.actor.default-dispatcher-34] [akka.tcp://PredictSystem@10.1.4.166:48675/user/predictor] -------------------------------->IdentifyRequest
[ERROR] [09/12/2017 17:44:19.642] [PredictSystem-akka.remote.default-remote-dispatcher-12] [akka.tcp://PredictSystem@10.1.4.166:48675/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FgetPredict%4010.1.4.172%3A6666-0/endpointWriter] AssociationError [akka.tcp://PredictSystem@10.1.4.166:48675] -> [akka.tcp://getPredict@10.1.4.172:6666]: Error [scala.Option; local class incompatible: stream classdesc serialVersionUID = -2062608324514658839, local class serialVersionUID = -114498752079829388] [
java.io.InvalidClassException: scala.Option; local class incompatible: stream classdesc serialVersionUID = -2062608324514658839, local class serialVersionUID = -114498752079829388
	at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:617)
	at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1622)
	at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)
	at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1622)
	at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
	at akka.serialization.JavaSerializer$$anonfun$1.apply(Serializer.scala:136)
	at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
	at akka.serialization.JavaSerializer.fromBinary(Serializer.scala:136)
	at akka.serialization.Serialization$$anonfun$deserialize$1.apply(Serialization.scala:104)
	at scala.util.Try$.apply(Try.scala:192)
	at akka.serialization.Serialization.deserialize(Serialization.scala:98)
	at akka.remote.MessageSerializer$.deserialize(MessageSerializer.scala:23)
	at akka.remote.DefaultMessageDispatcher.payload$lzycompute$1(Endpoint.scala:58)
	at akka.remote.DefaultMessageDispatcher.payload$1(Endpoint.scala:58)
	at akka.remote.DefaultMessageDispatcher.dispatch(Endpoint.scala:76)
	at akka.remote.EndpointReader$$anonfun$receive$2.applyOrElse(Endpoint.scala:967)
	at akka.actor.Actor$class.aroundReceive(Actor.scala:467)
	at akka.remote.EndpointActor.aroundReceive(Endpoint.scala:437)
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
	at akka.actor.ActorCell.invoke(ActorCell.scala:487)
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
	at akka.dispatch.Mailbox.run(Mailbox.scala:220)
	at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
	at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
]

 

I have noticed that the cause is Scala 2.10(CDH uses) don't code the SerivalVersionUID, but Scala 2.11 is. So the  SerivalVersionUID is not consistent.

 

My problem is if I use Scala2.10 compiler,  the AKKA version is not correct. Please tell me which Scala version and AKKA version I should use if I want to running the application in CDH.

 

The cluster enviroment is:

CDH-5.12.0-1.cdh5.12.0.p0.29 with SPARK2-2.1.0.cloudera1-1.cdh5.7.0.p0.120904

 

Thanks in adavance.