Reply
New Contributor
Posts: 1
Registered: ‎09-12-2017

[Serialization Error] InvalidClassException: scala.Option local class incompatible

I wrote an application and compiled as a jar, the application use akka as a channel to communicate with others outside of the cluster.

 

My build sbt is:

scalaVersion := "2.11.8"

libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % "2.1.0" % "provided",
"org.apache.spark" %% "spark-streaming" % "2.1.0" % "provided",
"com.typesafe.akka" %% "akka-actor" % "2.3.16",
"com.typesafe.akka" %% "akka-remote" % "2.3.16"
)

assemblyOption in assembly := (assemblyOption in assembly).value.copy(includeScala = false)

 

But when I submit to CDH-Spark cluster, and receive a AKKA message from the outside, I got a Serialization error.

[INFO] [09/12/2017 17:44:19.443] [PredictSystem-akka.actor.default-dispatcher-34] [akka.tcp://PredictSystem@10.1.4.166:48675/user/predictor] [PredictActor] preStart()
[INFO] [09/12/2017 17:44:19.443] [PredictSystem-akka.actor.default-dispatcher-34] [akka.tcp://PredictSystem@10.1.4.166:48675/user/predictor] -------------------------------->IdentifyRequest
[ERROR] [09/12/2017 17:44:19.642] [PredictSystem-akka.remote.default-remote-dispatcher-12] [akka.tcp://PredictSystem@10.1.4.166:48675/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FgetPredict%4010.1.4.172%3A6666-0/endpointWriter] AssociationError [akka.tcp://PredictSystem@10.1.4.166:48675] -> [akka.tcp://getPredict@10.1.4.172:6666]: Error [scala.Option; local class incompatible: stream classdesc serialVersionUID = -2062608324514658839, local class serialVersionUID = -114498752079829388] [
java.io.InvalidClassException: scala.Option; local class incompatible: stream classdesc serialVersionUID = -2062608324514658839, local class serialVersionUID = -114498752079829388
	at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:617)
	at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1622)
	at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)
	at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1622)
	at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
	at akka.serialization.JavaSerializer$$anonfun$1.apply(Serializer.scala:136)
	at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
	at akka.serialization.JavaSerializer.fromBinary(Serializer.scala:136)
	at akka.serialization.Serialization$$anonfun$deserialize$1.apply(Serialization.scala:104)
	at scala.util.Try$.apply(Try.scala:192)
	at akka.serialization.Serialization.deserialize(Serialization.scala:98)
	at akka.remote.MessageSerializer$.deserialize(MessageSerializer.scala:23)
	at akka.remote.DefaultMessageDispatcher.payload$lzycompute$1(Endpoint.scala:58)
	at akka.remote.DefaultMessageDispatcher.payload$1(Endpoint.scala:58)
	at akka.remote.DefaultMessageDispatcher.dispatch(Endpoint.scala:76)
	at akka.remote.EndpointReader$$anonfun$receive$2.applyOrElse(Endpoint.scala:967)
	at akka.actor.Actor$class.aroundReceive(Actor.scala:467)
	at akka.remote.EndpointActor.aroundReceive(Endpoint.scala:437)
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
	at akka.actor.ActorCell.invoke(ActorCell.scala:487)
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
	at akka.dispatch.Mailbox.run(Mailbox.scala:220)
	at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
	at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
]

 

I have noticed that the cause is Scala 2.10(CDH uses) don't code the SerivalVersionUID, but Scala 2.11 is. So the  SerivalVersionUID is not consistent.

 

My problem is if I use Scala2.10 compiler,  the AKKA version is not correct. Please tell me which Scala version and AKKA version I should use if I want to running the application in CDH.

 

The cluster enviroment is:

CDH-5.12.0-1.cdh5.12.0.p0.29 with SPARK2-2.1.0.cloudera1-1.cdh5.7.0.p0.120904

 

Thanks in adavance.

 

Announcements