I am getting invalid class serialize exception spark submit client mode and it working good local mode.Can any one help me out.
java.io.InvalidClassException:org.apache.spark.sql.hive.execution.InsertIntoHiveTable; local class incompatible: stream classdesc serialVersionUID=439602153812587, local class serailVersionUID=1424907733712746510
This usually happens if you have a conflicting version of JAR that contains the class "org.apache.spark.sql.hive.execution.InsertIntoHiveTable"
- So can you please check the classpath of spark-submit in client mode to see from which jar is it taking that class?
Spark 1.6 and 2.0 jars are not mixed. For example, the "spark-assembly-1.6*xxx.jar" (spark) and "spark-hive_2.xxxx.jar" (spark2) both contains this class. So please make sure that you are not mixing the claspath to include both the JARs. You can check the "--jars" parameter values.
Thank you for your quick response.On server side it using the following assembly spark-assembly-188.8.131.52.5.3.44-1-hadoop184.108.40.206.5.3.44-1.jar and on driver side i am using spark-hive_2.10-1.6.2 jar file and am using hortonworks repository on driver side.I am completely block here and unable to move forward.I apperciate your help.
I didn't find spark-assembly-1.6xxxxx.jar file in horton repo.(So please make sure that you are not mixing the claspath to include both the JARs)Do you know how to verify mixing class paths on server side.On my cluster we have spar1.6.2 and spark2 also installed.