Created 12-10-2015 07:10 PM
hi,I upgrade cdh from 5.4.8 to 5.5 ,and now the
spark 1.5 cann't load data from oracle , it appears
java.lang.NoSuchMethodError: com.fasterxml.jackson.databind.Module$SetupContext.setClassIntrospector(Lcom/fasterxml/jack son/databind/introspect/ClassIntrospector;)V
I replace the jar jackson-module-scala_2.10-2.2.3 with jackson-module-scala_2.10-2.4.4 ,it doesn't work
Created 12-12-2015 08:47 AM
Could you share your script, and also let us know if your spark application jar bundles any form of custom jackson libraries, or any dependencies in it that transiently do so? It would be helpful for us to have the output of jar tf your-jar.jar and/or javap -classpath your-jar.jar
'com.fasterxml.jackson.databind.Module$SetupContext'.
The missing method has been present in jackson-databind since 2.2, so the issue is that there's a version older than 2.2 on your Spark classpath, coming in from somewhere unintentionally.
I'm able to run a simple RDBMS query on a clean 5.5.0 environment's spark-shell, and the below is my classpath.txt analysis:
~> for jr in $(grep jackson /etc/spark/conf/classpath.txt); do jar tf $jr | grep databind/Module.class && javap -classpath $jr 'com.fasterxml.jackson.databind.Module$SetupContext' | grep setClassIntrospector; done com/fasterxml/jackson/databind/Module.class public abstract void setClassIntrospector(com.fasterxml.jackson.databind.introspect.ClassIntrospector); com/fasterxml/jackson/databind/Module.class public abstract void setClassIntrospector(com.fasterxml.jackson.databind.introspect.ClassIntrospector); com/fasterxml/jackson/databind/Module.class public abstract void setClassIntrospector(com.fasterxml.jackson.databind.introspect.ClassIntrospector);
Created 12-13-2015 06:29 PM
hi,thank you for you prompt reply. I've downgraded it to 5.4.8. (spark 1.3, it works well), I was wondering if you could wait for a while until I turn it back and try again.
Created 12-14-2015 07:18 AM
I think I understand what causes the problem, I use phoenix (for cdh 5.5) ,and in its pom.xml,it integrates with spark 1.4.
in /etc/spark/conf/spark-default.conf
spark.executor.extraClassPath=/opt/cloudera/parcels/CLABS_PHOENIX/lib/phoenix/phoenix-1.2.0-client.jar spark.driver.extraClassPath=/opt/cloudera/parcels/CLABS_PHOENIX/lib/phoenix/phoenix-1.2.0-client.jar
command
./spark-shell --jars /home/app/product/client_1/jdbc/ojdbc6.jar scala> val jdbcDF = sqlContext.read.format("jdbc").options(Map("url" -> "jdbc:oracle:thin:test/test@//192.168.1.10:1521/orcl","dbtable" -> "store","driver" -> "oracle.jdbc.OracleDriver")).load() scala > jdbcDF.show() java.lang.NoSuchMethodError: com.fasterxml.jackson.databind.Module$SetupContext.setClassIntrospector(Lcom/fasterxml/jackson/databind/introspect/ClassIntrospector;)V at com.fasterxml.jackson.module.scala.introspect.ScalaClassIntrospectorModule$$anonfun$1.apply(ScalaClassIntrospector.scala:32) ....
Created 12-28-2015 08:33 AM
Created 12-28-2015 05:21 PM
Created 12-28-2015 07:53 PM
remove
spark.executor.extraClassPath=/opt/cloudera/parcels/CLABS_PHOENIX/lib/phoenix/phoenix-1.2.0-client.jar spark.driver.extraClassPath=/opt/cloudera/parcels/CLABS_PHOENIX/lib/phoenix/phoenix-1.2.0-client.jar
from /etc/spark/conf/spark-default.conf
or downgrade cdh from 5.5 to 5.4.8 .
Created 12-28-2015 08:05 PM
Created 12-28-2015 08:07 PM
Created 01-08-2016 01:52 AM
I faced the same problem with CDH 5.5.1 + Phoenix 4.5.2, both installed with Parcels. I resolved (or worked around?) by prepending jackson-databind-2.2.3.jar in spark-submit jar list.
spark-submit --master yarn --deploy-mode client ... --jars /opt/cloudera/parcels/CDH/jars/jackson-databind-2.2.3.jar,<my jars, phoenix jar>
(I can only run my spark program in client mode. Not sure whether this workaround is usable in other modes)
There are several jackson-databind*jar in CDH Parcel jar directory. The reason I use jackson-databind-2.2.3.jar, is that there is a jackson-module-scala_2.10-2.2.3.jar. Therefore, I sticked to version 2.2.3.