Reply
Explorer
Posts: 16
Registered: ‎10-27-2015

error when spark 1.5 load data from relation database(oracle) using dataframe

hi,I upgrade cdh from 5.4.8 to 5.5 ,and now the

 spark 1.5 cann't load data from oracle , it appears

 

 java.lang.NoSuchMethodError: com.fasterxml.jackson.databind.Module$SetupContext.setClassIntrospector(Lcom/fasterxml/jack son/databind/introspect/ClassIntrospector;)V 

I replace the jar jackson-module-scala_2.10-2.2.3 with jackson-module-scala_2.10-2.4.4 ,it doesn't work

Posts: 1,896
Kudos: 433
Solutions: 303
Registered: ‎07-31-2013

Re: error when spark 1.5 load data from relation database(oracle) using dataframe

Could you share your script, and also let us know if your spark application jar bundles any form of custom jackson libraries, or any dependencies in it that transiently do so? It would be helpful for us to have the output of jar tf your-jar.jar and/or javap -classpath your-jar.jar

'com.fasterxml.jackson.databind.Module$SetupContext'.

 

The missing method has been present in jackson-databind since 2.2, so the issue is that there's a version older than 2.2 on your Spark classpath, coming in from somewhere unintentionally.

 

I'm able to run a simple RDBMS query on a clean 5.5.0 environment's spark-shell, and the below is my classpath.txt analysis:

 

~> for jr in $(grep jackson /etc/spark/conf/classpath.txt); do jar tf $jr | grep databind/Module.class && javap -classpath $jr 'com.fasterxml.jackson.databind.Module$SetupContext' | grep setClassIntrospector; done
com/fasterxml/jackson/databind/Module.class
public abstract void setClassIntrospector(com.fasterxml.jackson.databind.introspect.ClassIntrospector);
com/fasterxml/jackson/databind/Module.class
public abstract void setClassIntrospector(com.fasterxml.jackson.databind.introspect.ClassIntrospector);
com/fasterxml/jackson/databind/Module.class
public abstract void setClassIntrospector(com.fasterxml.jackson.databind.introspect.ClassIntrospector);

 

Explorer
Posts: 16
Registered: ‎10-27-2015

Re: error when spark 1.5 load data from relation database(oracle) using dataframe

hi,thank you for you prompt reply. I've downgraded it to 5.4.8.   (spark 1.3, it works well),   I was wondering if you could wait for a while  until I  turn it back  and try again.  

Explorer
Posts: 16
Registered: ‎10-27-2015

Re: error when spark 1.5 load data from relation database(oracle) using dataframe

I think I understand what causes the problem, I use phoenix (for cdh 5.5) ,and in its pom.xml,it integrates  with spark 1.4.

 

in /etc/spark/conf/spark-default.conf

spark.executor.extraClassPath=/opt/cloudera/parcels/CLABS_PHOENIX/lib/phoenix/phoenix-1.2.0-client.jar
spark.driver.extraClassPath=/opt/cloudera/parcels/CLABS_PHOENIX/lib/phoenix/phoenix-1.2.0-client.jar

command

./spark-shell  --jars /home/app/product/client_1/jdbc/ojdbc6.jar

scala> val jdbcDF = sqlContext.read.format("jdbc").options(Map("url" -> "jdbc:oracle:thin:test/test@//192.168.1.10:1521/orcl","dbtable" -> "store","driver" -> "oracle.jdbc.OracleDriver")).load()
scala > jdbcDF.show()

java.lang.NoSuchMethodError: com.fasterxml.jackson.databind.Module$SetupContext.setClassIntrospector(Lcom/fasterxml/jackson/databind/introspect/ClassIntrospector;)V
        at com.fasterxml.jackson.module.scala.introspect.ScalaClassIntrospectorModule$$anonfun$1.apply(ScalaClassIntrospector.scala:32)
....

 

 

 

 

New Contributor
Posts: 7
Registered: ‎12-28-2015

Re: error when spark 1.5 load data from relation database(oracle) using dataframe

So could you tell me your solution? Rebuild the phoenix with spark 1.5? Or notify the phoenix the problem and wait for official answer?
New Contributor
Posts: 7
Registered: ‎12-28-2015

Re: error when spark 1.5 load data from relation database(oracle) using dataframe

I got this problem too ,I wish to know how did you solve it
Explorer
Posts: 16
Registered: ‎10-27-2015

Re: error when spark 1.5 load data from relation database(oracle) using dataframe

remove 

spark.executor.extraClassPath=/opt/cloudera/parcels/CLABS_PHOENIX/lib/phoenix/phoenix-1.2.0-client.jar
spark.driver.extraClassPath=/opt/cloudera/parcels/CLABS_PHOENIX/lib/phoenix/phoenix-1.2.0-client.jar

from /etc/spark/conf/spark-default.conf

 

or  downgrade cdh from 5.5 to 5.4.8  . 

New Contributor
Posts: 7
Registered: ‎12-28-2015

Re: error when spark 1.5 load data from relation database(oracle) using dataframe

I hope to use the apache phoenix plugin to get data from hbase phoenix
New Contributor
Posts: 7
Registered: ‎12-28-2015

Re: error when spark 1.5 load data from relation database(oracle) using dataframe

I hope to use the apache phoenix plugin to get data from hbase phoenix,so if i remove them from the conf file ,will the plugin work?
Contributor
Posts: 46
Registered: ‎11-03-2014

Re: error when spark 1.5 load data from relation database(oracle) using dataframe

I faced the same problem with CDH 5.5.1 + Phoenix 4.5.2, both installed with Parcels. I resolved (or worked around?) by prepending jackson-databind-2.2.3.jar in spark-submit jar list.

 

spark-submit --master yarn --deploy-mode client ... --jars /opt/cloudera/parcels/CDH/jars/jackson-databind-2.2.3.jar,<my jars, phoenix jar>

(I can only run my spark program in client mode. Not sure whether this workaround is usable in other modes)


There are several jackson-databind*jar in CDH Parcel jar directory. The reason I use jackson-databind-2.2.3.jar, is that there is a jackson-module-scala_2.10-2.2.3.jar. Therefore, I sticked to version 2.2.3.