Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

error when spark 1.5 load data from relation database(oracle) using dataframe

error when spark 1.5 load data from relation database(oracle) using dataframe

Explorer

hi,I upgrade cdh from 5.4.8 to 5.5 ,and now the

 spark 1.5 cann't load data from oracle , it appears

 

 java.lang.NoSuchMethodError: com.fasterxml.jackson.databind.Module$SetupContext.setClassIntrospector(Lcom/fasterxml/jack son/databind/introspect/ClassIntrospector;)V 

I replace the jar jackson-module-scala_2.10-2.2.3 with jackson-module-scala_2.10-2.4.4 ,it doesn't work

9 REPLIES 9
Highlighted

Re: error when spark 1.5 load data from relation database(oracle) using dataframe

Master Guru

Could you share your script, and also let us know if your spark application jar bundles any form of custom jackson libraries, or any dependencies in it that transiently do so? It would be helpful for us to have the output of jar tf your-jar.jar and/or javap -classpath your-jar.jar

'com.fasterxml.jackson.databind.Module$SetupContext'.

 

The missing method has been present in jackson-databind since 2.2, so the issue is that there's a version older than 2.2 on your Spark classpath, coming in from somewhere unintentionally.

 

I'm able to run a simple RDBMS query on a clean 5.5.0 environment's spark-shell, and the below is my classpath.txt analysis:

 

~> for jr in $(grep jackson /etc/spark/conf/classpath.txt); do jar tf $jr | grep databind/Module.class && javap -classpath $jr 'com.fasterxml.jackson.databind.Module$SetupContext' | grep setClassIntrospector; done
com/fasterxml/jackson/databind/Module.class
public abstract void setClassIntrospector(com.fasterxml.jackson.databind.introspect.ClassIntrospector);
com/fasterxml/jackson/databind/Module.class
public abstract void setClassIntrospector(com.fasterxml.jackson.databind.introspect.ClassIntrospector);
com/fasterxml/jackson/databind/Module.class
public abstract void setClassIntrospector(com.fasterxml.jackson.databind.introspect.ClassIntrospector);

 

Re: error when spark 1.5 load data from relation database(oracle) using dataframe

Explorer

hi,thank you for you prompt reply. I've downgraded it to 5.4.8.   (spark 1.3, it works well),   I was wondering if you could wait for a while  until I  turn it back  and try again.  

Re: error when spark 1.5 load data from relation database(oracle) using dataframe

Explorer

I think I understand what causes the problem, I use phoenix (for cdh 5.5) ,and in its pom.xml,it integrates  with spark 1.4.

 

in /etc/spark/conf/spark-default.conf

spark.executor.extraClassPath=/opt/cloudera/parcels/CLABS_PHOENIX/lib/phoenix/phoenix-1.2.0-client.jar
spark.driver.extraClassPath=/opt/cloudera/parcels/CLABS_PHOENIX/lib/phoenix/phoenix-1.2.0-client.jar

command

./spark-shell  --jars /home/app/product/client_1/jdbc/ojdbc6.jar

scala> val jdbcDF = sqlContext.read.format("jdbc").options(Map("url" -> "jdbc:oracle:thin:test/test@//192.168.1.10:1521/orcl","dbtable" -> "store","driver" -> "oracle.jdbc.OracleDriver")).load()
scala > jdbcDF.show()

java.lang.NoSuchMethodError: com.fasterxml.jackson.databind.Module$SetupContext.setClassIntrospector(Lcom/fasterxml/jackson/databind/introspect/ClassIntrospector;)V
        at com.fasterxml.jackson.module.scala.introspect.ScalaClassIntrospectorModule$$anonfun$1.apply(ScalaClassIntrospector.scala:32)
....

 

 

 

 

Re: error when spark 1.5 load data from relation database(oracle) using dataframe

New Contributor
So could you tell me your solution? Rebuild the phoenix with spark 1.5? Or notify the phoenix the problem and wait for official answer?

Re: error when spark 1.5 load data from relation database(oracle) using dataframe

New Contributor
I got this problem too ,I wish to know how did you solve it

Re: error when spark 1.5 load data from relation database(oracle) using dataframe

Explorer

remove 

spark.executor.extraClassPath=/opt/cloudera/parcels/CLABS_PHOENIX/lib/phoenix/phoenix-1.2.0-client.jar
spark.driver.extraClassPath=/opt/cloudera/parcels/CLABS_PHOENIX/lib/phoenix/phoenix-1.2.0-client.jar

from /etc/spark/conf/spark-default.conf

 

or  downgrade cdh from 5.5 to 5.4.8  . 

Re: error when spark 1.5 load data from relation database(oracle) using dataframe

New Contributor
I hope to use the apache phoenix plugin to get data from hbase phoenix

Re: error when spark 1.5 load data from relation database(oracle) using dataframe

New Contributor
I hope to use the apache phoenix plugin to get data from hbase phoenix,so if i remove them from the conf file ,will the plugin work?

Re: error when spark 1.5 load data from relation database(oracle) using dataframe

Rising Star

I faced the same problem with CDH 5.5.1 + Phoenix 4.5.2, both installed with Parcels. I resolved (or worked around?) by prepending jackson-databind-2.2.3.jar in spark-submit jar list.

 

spark-submit --master yarn --deploy-mode client ... --jars /opt/cloudera/parcels/CDH/jars/jackson-databind-2.2.3.jar,<my jars, phoenix jar>

(I can only run my spark program in client mode. Not sure whether this workaround is usable in other modes)


There are several jackson-databind*jar in CDH Parcel jar directory. The reason I use jackson-databind-2.2.3.jar, is that there is a jackson-module-scala_2.10-2.2.3.jar. Therefore, I sticked to version 2.2.3.

Don't have an account?
Coming from Hortonworks? Activate your account here