Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

spark action in oozie throw intermittent dependency error

Highlighted

spark action in oozie throw intermittent dependency error

Contributor

Hi,


I am using hortonwork HDP3.0 which has oozie 4.3.1 and spark 2.3.1. My spark job throws intermittent dependency error:


2019-04-11 06:29:07,178 [Driver] ERROR org.apache.spark.deploy.yarn.ApplicationMaster - User class threw exception: java.lang.NoClassDefFoundError: Could not initialize class com.fasterxml.jackson.databind.SerializationConfig

java.lang.NoClassDefFoundError: Could not initialize class com.fasterxml.jackson.databind.SerializationConfig

at com.fasterxml.jackson.databind.ObjectMapper.<init>(ObjectMapper.java:565)

at com.fasterxml.jackson.databind.ObjectMapper.<init>(ObjectMapper.java:480)



Caused by: com.fasterxml.jackson.databind.JsonMappingException: Incompatible Jackson version: 2.9.0

at com.fasterxml.jackson.module.scala.JacksonModule$class.setupModule(JacksonModule.scala:64)

at com.fasterxml.jackson.module.scala.DefaultScalaModule.setupModule(DefaultScalaModule.scala:19)

at com.fasterxml.jackson.databind.ObjectMapper.registerModule(ObjectMapper.java:751)

at org.apache.spark.util.JsonProtocol$.<init>(JsonProtocol.scala:59)


at org.apache.spark.util.JsonProtocol$.<clinit>(JsonProtocol.scala)



The same spark job running via spark-submit always pass.


I have tried below settings but no luck:

<configuration>
    <property>
        <name>oozie.launcher.mapreduce.user.classpath.first</name>
        <value>true</value>
    </property>
    <property>
        <name>spark.yarn.user.classpath.first</name>
        <value>true</value>
    </property>
    <property>
        <name>oozie.launcher.mapreduce.task.classpath.user.precedence</name>
        <value>true</value>
    </property>
</configuration>


Why spark action has such issue? Thanks for any clue.

1 REPLY 1

Re: spark action in oozie throw intermittent dependency error

Contributor

I read https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.0.0/running-spark-applications/content/automati...


However, I don't know where should I set "oozie.action.sharelib.for.spark.exclude=oozie/jackson.*=". Should I set in job.properties? or oozie-site? If oozie-site, I need restart oozie (it is not mentioned in the document) before running sharelibupdate. Thanks for any hints on this.


Also, I deleted all files under /user/oozie/share/lib/lib_{TS}/oozie/ (except oozie-hadoop-utils-hadoop-2-4.3.1.3.0.0.0-1634.jar and oozie-sharelib-oozie-4.3.1.3.0.0.0-1634.jar) which should have the same effect of oozie.action.sharelib.for.spark.exclude, but I still got below exception:


Caused by: com.fasterxml.jackson.databind.JsonMappingException: Incompatible Jackson version: 2.9.0
    at com.fasterxml.jackson.module.scala.JacksonModule$class.setupModule(JacksonModule.scala:64)
    at com.fasterxml.jackson.module.scala.DefaultScalaModule.setupModule(DefaultScalaModule.scala:19)
    at com.fasterxml.jackson.databind.ObjectMapper.registerModule(ObjectMapper.java:751)
    at org.apache.spark.rdd.RDDOperationScope$.<init>(RDDOperationScope.scala:82)
    at org.apache.spark.rdd.RDDOperationScope$.<clinit>(RDDOperationScope.scala)


Looks like the conflict is not coming from oozie share lib. Where this conflict could come from? Thanks again!

Don't have an account?
Coming from Hortonworks? Activate your account here