spark action in oozie throw intermittent dependency error

[#1]

Hi,

I am using hortonwork HDP3.0 which has oozie 4.3.1 and spark 2.3.1. My spark job throws intermittent dependency error:

2019-04-11 06:29:07,178 [Driver] ERROR org.apache.spark.deploy.yarn.ApplicationMaster - User class threw exception: java.lang.NoClassDefFoundError: Could not initialize class com.fasterxml.jackson.databind.SerializationConfig

java.lang.NoClassDefFoundError: Could not initialize class com.fasterxml.jackson.databind.SerializationConfig

at com.fasterxml.jackson.databind.ObjectMapper.<init>(ObjectMapper.java:565)

at com.fasterxml.jackson.databind.ObjectMapper.<init>(ObjectMapper.java:480)

Caused by: com.fasterxml.jackson.databind.JsonMappingException: Incompatible Jackson version: 2.9.0

at com.fasterxml.jackson.module.scala.JacksonModule$class.setupModule(JacksonModule.scala:64)

at com.fasterxml.jackson.module.scala.DefaultScalaModule.setupModule(DefaultScalaModule.scala:19)

at com.fasterxml.jackson.databind.ObjectMapper.registerModule(ObjectMapper.java:751)

at org.apache.spark.util.JsonProtocol$.<init>(JsonProtocol.scala:59)

at org.apache.spark.util.JsonProtocol$.<clinit>(JsonProtocol.scala)

The same spark job running via spark-submit always pass.

I have tried below settings but no luck:

<configuration>
    <property>
        <name>oozie.launcher.mapreduce.user.classpath.first</name>
        <value>true</value>
    </property>
    <property>
        <name>spark.yarn.user.classpath.first</name>
        <value>true</value>
    </property>
    <property>
        <name>oozie.launcher.mapreduce.task.classpath.user.precedence</name>
        <value>true</value>
    </property>
</configuration>

Why spark action has such issue? Thanks for any clue.

[#2]

I read https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.0.0/running-spark-applications/content/automati...

However, I don't know where should I set "oozie.action.sharelib.for.spark.exclude=oozie/jackson.*=". Should I set in job.properties? or oozie-site? If oozie-site, I need restart oozie (it is not mentioned in the document) before running sharelibupdate. Thanks for any hints on this.

Also, I deleted all files under /user/oozie/share/lib/lib_{TS}/oozie/ (except oozie-hadoop-utils-hadoop-2-4.3.1.3.0.0.0-1634.jar and oozie-sharelib-oozie-4.3.1.3.0.0.0-1634.jar) which should have the same effect of oozie.action.sharelib.for.spark.exclude, but I still got below exception:

Caused by: com.fasterxml.jackson.databind.JsonMappingException: Incompatible Jackson version: 2.9.0
    at com.fasterxml.jackson.module.scala.JacksonModule$class.setupModule(JacksonModule.scala:64)
    at com.fasterxml.jackson.module.scala.DefaultScalaModule.setupModule(DefaultScalaModule.scala:19)
    at com.fasterxml.jackson.databind.ObjectMapper.registerModule(ObjectMapper.java:751)
    at org.apache.spark.rdd.RDDOperationScope$.<init>(RDDOperationScope.scala:82)
    at org.apache.spark.rdd.RDDOperationScope$.<clinit>(RDDOperationScope.scala)

Looks like the conflict is not coming from oozie share lib. Where this conflict could come from? Thanks again!