spark2 job on oozie

[#1]

when I use oozie to schedule spark2 on yarn.I have e pproblem :

Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SparkMain], main() threw exception, null
java.lang.NullPointerException
        at scala.collection.mutable.ArrayOps$ofRef$.newBuilder$extension(ArrayOps.scala:190)
        at scala.collection.mutable.ArrayOps$ofRef.newBuilder(ArrayOps.scala:186)
        at scala.collection.TraversableLike$class.filterImpl(TraversableLike.scala:246)
        at scala.collection.TraversableLike$class.filter(TraversableLike.scala:259)
        at scala.collection.mutable.ArrayOps$ofRef.filter(ArrayOps.scala:186)
        at org.apache.spark.deploy.yarn.Client$anonfun$prepareLocalResources$6.apply(Client.scala:531)
        at org.apache.spark.deploy.yarn.Client$anonfun$prepareLocalResources$6.apply(Client.scala:527)
        at scala.collection.mutable.ArraySeq.foreach(ArraySeq.scala:74)
        at org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:527)
        at org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:868)
        at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:170)
        at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:56)
        at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:156)
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:509)
        at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)

[#2]

@Xiong Duan Based on the error stack I think it could be due missing configuration. It would be helpful if you share your workflow file and properties file. Have you tried using shell action instead ?

[#3]

Thank you very much! It is a problem abour jar conflict.

[#4 · Accepted solution]

When use spark2.1,We have to add

oozie-sharelib-spark-4.2.0.jar

in spark2

[#5]

Hi , I am facing this issues .

2018-08-21 14:27:16,632 ERROR [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
java.lang.IllegalArgumentException: Invalid ContainerId: container_e63_1532612094367_17160_02_000001
	at org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:182) 

at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1343)

[#6]

@Felix Albani i am facing the issues while i am running the job in oozie . Below is the error

2018-08-21 14:27:16,632 ERROR [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
java.lang.IllegalArgumentException: Invalid ContainerId: container_e63_1532612094367_17160_02_000001
	at org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:182)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1343)