Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Spark2 job in oozie

Highlighted

Spark2 job in oozie

I am trying to run a spark2 job in oozie and using HDP 2.6.0 and it fails with the following reason

java.lang.IllegalArgumentException: Attempt to add (hdfs://172.16.3.195/user/oozie/share/lib/lib_20170726131404/oozie/aws-java-sdk-core-1.10.6.jar) multiple times to the distributed cache.

and from whatever search i have made, the current oozie version supported in HDP 2.6 doesnt support spark2 job.
IS there a workaround to make it work?

Any thoughts would be great
1 REPLY 1

Re: Spark2 job in oozie

Explorer

you need remove duplicate jars from ../spark2/ directory

follow this: https://community.hortonworks.com/content/supportkb/150151/main-class-orgapacheoozieactionhadoopspar...

Don't have an account?
Coming from Hortonworks? Activate your account here