Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

CDH6.1.1 oozie spark-workflow read json bug

CDH6.1.1 oozie spark-workflow read json bug

New Contributor

when i use oozie spark-workflow on CDH6.1.1,i try to read the json file in my spark code,i got the error like these:

Failing Oozie Launcher, Illegal pattern component: XXX
java.lang.IllegalArgumentException: Illegal pattern component: XXX
 at org.apache.commons.lang3.time.FastDatePrinter.parsePattern(FastDatePrinter.java:282)
 at org.apache.commons.lang3.time.FastDatePrinter.init(FastDatePrinter.java:149)
 at org.apache.commons.lang3.time.FastDatePrinter.<init>(FastDatePrinter.java:142)
 at org.apache.commons.lang3.time.FastDateFormat.<init>(FastDateFormat.java:384)
------------------------------------------------------
Lost task 0.0 in stage 0.0 (TID 0, bigdata166, executor 1): java.io.InvalidClassException: org.apache.commons.lang3.time.FastDateParser; local class incompatible: stream classdesc serialVersionUID = 3, local class serialVersionUID = 2

i rename jar to commons-lang3-3.7-oozie.jar on hdfs-ooziesharlib, then i restart the oozie service , now the workflow runs well but sometimes still got the same error.

3 REPLIES 3

Re: CDH6.1.1 oozie spark-workflow read json bug

Guru
Hi ,

The error message indicates that there is an incompatibility between class versions, where the expected and actual found org.apache.commons.lang3 class versions are different.

In your case, the expected version 3 is higher than the one that is found 2.

Can you please confirm if you have multiple versions of Spark jars installed in your Oozie workflow's lib directory or under sharelib? Please list out all the JARs you have and see if there is any duplicates of same Jars from different versions.

Cheers
Eric

Re: CDH6.1.1 oozie spark-workflow read json bug

New Contributor

thanks for your reply,i replaced all commons-lang3-*.jar to commons-lang3-3.7.jar on my CDHcluster , but it still got the same error:

 java.io.InvalidClassException: org.apache.commons.lang3.time.FastDateParser; local class incompatible: stream classdesc serialVersionUID = 3, local class serialVersionUID = 2

 

So i guess there was a commons-lang3-3.7.jar of different version was included in some jars that i dont know.

 

But now the problem solved,i package my sparkcode with commons-lang3 dependence,it works,the conflict has been avoided.

Highlighted

Re: CDH6.1.1 oozie spark-workflow read json bug

Guru
Good to know that you have resolved it.

However, I think org.apache.commons.lang3 has been shipped with CDH, that's probably the reason. Can you try to remove this package to see if it can also resolve? Unless you application depends that particular version other than the one in CDH.

Cheers
Eric