Reply
Highlighted
New Contributor
Posts: 4
Registered: ‎04-08-2019

CDH6.1.1 oozie spark-workflow read json bug

[ Edited ]

when i use oozie spark-workflow on CDH6.1.1,i try to read the json file in my spark code,i got the error like these:

Failing Oozie Launcher, Illegal pattern component: XXX
java.lang.IllegalArgumentException: Illegal pattern component: XXX
 at org.apache.commons.lang3.time.FastDatePrinter.parsePattern(FastDatePrinter.java:282)
 at org.apache.commons.lang3.time.FastDatePrinter.init(FastDatePrinter.java:149)
 at org.apache.commons.lang3.time.FastDatePrinter.<init>(FastDatePrinter.java:142)
 at org.apache.commons.lang3.time.FastDateFormat.<init>(FastDateFormat.java:384)
------------------------------------------------------
Lost task 0.0 in stage 0.0 (TID 0, bigdata166, executor 1): java.io.InvalidClassException: org.apache.commons.lang3.time.FastDateParser; local class incompatible: stream classdesc serialVersionUID = 3, local class serialVersionUID = 2

i rename jar to commons-lang3-3.7-oozie.jar on hdfs-ooziesharlib, then i restart the oozie service , now the workflow runs well but sometimes still got the same error.

Cloudera Employee
Posts: 776
Registered: ‎03-23-2015

Re: CDH6.1.1 oozie spark-workflow read json bug

Hi ,

The error message indicates that there is an incompatibility between class versions, where the expected and actual found org.apache.commons.lang3 class versions are different.

In your case, the expected version 3 is higher than the one that is found 2.

Can you please confirm if you have multiple versions of Spark jars installed in your Oozie workflow's lib directory or under sharelib? Please list out all the JARs you have and see if there is any duplicates of same Jars from different versions.

Cheers
Eric
New Contributor
Posts: 4
Registered: ‎04-08-2019

Re: CDH6.1.1 oozie spark-workflow read json bug

[ Edited ]

thanks for your reply,i replaced all commons-lang3-*.jar to commons-lang3-3.7.jar on my CDHcluster , but it still got the same error:

 java.io.InvalidClassException: org.apache.commons.lang3.time.FastDateParser; local class incompatible: stream classdesc serialVersionUID = 3, local class serialVersionUID = 2

 

So i guess there was a commons-lang3-3.7.jar of different version was included in some jars that i dont know.

 

But now the problem solved,i package my sparkcode with commons-lang3 dependence,it works,the conflict has been avoided.

Cloudera Employee
Posts: 776
Registered: ‎03-23-2015

Re: CDH6.1.1 oozie spark-workflow read json bug

Good to know that you have resolved it.

However, I think org.apache.commons.lang3 has been shipped with CDH, that's probably the reason. Can you try to remove this package to see if it can also resolve? Unless you application depends that particular version other than the one in CDH.

Cheers
Eric
Announcements