Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

ClassDefNotFound on joda in 5.5.2

ClassDefNotFound on joda in 5.5.2

New Contributor

Hi.

 

I'm seeing an issue that looks like:

https://community.cloudera.com/t5/Batch-Processing-and-Workflow/CDH-5-3-CLASSPATH-Issue/m-p/23901#M8...

 

On cloudera 5.5.2, I can't submit any job that has joda in any major capacity, for example, if I write this:

 

org.joda.time.format.DateTimeFormatter sdf = DateTimeFormat.forPattern("yyyy-MM-dd HH:mm:ss");

 

In a jar and submit it to a cluster I get this error:

Exception in thread "main" java.lang.NoClassDefFoundError: org/joda/time/format/DateTimeFormat
    -----------(My code line)----------------
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    -----------(My code line)----------------
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.lang.ClassNotFoundException: org.joda.time.format.DateTimeFormat
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 9 more

 

Including or excluding joda from the jar submitted makes no difference at all. It wouldn't be a big issue if I didn't need to use other libraries that use joda-time. Like AWS SDK

 

My guess is that, since cloudera itself has aws-sdk jars, but doesn't have joda jars. Joda is probably embeded into another jar with bits missing. I have worked around the issue in my project. But that's something you might want to look into.

 

To clearify, I only have hadoop, yarn(both rm&nm) and hdfs(both dn&nn) installed on mint

3 REPLIES 3

Re: ClassDefNotFound on joda in 5.5.2

Super Collaborator

If you need a version of a library that is already part of hadoop I would strongly recommend that you include your version of the library in a shaded form in your application.

The shading will make sure that you get your version and that it will not interfere with existing versions.

 

We are currently writing a knowledge base article on how to do this and for know you will need to check online for "maven shade" or "gradle shadow" depending on how you build your application.

 

Wilfred

Re: ClassDefNotFound on joda in 5.5.2

New Contributor
Hi, thank you for the reply, but I don't think I can make AWS SDK use the
shaded version without recompiling it, which was the use case(correct me if
I'm wrong, I'm no expert in dependency management).

I couldn't get to AWS services because SDK was dependent on Joda. The basic
example I wrote was just a way to test the issue.

Re: ClassDefNotFound on joda in 5.5.2

Super Collaborator

You should not ,and can not, rely on the joda version that the AWS SDK brings in. If they use a shaded version than you can not reach it and you would not see it. If they have an unshaded version then you need to shade your version. You need to declare it as your dependency and then shade your version in your build.

 

It is not the simplest thing to figure out, specially if you have never done it before, but after sorting it once it should not cost you anything and probably make upgrades of hadoop and maintenance of your application simpler.

 

Wilfred