Reply
New Contributor
Posts: 2
Registered: ‎02-22-2016

ClassDefNotFound on joda in 5.5.2

Hi.

 

I'm seeing an issue that looks like:

https://community.cloudera.com/t5/Batch-Processing-and-Workflow/CDH-5-3-CLASSPATH-Issue/m-p/23901#M8...

 

On cloudera 5.5.2, I can't submit any job that has joda in any major capacity, for example, if I write this:

 

org.joda.time.format.DateTimeFormatter sdf = DateTimeFormat.forPattern("yyyy-MM-dd HH:mm:ss");

 

In a jar and submit it to a cluster I get this error:

Exception in thread "main" java.lang.NoClassDefFoundError: org/joda/time/format/DateTimeFormat
    -----------(My code line)----------------
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    -----------(My code line)----------------
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.lang.ClassNotFoundException: org.joda.time.format.DateTimeFormat
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 9 more

 

Including or excluding joda from the jar submitted makes no difference at all. It wouldn't be a big issue if I didn't need to use other libraries that use joda-time. Like AWS SDK

 

My guess is that, since cloudera itself has aws-sdk jars, but doesn't have joda jars. Joda is probably embeded into another jar with bits missing. I have worked around the issue in my project. But that's something you might want to look into.

 

To clearify, I only have hadoop, yarn(both rm&nm) and hdfs(both dn&nn) installed on mint

Cloudera Employee
Posts: 322
Registered: ‎01-16-2014

Re: ClassDefNotFound on joda in 5.5.2

If you need a version of a library that is already part of hadoop I would strongly recommend that you include your version of the library in a shaded form in your application.

The shading will make sure that you get your version and that it will not interfere with existing versions.

 

We are currently writing a knowledge base article on how to do this and for know you will need to check online for "maven shade" or "gradle shadow" depending on how you build your application.

 

Wilfred

New Contributor
Posts: 2
Registered: ‎02-22-2016

Re: ClassDefNotFound on joda in 5.5.2

Hi, thank you for the reply, but I don't think I can make AWS SDK use the
shaded version without recompiling it, which was the use case(correct me if
I'm wrong, I'm no expert in dependency management).

I couldn't get to AWS services because SDK was dependent on Joda. The basic
example I wrote was just a way to test the issue.
Cloudera Employee
Posts: 322
Registered: ‎01-16-2014

Re: ClassDefNotFound on joda in 5.5.2

You should not ,and can not, rely on the joda version that the AWS SDK brings in. If they use a shaded version than you can not reach it and you would not see it. If they have an unshaded version then you need to shade your version. You need to declare it as your dependency and then shade your version in your build.

 

It is not the simplest thing to figure out, specially if you have never done it before, but after sorting it once it should not cost you anything and probably make upgrades of hadoop and maintenance of your application simpler.

 

Wilfred