Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

spark job got java.lang.NoClassDefFoundError: org/joda/time/format/DateTimeFormat

spark job got java.lang.NoClassDefFoundError: org/joda/time/format/DateTimeFormat

New Contributor

Hi,

 

I am trying to run a spark job on CDH 5.3.3. (The job works properly in a standalone spark cluster. ) The job failed with the following error:

Exception in thread "main" java.lang.NoClassDefFoundError: org/joda/time/format/DateTimeFormat
at com.amazonaws.auth.AWS4Signer.<clinit>(AWS4Signer.java:44)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at java.lang.Class.newInstance(Class.java:374)
at com.amazonaws.auth.SignerFactory.createSigner(SignerFactory.java:119)
at com.amazonaws.auth.SignerFactory.lookupAndCreateSigner(SignerFactory.java:105)
at com.amazonaws.auth.SignerFactory.getSigner(SignerFactory.java:78)
at com.amazonaws.AmazonWebServiceClient.computeSignerByServiceRegion(AmazonWebServiceClient.java:307)
at com.amazonaws.AmazonWebServiceClient.computeSignerByURI(AmazonWebServiceClient.java:280)
at com.amazonaws.AmazonWebServiceClient.setEndpoint(AmazonWebServiceClient.java:160)
at com.amazonaws.services.sqs.AmazonSQSClient.init(AmazonSQSClient.java:265)
at com.amazonaws.services.sqs.AmazonSQSClient.<init>(AmazonSQSClient.java:244)
at com.amazonaws.services.sqs.AmazonSQSClient.<init>(AmazonSQSClient.java:219)
at com.tune.dataflow.aws.AWSUtils.createSQSClient(AWSUtils.java:383)
at com.tune.dataflow.aws.AWSUtils.sendSQSMessage(AWSUtils.java:403)
at costData.costData$$anonfun$main$2.apply(costData.scala:75)
at costData.costData$$anonfun$main$2.apply(costData.scala:74)
at scala.collection.immutable.List.foreach(List.scala:318)
at costData.costData$.main(costData.scala:74)
at costData.costData.main(costData.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:358)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.joda.time.format.DateTimeFormat
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 29 more

 

The spark submit command I am using is:

spark-submit --master yarn --class "costData.costData" --jars /ephemeral0/dataflow/lib/aws-java-sdk-1.9.16.jar,/ephemeral0/dataflow/lib/joda-time-2.3.jar --files /ephemeral0/dataflow/conf/log4j.properties /ephemeral0/dataflow/app/costData/target/scala-2.10/costdata_2.10-1.0.jar /ephemeral0/dataflow/app/costData/testing/cost_data_new.json

 

I added the joda-time jar to the --jars and it doesn't help.  The --files /ephemeral0/dataflow/conf/log4j.properties also doesn't have any impact on the logging behavior.  So it seems that the spark-submit through yarn doesn't pass all the information to the spark nodes.

 

Could you please help me out?

 

Thanks.

Don't have an account?
Coming from Hortonworks? Activate your account here