Reply
New Contributor
Posts: 3
Registered: ‎02-19-2016

Spark Job as yarn-client fails: NoClassDefFoundError: org/apache/spark/Logging

Hi,

 

I'm have my notebook as a client + 1 master server (aaa) + 5 nodes.

On the master and the 5 nodes, Cloudera Manager with CDH 5.5.2 is running: HBase, HDFS, Spark, YARN, ZooKeeper.

 

On my notebook, I run a Java application that creates a SparkContext. In the Configuration, I have set the master to yarn-client. The environment variables YARN_CONF_DIR and HADOOP_CONF_DIR both show to /opt/yarn. In /opt/yarn on my notebook, I have copied the files from my cluster from /etc/hadoop/conf/* and /etc/hadoop/conf.cloudera.yarn/*

 

Now I start the Java application on my notebook and it submits the Spark job to aaa:

 

16/03/08 09:53:31 INFO impl.YarnClientImpl: Submitted application application_1457367239295_0002 to ResourceManager at aaa/192.168.18.4:8032
16/03/08 09:53:32 INFO yarn.Client: Application report for application_1457367239295_0002 (state: ACCEPTED)

16/03/08 09:53:33 INFO yarn.Client: Application report for application_1457367239295_0002 (state: FAILED)

 

On the Hadoop YARN ResourceManager-Web-UI, there is the application with status FAILED, two attempts, and in the log this:

 

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/Logging
	at java.lang.ClassLoader.defineClass1(Native Method)
	at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
	at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
	at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at org.apache.spark.deploy.yarn.ExecutorLauncher$.main(ApplicationMaster.scala:642)
	at org.apache.spark.deploy.yarn.ExecutorLauncher.main(ApplicationMaster.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.Logging
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

 

New Contributor
Posts: 3
Registered: ‎02-19-2016

Re: Spark Job as yarn-client fails: NoClassDefFoundError: org/apache/spark/Logging

Okay, not the best solution, but I added all jars to a folder in the YARN classpath:
ln -s /opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/jars/* /opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/lib/hadoop-yarn/lib