Perfect - Hadoop home was pointing to the wrong place, that was being picked up.  I am able to submit applications just fine now.  Thanks. ... View more

I am compiling the jar again the Hadoop 0.20.2 dependency and the Spark jar that is loaded on the cluster (the same point I am pointing the spark-class command to).   I do not think this particular error has anything to do with my compiled jar though, because when I run the spark-class while leaving off the jar argument, it fails with the same error (ie, looks like it does not even get to parsing the arguments).    I executed that with the debug command you supplied, here is what I got: 10:32 AM ~/lib/spark-0.9.0-incubating/bin: export SPARK_PRINT_LAUNCH_COMMAND=1 10:34 AM ~/lib/spark-0.9.0-incubating/bin: $SPARK_HOME/bin/spark-class org.apache.spark.deploy.yarn.Client Spark Command: java -cp :/opt/cloudera/parcels/CDH-5.0.1-1.cdh5.0.1.p0.47/lib/spark/conf:/opt/cloudera/parcels/CDH-5.0.1-1.cdh5.0.1.p0.47/lib/spark/assembly/lib/*:/opt/cloudera/parcels/CDH-5.0.1-1.cdh5.0.1.p0.47/lib/spark/examples/lib/*:/etc/hadoop/conf:/home/tclay/tools/hadoop-0.20.2/*:/home/tclay/tools/hadoop-0.20.2/../hadoop-hdfs/*:/home/tclay/tools/hadoop-0.20.2/../hadoop-yarn/*:/home/tclay/tools/hadoop-0.20.2/../hadoop-mapreduce/*:/opt/cloudera/parcels/CDH-5.0.1-1.cdh5.0.1.p0.47/lib/spark/lib/scala-library.jar:/opt/cloudera/parcels/CDH-5.0.1-1.cdh5.0.1.p0.47/lib/spark/lib/scala-compiler.jar:/opt/cloudera/parcels/CDH-5.0.1-1.cdh5.0.1.p0.47/lib/spark/lib/jline.jar -Djava.library.path=/opt/cloudera/parcels/CDH-5.0.1-1.cdh5.0.1.p0.47/lib/spark/lib:/home/tclay/tools/hadoop-0.20.2/lib/native -Xms512m -Xmx512m org.apache.spark.deploy.yarn.Client Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/yarn/client/api/impl/YarnClientImpl     at java.lang.ClassLoader.defineClass1(Native Method)     at java.lang.ClassLoader.defineClass(ClassLoader.java:800)     at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)     at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)     at java.net.URLClassLoader.access$100(URLClassLoader.java:71)     at java.net.URLClassLoader$1.run(URLClassLoader.java:361)     at java.net.URLClassLoader$1.run(URLClassLoader.java:355)     at java.security.AccessController.doPrivileged(Native Method)     at java.net.URLClassLoader.findClass(URLClassLoader.java:354)     at java.lang.ClassLoader.loadClass(ClassLoader.java:425)     at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)     at java.lang.ClassLoader.loadClass(ClassLoader.java:358)     at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:482) Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.yarn.client.api.impl.YarnClientImpl     at java.net.URLClassLoader$1.run(URLClassLoader.java:366)     at java.net.URLClassLoader$1.run(URLClassLoader.java:355)     at java.security.AccessController.doPrivileged(Native Method)     at java.net.URLClassLoader.findClass(URLClassLoader.java:354)     at java.lang.ClassLoader.loadClass(ClassLoader.java:425)     at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)     at java.lang.ClassLoader.loadClass(ClassLoader.java:358)   ... View more

Yep, source spark-env and set the SPARK_JAR environment variable as the instructions suggested (and I used the same jar during the compilation of my jar), ... View more

Here is the command I am running:   $SPARK_HOME/bin/spark-class org.apache.spark.deploy.yarn.Client --jar ~/samples/syn-spark-project-0.0.1-SNAPSHOT.jar --class spark.WorkCountJob --args yarn-standalone --args input/hello.txt --args output.txt My cluster is setup with Yarn, and the syn-spark-project is a jar that I assembled using the jar stored in the $SPARK_HOME directory on my cluster.   I am looking at the "Running Spark Applications" entry in the documentation. ... View more