Reply
New Contributor
Posts: 1
Registered: ‎10-15-2015

Submitting Spark Job with Yarn Client

I am trying to figure out how to submit a spark job to a Yarn cluster from a stand alone java app. I am currently using the Yarn client java class, the job makes it to the cluster, it is put into an ACCEPTED state, but then it fails and I get this stack trace in the logs:

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/Logging
	at java.lang.ClassLoader.defineClass1(Native Method)
	at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
	at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
	at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
	at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:482)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.Logging
	at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
	... 13 more

I am running this class locally on my machine:

 

package main;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.spark.SparkConf;
import org.apache.spark.deploy.yarn.Client;
import org.apache.spark.deploy.yarn.ClientArguments;

import org.apache.spark.Logging;

public class SparkYarnTest {

    public static void main(String[] arguments) throws Exception {

        // prepare arguments to be passed to
        // org.apache.spark.deploy.yarn.Client object
        String[] args = new String[]{
                // the name of your application
                "--name",
                "Submitted by Java App",

                // memory for driver (optional)
                "--driver-memory",
                "1000M",

                // path to your application's JAR file
                // required in yarn-cluster mode
                "--jar",
                "${JAR_LOCATION}/spark-test.jar",

                // name of your application's main class (required)
                "--class",
                "main.SparkTest",

                // comma separated list of local jars that want
                // SparkContext.addJar to work with
                "--addJllars",
                "",

                // argument 1 to your Spark program (SparkFriendRecommendation)
                "--arg",
                "3",

                // argument 2 to your Spark program (SparkFriendRecommendation)
                "--arg",
                "/input",

                // argument 3 to your Spark program (SparkFriendRecommendation)
                "--arg",
                "/output",

                // argument 4 to your Spark program (SparkFriendRecommendation)
                // this is a helper argument to create a proper JavaSparkContext object
                // make sure that you create the following in SparkFriendRecommendation program
                // ctx = new JavaSparkContext("yarn-cluster", "SparkFriendRecommendation");
                "--arg",
                "yarn-cluster"
        };

        // create a Hadoop Configuration object
        Configuration config = createConfiguration();

        // identify that you will be using Spark as YARN mode
        System.setProperty("SPARK_YARN_MODE", "true");

        // create an instance of SparkConf object
        SparkConf sparkConf = new SparkConf();

        // create ClientArguments, which will be passed to Client
        ClientArguments cArgs = new ClientArguments(args, sparkConf);

        // create an instance of yarn Client client
        Client client = new Client(cArgs, config, sparkConf);

        // submit Spark job to YARN
        client.run();
    }


    private static Configuration createConfiguration() {
        Configuration configuration = new Configuration();
        configuration.addResource(new Path("${HADOOP_CONF}/cdh5.4.7-conf/yarn-conf/core-site.xml"));
        configuration.addResource(new Path("${HADOOP_CONF}/cdh5.4.7-conf/yarn-conf/yarn-site.xml"));
        return configuration;
    }
}

Wasn't sure if anyone else had encountered something similar or could point me in the correct direction. I can run the spark job on the cluster using the "spark-submit" command, but when it comes to running it with this class it doesn't work. Thanks in advance.

New Contributor
Posts: 1
Registered: ‎11-26-2015

Re: Submitting Spark Job with Yarn Client

HI 

 

Did you solve this issue?

 

Thanx

Posts: 1,826
Kudos: 406
Solutions: 292
Registered: ‎07-31-2013

Re: Submitting Spark Job with Yarn Client

Announcements