Support Questions
Find answers, ask questions, and share your expertise

problem in adding library to spark java code

problem in adding library to spark java code

Contributor

hello,

I want to write my code in spark 1.5.2 environment. since I will need graph methods a lot, I want to add a graph library name "JgraphT" to my project. I'm using maven to package my java project.

I tried these steps but I faced an exception. any idea what's going wrong here???

a sample code I try to run including "jgraphT":

import java.net.*;

import org.jgrapht.**;
import org.jgrapht.graph.**;

/***
 ** A simple introduction to using JGraphT.
 **
 ** @author Barak Naveh
 ** @since Jul 27, 2003
 **/
public final class HelloJGraphT
{

    //~ Constructors ———————————————————

    private HelloJGraphT()
    {
    } // ensure non-instantiability.

    //~ Methods ———————————————————

    /***
     ** The starting point for the demo.
     **
     ** @param args ignored.
     */
    public static void main(String [] args)
    {
        UndirectedGraph<String, DefaultEdge> stringGraph = createStringGraph();

        // note undirected edges are printed as: {<v1>,<v2>}
        System.out.println(stringGraph.toString());

        // create a graph based on URL objects
        DirectedGraph<URL, DefaultEdge> hrefGraph = createHrefGraph();

        // note directed edges are printed as: (<v1>,<v2>)
        System.out.println(hrefGraph.toString());
    }

    /***
     ** Creates a toy directed graph based on URL objects that represents link
     ** structure.
     **
     ** @return a graph based on URL objects.
     **/
    private static DirectedGraph<URL, DefaultEdge> createHrefGraph()
    {
        DirectedGraph<URL, DefaultEdge> g =
            new DefaultDirectedGraph<URL, DefaultEdge>(DefaultEdge.class);

        try {
            URL amazon = new URL("[http://www.amazon.com](http://www.amazon.com)");
            URL yahoo = new URL("[http://www.yahoo.com](http://www.yahoo.com)");
            URL ebay = new URL("[http://www.ebay.com](http://www.ebay.com)");

            // add the vertices
            g.addVertex(amazon);
            g.addVertex(yahoo);
            g.addVertex(ebay);

            // add edges to create linking structure
            g.addEdge(yahoo, amazon);
            g.addEdge(yahoo, ebay);
        } catch (MalformedURLException e) {
            e.printStackTrace();
        }

        return g;
    }

    /***
     ** Craete a toy graph based on String objects.
     **
     ** @return a graph based on String objects.
     */
    private static UndirectedGraph<String, DefaultEdge> createStringGraph()
    {
        UndirectedGraph<String, DefaultEdge> g =
            new SimpleGraph<String, DefaultEdge>(DefaultEdge.class);

        String v1 = "v1";
        String v2 = "v2";
        String v3 = "v3";
        String v4 = "v4";

        // add the vertices
        g.addVertex(v1);
        g.addVertex(v2);
        g.addVertex(v3);
        g.addVertex(v4);

        // add edges to create a circuit
        g.addEdge(v1, v2);
        g.addEdge(v2, v3);
        g.addEdge(v3, v4);
        g.addEdge(v4, v1);

        return g;
    }
}

my pom.xml: (the built is done successfully and I added dependency from this link.)

<project>
  <groupId>edu.berkeley</groupId>
  <artifactId>simple-project</artifactId>
  <modelVersion>4.0.0</modelVersion>
  <name>Simple Project</name>
  <packaging>jar</packaging>
  <version>1.0</version>
  <dependencies>
    <dependency> <!-- Spark dependency -->
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-core_2.10</artifactId>
      <version>1.5.2</version>
    </dependency>

    <dependency>
      <groupId>org.jgrapht</groupId>
      <artifactId>jgrapht-core</artifactId>
      <version>1.0.0</version>
    </dependency>

  </dependencies>
</project>

and the exception I get:

Exception in thread "main" java.lang.NoClassDefFoundError: org/jgrapht/UndirectedGraph
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:348)
    at org.apache.spark.util.Utils$.classForName(Utils.scala:173)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:677)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.jgrapht.UndirectedGraph
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 8 more
2 REPLIES 2

Re: problem in adding library to spark java code

In the pom file the dependency is necessary to compile.

For execution (especially in distributed environment), the runtime needs to know about the library too. Try --packages like

mvn package
spark-submit --packages "org.jgrapht:jgrapht-core:1.0.0" \
             --class HelloJGraphT target/simple-project-1.0.jar

Re: problem in adding library to spark java code

New Contributor

you may google it, for a sign in assistance visit blog.