Support Questions
Find answers, ask questions, and share your expertise

Eclipse to Sandbox

Eclipse to Sandbox

Contributor

Hello,

I know this question has been asked before but with no answers, so I am asking it again.

I am new to HDP and hadoop.I managed to install HDP 2.2 sandbox on Virtual box.

I tried a few sample programs and they are working fine from the sandbox.

I have installed Eclipse in my Windows machine.

At present ,I use Maven and package my application and deploy the jar in the HDP Sandbox for execution.

I would like to execute programs from my Eclipse against the HDP sandbox directly instead of packaging it each and every time.

A sample code which I am trying to modify

SparkConf conf = new SparkConf().setAppName("sparkApp").setMaster( "local[2]");

I guess , I have to change the local[2] to the master node / yarn cluster url.

How do I get the url from the sandbox ?

Any other configurations which has to be done on the Virtual box or on my code ?

37 REPLIES 37

Re: Eclipse to Sandbox

@emad rifai

If you running spark on yarn then you need to use below conf.

new SparkConf().setAppName("test-app").setMaster("yarn-client");

We also need to make sure we have all yarn client jars and below yarn confs in your eclipse project path which has cluster RM and other confs.

  • core-site.xml
  • hdfs-site.xml
  • yarn-site.xml

Re: Eclipse to Sandbox

Contributor

@Jitendra Yadav thank you for your answer, 1- should I call the files of yarn confs in my java code or just put them in eclipse project path?

2-this is my code   SparkConf conf = new SparkConf().setAppName("sparkForSMA")
                   .set("spark.master", "yarn-client")
                         //  .set("spark.driver.host", "10.24.246.183");
            // .setMaster("spark://sandbox:7077")
                    // .set("spark.driver.host","sandbox")
                     .set("spark.local.ip","127.0.0.1") 
                     .set("spark.driver.host","10.24.246.183");

But i get this error .

failed to bind to /10.24.246.183:0, shutting down Netty transport
Failed to bind to: /10.24.246.183:0: Service 'sparkDriver' failed after 16 retries!

Re: Eclipse to Sandbox

@emad rifai

Please keep spark conf like this, lets spark use default port for driver as of now. And its better to keep the confs file inside project lib path.

newSparkConf().setAppName("test-app").setMaster("yarn-client");

Also please share the full logs when you start spark application.

Re: Eclipse to Sandbox

Contributor

@Jitendra Yadav As I am using Maven in my project i put the conf files in ( resources) .

And i did change SparkConf as you suggested .

And the full logs now.

Exception in thread "main" java.lang.ExceptionInInitializerError
at org.apache.spark.util.Utils$.getSparkOrYarnConfig(Utils.scala:1784)
at org.apache.spark.storage.BlockManager.<init>(BlockManager.scala:105)
at org.apache.spark.storage.BlockManager.<init>(BlockManager.scala:180)
at org.apache.spark.SparkEnv$.create(SparkEnv.scala:292)
at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:159)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:232)
at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:61)
at com.worldline.bfi.labs.sma.test.SparTest.main(SparTest.java:24)
Caused by: org.apache.spark.SparkException: Unable to load YARN support
at org.apache.spark.deploy.SparkHadoopUtil$.liftedTree1$1(SparkHadoopUtil.scala:199)
at org.apache.spark.deploy.SparkHadoopUtil$.<init>(SparkHadoopUtil.scala:194)
at org.apache.spark.deploy.SparkHadoopUtil$.<clinit>(SparkHadoopUtil.scala)
... 8 more
Caused by: java.lang.ClassNotFoundException: org.apache.spark.deploy.yarn.YarnSparkHadoopUtil
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Unknown Source)
at org.apache.spark.deploy.SparkHadoopUtil$.liftedTree1$1(SparkHadoopUtil.scala:195)
... 10 more

Re: Eclipse to Sandbox

@emad rifai

Jar spark-assembly-<version>.jar is missing, please add and run it again.

Re: Eclipse to Sandbox

Contributor
@Jitendra Yadav

Done ,And it still giving almost the same error

Exception in thread "main" java.lang.ExceptionInInitializerError
at org.apache.spark.util.Utils$.getSparkOrYarnConfig(Utils.scala:2120)
at org.apache.spark.storage.BlockManager.<init>(BlockManager.scala:97)
at org.apache.spark.storage.BlockManager.<init>(BlockManager.scala:173)
at org.apache.spark.SparkEnv$.create(SparkEnv.scala:338)
at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:188)
at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:267)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:424)
at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:61)
at com.worldline.bfi.labs.sma.test.SparTest.main(SparTest.java:24)
Caused by: org.apache.spark.SparkException: Unable to load YARN support
at org.apache.spark.deploy.SparkHadoopUtil$.liftedTree1$1(SparkHadoopUtil.scala:350)
at org.apache.spark.deploy.SparkHadoopUtil$.<init>(SparkHadoopUtil.scala:345)
at org.apache.spark.deploy.SparkHadoopUtil$.<clinit>(SparkHadoopUtil.scala)
... 9 more
Caused by: java.lang.ClassNotFoundException: org.apache.spark.deploy.yarn.YarnSparkHadoopUtil
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Unknown Source)
at org.apache.spark.deploy.SparkHadoopUtil$.liftedTree1$1(SparkHadoopUtil.scala:346)
... 11 more

This is the jar's concerning Spark that i add in my project

<dependency>
<groupId> org.apache.spark</groupId>
<artifactId>spark-core_2.10</artifactId>
<version>1.4.1</version>
</dependency>


<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-yarn-client</artifactId>
<version>2.7.1</version>
</dependency>


<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-assembly_2.10</artifactId>
<version>1.1.1</version>
<type>pom</type>
</dependency>

Re: Eclipse to Sandbox

@emad rifai

Can you check manual if you got that dependency jar?

Also HDP spark mvn repro doesn't have 1.1.1 version for spark, please see link for available version artficats.

http://repo.hortonworks.com/content/repositories/releases/org/apache/spark/spark-assembly_2.10/

Re: Eclipse to Sandbox

Contributor

@Jitendra Yadav all the jar's are there, and I tried to change the version but same error,

thank you for you answers.

Re: Eclipse to Sandbox

@emad rifai

we still are missing jars in the classpath somehow, i would suggest to recreate the project and specify correct HDP maven repo. Here is a doc which might help you.

https://sparktutorials.github.io/2015/04/02/setting-up-a-spark-project-with-maven.html