Support Questions
Find answers, ask questions, and share your expertise

Eclipse to Sandbox

Re: Eclipse to Sandbox

Contributor

@Sandeep Nemuri does the following mean any thing to you?

[2016-05-31 16:15:37,445][INFO] Application report for application_1464703415943_0002 (state: ACCEPTED)
[2016-05-31 16:15:37,453][DEBUG] 
 client token: N/A
 diagnostics: N/A
 ApplicationMaster host: N/A
 ApplicationMaster RPC port: -1
 queue: default
 start time: 1464704136551
 final status: UNDEFINED
 tracking URL: http://sandbox.hortonworks.com:8088/proxy/application_1464703415943_0002/

Re: Eclipse to Sandbox

@emad rifai, Can you attach your class and pom files?

Re: Eclipse to Sandbox

Contributor

eclipse-files.zip @Sandeep Nemuri

please find the pom.xml and java classes in the attatchment.

and when i do netstat -at | grep 7077 in the virtual machine it return nothing.

Re: Eclipse to Sandbox

Sorry for the delay . Please add the hadoop-yarn-client dependency

<artifactId>hadoop-yarn-client</artifactId>

Re: Eclipse to Sandbox

Explorer

Hello @Sandeep Nemuri

I have the similar case as Emad. I followed your recommendations but still have the issues with remote vm.

Here is my code snippet

SparkConf conf = new SparkConf();
        conf.set("spark.master", "yarn-client");
        conf.set("spark.local.ip","192.168.144.133") ;
        conf.set("spark.driver.host","localhost");
        conf.set("spark.sql.hive.metastore.jars", "builtin");
        conf.setAppName("Test Analyzer");
        this.sparkSession = SparkSession.builder().config(conf).getOrCreate();

In pom.xml I have spark-sql and spark-yarn dependencies:

<dependency>
       <groupId>org.apache.spark</groupId>
       <artifactId>spark-sql_2.10</artifactId>
       <version>2.0.1</version>
       <scope>provided</scope>
     </dependency>
     <dependency>
       <groupId>org.apache.spark</groupId>
       <artifactId>spark-yarn_2.10</artifactId>
       <version>2.0.1</version>
       <scope>provided</scope>
     </dependency>

I put core-site.xml, yarn-site.xml, hdfs-site.xml into src/main/resources folder.

But when running application I got next exception:

java.io.IOException: HADOOP_HOME or hadoop.home.dir are not set. at org.apache.hadoop.util.Shell.checkHadoopHome(Shell.java:225) at org.apache.hadoop.util.Shell.<clinit>(Shell.java:250) at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:76) at org.apache.hadoop.yarn.conf.YarnConfiguration.<clinit>(YarnConfiguration.java:345)

If I remove Hadoop config files application works the same. It seems like it ignores config files.

Do you have some ideas how to deal with this? Could you please share exact steps how to set up eclipse and maven dependencies to make Spark job working from Eclipse?

Thank you for your time,

Olga

Re: Eclipse to Sandbox

Contributor

Jitendra, Good morning, I am trying to setup elcipse to execute my scala code against hatonworks sandbox and based on the post, you are able to connect but not all the steps are clear to me. Can you please share the steps. Thank you for your help.

Thanks

Ram

Re: Eclipse to Sandbox

Contributor

Hi, I was able to make progress in setting up the environment to execute Scala code against sandbox but I am getting the following error

16/07/02 15:28:20 INFO SparkEnv: Registering BlockManagerMaster 16/07/02 15:28:20 INFO DiskBlockManager: Created local directory at C:\Users\rampratapa\AppData\Local\Temp\blockmgr-a7a4881f-ae67-45f0-ae8f-827e7fcd71b5 16/07/02 15:28:20 INFO MemoryStore: MemoryStore started with capacity 5.1 GB 16/07/02 15:28:20 INFO SparkEnv: Registering OutputCommitCoordinator 16/07/02 15:28:20 INFO Utils: Successfully started service 'SparkUI' on port 4040. 16/07/02 15:28:20 INFO SparkUI: Started SparkUI at http://10.101.121.49:4040 16/07/02 15:28:20 INFO RMProxy: Connecting to ResourceManager at /0.0.0.0:8032 16/07/02 15:28:22 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 16/07/02 15:28:24 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 16/07/02 15:28:26 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 16/07/02 15:28:28 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 16/07/02 15:28:30 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 16/07/02 15:28:32 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 5 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)

can any one help.

Thanks

Ram

Re: Eclipse to Sandbox

Explorer

Hello @emad rifai

Could you please share how did you resolve issue with

org.apache.spark.SparkException: Yarn application has already ended!

Thanks in advance,

Olga