Support Questions
Find answers, ask questions, and share your expertise

Eclipse to Sandbox

Re: Eclipse to Sandbox

Contributor

@Rajkumar Singh, actually that's what I need to do , I need to run spark main class as java class, do you have a link explain how to do that?

Re: Eclipse to Sandbox

@emad rifai

I think you can not do that until you have your custom spark-submit-runner like this

https://github.com/sequenceiq/sequenceiq-samples/tree/master/spark-submit

Re: Eclipse to Sandbox

@emad rifai, If you want to submit the jobs directly via eclipse you need to set the below configs.

SparkConf conf = new SparkConf().setAppName("sparkForSMA").set("spark.master", "yarn-client")            .set("spark.local.ip","<sandbox-ip>").set("spark.driver.host","localhost");

and if you want to submit a standalone app you may need to go with spark submit option which @Rajkumar Singh has shared.

Re: Eclipse to Sandbox

Contributor
@Sandeep Nemuri

thank you for your answer , from where you get the spark.local.ip?

Re: Eclipse to Sandbox

@emad rifai That should be your sandbox ip.

Re: Eclipse to Sandbox

Contributor

@Sandeep Nemuri

ok thank you now it's work but i got this error ,

org.apache.hadoop.security.AccessControlException: Permission denied: user=A62, access=WRITE, inode="/user/A62/.sparkStaging/application_1464688052729_0002":hdfs:hdfs:drwxr-xr-x

Re: Eclipse to Sandbox

@emad rifai

The user you are running with doesn't have permissions on hdfs.

Try setting System.setProperty("HADOOP_USER_NAME", "hdfs");

Re: Eclipse to Sandbox

Contributor

@Sandeep Nemuri thank you for all your answers,

Do you know what cause the following error ?

org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master.

Re: Eclipse to Sandbox

@emad rifai

Check the logs of the application by running :

yarn logs -applicationId <application ID>

and also try setting conf.setSparkHome("<path to spark home>");

Re: Eclipse to Sandbox

Contributor

@Sandeep Nemuri

when I try to check the logs by the command yarn logs -applicationId i get

16/05/31 13:36:47 INFO impl.TimelineClientImpl: Timeline service address: http://sandbox.hortonworks.com:8188/ws/v1/timeline/
16/05/31 13:36:48 INFO client.RMProxy: Connecting to ResourceManager at sandbox.hortonworks.com/10.24.244.31:8050
/app-logs/root/logs/application_1464699667428_0001does not exist.
Log aggregation has not completed or is not enabled.