Can't run spark-submit to yarn cluster

[#1]

I have a problem while try to run spark-submit to yarn-cluster

below is my spark-submit code

spark-submit --master yarn-cluster --name spark_ml user_recommendation.py

getting below error :

java.io.FileNotFoundException: File does not exist: hdfs://name-node:8020/user/spark/.sparkStaging/application_1450092198211_0007/pyspark.zip

Is it configuration issue?

Thanks,

Coktra

[#2 · Accepted solution]

@cokorda putra susila

Please see this https://issues.apache.org/jira/browse/SPARK-10795

[#3]

Hi Neeraj,

thanks for the link, so there is no solution yet for this problem?

[#4]

@cokorda putra susila No. You can update that jira and vote for it

[#5]

@Neeraj Sabharwal thank you, i will vote to jira

[#6]

@cokorda putra susila I guess we can close this question for now. You can do it by accepting the jira response if you like.

[#7]

@cokorda putra susila can you accept the best answer to close this thread or provide your own solution?

[#8]

I am submitting spark-submit jobs with python code and they are running fine in YARN Cluster mode. I would like to understand the question a bit further. Is your Cluster running Spark? Have you setup YARN_CONF_DIR variable?

[#9]

What's your version of HDP and spark ?

[#10]

may be this will help - try your luck,

https://stackoverflow.com/questions/44231261/spark-yarn-file-does-not-exist-on-hdfs