question Re: Can't run spark-submit to yarn cluster in Archives of Support Questions (Read Only)

Can't run spark-submit to yarn cluster

cokorda_putra — Mon, 14 Dec 2015 19:56:36 GMT

I have a problem while try to run spark-submit to yarn-cluster

below is my spark-submit code

spark-submit --master yarn-cluster --name spark_ml user_recommendation.py

getting below error :

java.io.FileNotFoundException: File does not exist: hdfs://name-node:8020/user/spark/.sparkStaging/application_1450092198211_0007/pyspark.zip

Is it configuration issue?

Thanks,

Coktra

Re: Can't run spark-submit to yarn cluster

nsabharwal — Mon, 14 Dec 2015 19:59:26 GMT

@cokorda putra susila

Please see this https://issues.apache.org/jira/browse/SPARK-10795

Re: Can't run spark-submit to yarn cluster

cokorda_putra — Mon, 14 Dec 2015 20:06:56 GMT

Hi Neeraj,

thanks for the link, so there is no solution yet for this problem?

Re: Can't run spark-submit to yarn cluster

nsabharwal — Mon, 14 Dec 2015 20:07:45 GMT

@cokorda putra susila No. You can update that jira and vote for it

Re: Can't run spark-submit to yarn cluster

cokorda_putra — Mon, 14 Dec 2015 22:07:21 GMT

@Neeraj Sabharwal thank you, i will vote to jira

Re: Can't run spark-submit to yarn cluster

nsabharwal — Mon, 14 Dec 2015 22:13:39 GMT

@cokorda putra susila I guess we can close this question for now. You can do it by accepting the jira response if you like.

Re: Can't run spark-submit to yarn cluster

aervits — Tue, 02 Feb 2016 10:16:15 GMT

@cokorda putra susila can you accept the best answer to close this thread or provide your own solution?

Re: Can't run spark-submit to yarn cluster

alunarbeach — Wed, 03 Feb 2016 03:50:53 GMT

I am submitting spark-submit jobs with python code and they are running fine in YARN Cluster mode. I would like to understand the question a bit further. Is your Cluster running Spark? Have you setup YARN_CONF_DIR variable?

Re: Can't run spark-submit to yarn cluster

jzhang — Thu, 04 Feb 2016 16:08:04 GMT

What's your version of HDP and spark ?

Re: Can't run spark-submit to yarn cluster

som — Thu, 27 Sep 2018 19:06:45 GMT

may be this will help - try your luck,

https://stackoverflow.com/questions/44231261/spark-yarn-file-does-not-exist-on-hdfs