question How to properly execute spark-submit command with Yarn? in Archives of Support Questions (Read Only)

How to properly execute spark-submit command with Yarn?

liana_napalkova — Fri, 04 May 2018 18:36:02 GMT

I should execute `spark-submit` in the Hadoop cluster created with Ambari. There are 3 instances: 1 master node and 2 executer nodes.

So, I logged in the master node as `centos` user and executed this command:

sudo -u hdfs spark-submit --master yarn --deploy-mode cluster --driver-memory 6g  --executor-memory 4g --executor-cores 2 --class org.tests.GraphProcessor graph.jar

But I got the error message that the file graph.jar does not exist. Therefore I tried to copy this file to HDFS as follows:

hdfs dfs -put graph.jar /home/hdfs/tmp

However, the error is:

No such file or directory: `hdfs://eureambarimaster1.local.eurecat.org:8020/home/hdfs/tmp'

Re: How to properly execute spark-submit command with Yarn?

falbani — Fri, 04 May 2018 19:48:37 GMT

@Liana Napalkova The graph.jar will be automatically copied to hdfs and distribute by the spark client. You only need to point to the location of graph.jar in the local file system. For example:

spark-submit --master yarn --deploy-mode cluster --driver-memory 6g--executor-memory 4g--executor-cores 2--class org.tests.GraphProcessor /path/to/graph.jar

HTH

*** If you found this answer addressed your question, please take a moment to login and click the "accept" link on the answer.

Re: How to properly execute spark-submit command with Yarn?

liana_napalkova — Fri, 04 May 2018 21:43:02 GMT

If I do this way (with "sudo -u hdfs"), the jar file is invisible for hdfs user (I get an error message)). But if I run without "sudo -u hdfs", then yarn mode cannot be entered. I think that it's the matter of permissions. But it's not clear to me how to solve this issue in a most correct way. Thanks.

Re: How to properly execute spark-submit command with Yarn?

liana_napalkova — Fri, 04 May 2018 23:56:11 GMT

In particular, if I do this way, I get the following error: Exception in thread "main" org.apache.hadoop.security.AccessControlException: Permission denied: user=centos, access=WRITE, inode="/user/centos/.sparkStaging/application_1523903913760_0007":hdfs:hdfs:drwxr-xr-x

Re: How to properly execute spark-submit command with Yarn?

liana_napalkova — Sat, 05 May 2018 00:01:30 GMT

I had to run this command to adjust permissions:

sudo -u hdfs hdfs dfs -chown centos:centos /user

After this I was able to run:

spark-submit --master yarn --deploy-mode cluster --driver-memory 6g--executor-memory 4g--executor-cores 2--class org.tests.GraphProcessor /path/to/graph.jar

Re: How to properly execute spark-submit command with Yarn?

falbani — Sat, 05 May 2018 00:10:39 GMT

@Liana Napalkova I advice against changing ownership of hdfs /usr directory

You should set correct permissions for /user/centos directory.

hdfs dfs -chown centos:centos /user/centos

HTH

Re: How to properly execute spark-submit command with Yarn?

falbani — Sat, 05 May 2018 00:15:15 GMT

@Liana Napalkova

You should set correct permissions for /user/centos directory.

hdfs dfs -chown centos:centos /user/centos

If you found this answer addressed your question, please take a moment to login and click the "accept" link on the answer.