question Re: Got permission denied when running pyspark --master yarn-client in Archives of Support Questions (Read Only)

Got permission denied when running pyspark --master yarn-client

cokorda_putra — Tue, 23 Feb 2016 22:03:06 GMT

Hi,

i got this error below

Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, 
most recent failure: Lost task 0.3 in stage 0.0 (TID 3, 
ip-internal): java.io.IOException:
 Cannot run program 
"/home/centos/apps/pyspark/venv/bin/python": error=13,
 Permission denied

while running

$ export HADOOP_USER_NAME=centos
$ export PYSPARK_PYTHON=/home/centos/apps/pyspark/venv/bin/python
$ pyspark --master yarn-client

I add user centos to hadoop as hdfs group

i'm using latest HDP 2.3.4 and centos 7.2

Re: Got permission denied when running pyspark --master yarn-client

iroberts — Tue, 23 Feb 2016 22:04:47 GMT

What are the permissions on /home/centos/apps/pyspark/venv/bin/python?

Re: Got permission denied when running pyspark --master yarn-client

cokorda_putra — Tue, 23 Feb 2016 22:10:18 GMT

hi @Ian Roberts

drwxrwxrwx 4 centos centos 57 Feb 23 13:22 pyspark

this is the permission for pyspark venv "777"

and i install the pyspark venv on every node manager with the same permission.

Re: Got permission denied when running pyspark --master yarn-client

iroberts — Tue, 23 Feb 2016 22:22:49 GMT

What are the permissions on python

ls -ltr /home/centos/apps/pyspark/venv/bin

Re: Got permission denied when running pyspark --master yarn-client

cokorda_putra — Tue, 23 Feb 2016 23:49:23 GMT

here's the permission for the python @Ian Roberts

-rwxrwxrwx 1 centos centos 7136 Feb 23 13:22 python

Re: Got permission denied when running pyspark --master yarn-client

cokorda_putra — Wed, 24 Feb 2016 01:58:30 GMT

I can fix this issue by set the /home/centos executable "775" @Ian Roberts

Re: Got permission denied when running pyspark --master yarn-client

sedatkestepe — Fri, 19 Oct 2018 20:25:17 GMT

For my case, previously I had given my anaconda installation.

Reverting it back to /usr/bin/python2.7 fixed the problem.