Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Got permission denied when running pyspark --master yarn-client

avatar
Contributor

Hi,

i got this error below

Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, 
most recent failure: Lost task 0.3 in stage 0.0 (TID 3, 
ip-internal): java.io.IOException:
 Cannot run program 
"/home/centos/apps/pyspark/venv/bin/python": error=13,
 Permission denied

while running

$ export HADOOP_USER_NAME=centos
$ export PYSPARK_PYTHON=/home/centos/apps/pyspark/venv/bin/python
$ pyspark --master yarn-client

I add user centos to hadoop as hdfs group

i'm using latest HDP 2.3.4 and centos 7.2

1 ACCEPTED SOLUTION

avatar
Contributor
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login
6 REPLIES 6

avatar
Expert Contributor

What are the permissions on /home/centos/apps/pyspark/venv/bin/python?

avatar
Contributor

hi @Ian Roberts

drwxrwxrwx 4 centos centos 57 Feb 23 13:22 pyspark

this is the permission for pyspark venv "777"

and i install the pyspark venv on every node manager with the same permission.

avatar
Expert Contributor

What are the permissions on python

ls -ltr /home/centos/apps/pyspark/venv/bin

avatar
Contributor

here's the permission for the python @Ian Roberts

-rwxrwxrwx 1 centos centos 7136 Feb 23 13:22 python

avatar
Contributor
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login

avatar
Expert Contributor

For my case, previously I had given my anaconda installation.

Reverting it back to /usr/bin/python2.7 fixed the problem.