Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Pig on Tez fails due to Permission denied on staging directory

Highlighted

Pig on Tez fails due to Permission denied on staging directory

New Contributor

job.txt

No staging directory work correctly after setting for these two paths:

DEFAULT

/tmp/${user.name}/staging

NEW

/user/${user.name}/.staging

${user.name} is replaced with "yarn" which is not a current user who is executing a Pig Job.

In my case ${user.name} should be replaced with "training" user to get correct permission

ERROR org.apache.pig.backend.hadoop.executionengine.tez.TezSessionManager - Exception while waiting for Tez client to be ready 
org.apache.tez.dag.api.TezUncheckedException: org.apache.hadoop.security.AccessControlException: Permission denied: user=training, access=WRITE, inode="/user/yarn/.staging":hdfs:hdfs:drwxr-xr-x
7 REPLIES 7

Re: Pig on Tez fails due to Permission denied on staging directory

New Contributor

Hi, I need a solution for this problem as well.
@Prashant Band, did you manage to solve this problem??

Re: Pig on Tez fails due to Permission denied on staging directory

New Contributor

Not yet, but I have written shell script which clears the staging directory created by any user. It is not a complete solution but my multiple users can now run Jobs without need of admin to clear staging directory every time.

Re: Pig on Tez fails due to Permission denied on staging directory

New Contributor

Hi, any news on this?

I have a similar situation when running a Pig script and working in a kerberized environment - the OS user obtaining the Kerb keytab is put into the Tez property /tmp/${user.name}/staging (e.g. /tmp/myosuser/staging), but obviously Pig is trying to get HDFS access with the user contained in the keytab (e.g. pighdfsuser). Then I am getting:

org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=pighdfsuser, access=EXECUTE, inode="/tmp/myosuser/staging":hdfs:hdfs...

I can not change the OS user because only this one may obtain Kerberos keytabs due to security settings.

What is the proper config to solve this, any suggestions?

Re: Pig on Tez fails due to Permission denied on staging directory

New Contributor

Colleague from HWX pointed out that the /tmp/myosuser was already existing with ownership by hdfs:hdfs. Hence Tez continued using this ownership.

Problem mitigated by deleting the directory, rerunning the Pig script creates the directory again with ownership by pighdfsuser.

Re: Pig on Tez fails due to Permission denied on staging directory

Contributor

@Prashant Band @Robert Fidler @William Ardianto

I assume you are running on a secure cluster (Kerberos is active)? right?

  • Cleanly (let your admin) delete the /tmp/${user.name} directory.
  • Make sure that your Kerberos principal you are using to run the jobs matches the OS user!
    • myusername@mymachine $ kinit myusername
  • Make sure the OS user exists on all (worker) nodes of the cluster.
  • It's bad practice to submit a job as OS user that is different to the Kerberos principal used to authenticate at the service.

Re: Pig on Tez fails due to Permission denied on staging directory

New Contributor

Kerberos is not active, but problem is in OS user I guess.

Re: Pig on Tez fails due to Permission denied on staging directory

New Contributor

@All

I used temporary solution by creating cron job, no admin require everytime to remove directory:

File: rmstaging.sh

su - hdfs -c 'hdfs dfs -ls /tmp/yarn' > /dev/null 2>&1
if [ $? == 0 ]
then
su - hdfs -c 'hdfs dfs -rmdir /tmp/yarn/staging/.tez' > /dev/null 2>&1
su - hdfs -c 'hdfs dfs -rmdir /tmp/yarn/staging' > /dev/null 2>&1
su - hdfs -c 'hdfs dfs -rmdir /tmp/yarn' > /dev/null 2>&1
fi