question Re: yarn logs of hive on spark requests are not accessible in Support Questions

yarn logs of hive on spark requests are not accessible

Swagat — Tue, 05 Oct 2021 02:49:22 GMT

 

Examples :

 

[ledapp3@leda-edge-u1-n01 ~]$ yarn logs -applicationId application_1616602302580_19604 >crash.log

WARNING: YARN_OPTS has been replaced by HADOOP_OPTS. Using value of YARN_OPTS.

Permission denied: user=ledapp3, access=READ_EXECUTE, inode="/tmp/logs/hive/logs/application_1616602302580_19604":hive:hadoop:drwxrwx---

 

[calapr01@callisto-edge-u2-n01 ~]$ yarn logs -applicationId application_1623858219829_386030 | head

WARNING: YARN_OPTS has been replaced by HADOOP_OPTS. Using value of YARN_OPTS.

Permission denied: user=calapr01, access=READ_EXECUTE, inode="/tmp/logs/hive/logs/application_1623858219829_386030":hive:hadoop:drwxrwx---

 

[gaiapr01@ganymede-edge-u2-n01 ~]$ yarn logs --applicationId application_1631641196769_7918 | head

WARNING: YARN_OPTS has been replaced by HADOOP_OPTS. Using value of YARN_OPTS.

21/10/01 15:23:44 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm249

Permission denied: user=gaiapr01, access=EXECUTE, inode="/tmp/logs/hive":hive:hadoop:drwxrwx--

 

[gaiapr01@io-edge-u1-n01 ~]$yarn logs --applicationId=application_1626699923211_6165

WARNING: YARN_OPTS has been replaced by HADOOP_OPTS. Using value of YARN_OPTS.

21/10/01 15:27:05 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm204

Permission denied: user=gaiapr01, access=EXECUTE, inode="/tmp/logs/hive":hive:hadoop:drwxrwx--

 

The problem seems to be that it is the user hive that creates such log files, and so the applicative accounts that launch the jobs cannot access it, and so we can not access it.

We need the applicative accounts to have Read permissions on those log files, otherwise we cannot figure out why our jobs failed since we do not have access to logs.



We tried to check the ACL rights :

[smohanty@io-edge-u1-n01 ~]$ hdfs dfs -getfacl /tmp/logs/hive/logs/
# file: /tmp/logs/hive/logs
# owner: hive
# group: hadoop
user::rwx
group::rwx
group:dsi_moe_bigdata_prod:r-x
mask::rwx
other::---
default:user::rwx
default:group::rwx
default:group:dsi_moe_bigdata_prod:r-x
default:mask::rwx
default:other::---

[smohanty@io-edge-u1-n01 ~]$

[smohanty@callisto-edge-u2-n01 ~]$ hdfs dfs -getfacl /tmp/logs/hive/logs/
# file: /tmp/logs/hive/logs
# owner: hive
# group: hadoop
user::rwx
user:calapr00:rwx
user:calapr01:rwx
group::rwx
group:dsi_moe_dev_bigdata:r-x
mask::rwx
other::---
default:user::rwx
default:group::rwx
default:group:dsi_moe_dev_bigdata:r-x
default:mask::rwx
default:other::---

[smohanty@callisto-edge-u2-n01 ~]$

Is there any ACL modification needed

Re: yarn logs of hive on spark requests are not accessible

Shifu — Tue, 05 Oct 2021 13:06:56 GMT

Hi @Swagat

(1). Can you run the below command in your Linux terminal and re-run the "yarn logs -applicationID <appID>" command
export HADOOP_USER_NAME=hdfs

(2). Based on the logs parent folder "/tmp/logs/hive" and its subfolders are with 770 permission/hive:hadoop so the other users do not have permission to access it. The users ledapp3, calapr01, and gaiapr01 will fall under others so they are not able to access it.

Can you run below command as HDFS user and ask your end users to try again.
hdfs dfs -chmod 775 <parent folder>
hdfs dfs -chmod -R 775 <parent folder>
hdfs dfs -chmod -R 777 <parent folder> ###Least Recommended.

Similar Issue Reference:

https://community.cloudera.com/t5/Support-Questions/Permission-denied-user-root-access-WRITE-inode-quot-user/td-p/4943

https://community.cloudera.com/t5/Support-Questions/Permission-denied-as-I-am-unable-to-delete-a-directory-in/m-p/322469#M228801

If you are happy with the reply, mark it Accept as Solution

Re: yarn logs of hive on spark requests are not accessible

VidyaSargur — Mon, 11 Oct 2021 05:37:13 GMT

@Swagat, Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future.

Re: yarn logs of hive on spark requests are not accessible

Swagat — Mon, 11 Oct 2021 05:48:32 GMT

Hello , I applied below solution to do the fix.

hdfs dfs -setfacl -R -m group:calapr01:r-x /tmp/logs/*

hdfs dfs -setfacl -m default:group:calapr01:r-x /tmp/logs/hive/logs

Seems I found this link helpful :

https://my.cloudera.com/knowledge/ERROR-quotPermission-denied-userltusernamegt-accessREADEXECUTE?id=293648