question Re: Hive error on HDP6 in Archives of Support Questions (Read Only)

Hive error on HDP6

winnie_philip — Fri, 06 Oct 2017 03:36:18 GMT

I'm keep getting permission error while loading data to Hive tables and/or querying tables. I granted full(r+w+x) permission to appl/hadoop/yarn/local/usercache in all of my cluster nodes. I'm still getting errors. I have 3 nodes cluster (HDP6).

Application application_1507209932823_0008 failed 2 times due to AM Container for appattempt_1507209932823_0008_000002 exited with exitCode: -1000 For more detailed output, check the application tracking page: http://mymaster.fisdev.local:8088/cluster/app/application_1507209932823_0008 Then click on links to logs of each attempt. Diagnostics: Could not set permissions for local dir /appl/hadoop/yarn/local/usercache Failing this attempt. Failing the application.

Re: Hive error on HDP6

winnie_philip — Fri, 06 Oct 2017 04:20:03 GMT

I also noticed the permission I added keeps changing to original value for below folder. So i get inconsistent response.

[appadmin@mygrpdata1 yarn]$ ls -al /appl/hadoop/yarn/local/

total 0

drwxrwxrwx 5 yarn hadoop 57 Oct 5 16:13 .

drwxrwxrwx 4 root root 30 Aug 24 14:36 ..

drwxr-xr-x 3 yarn hadoop 16 Aug 30 13:46 filecache

drwx------ 3 yarn hadoop 44 Oct 5 16:13 nmPrivate

drwxr-xr-x 5 yarn hadoop 57 Oct 5 09:24 usercache

Re: Hive error on HDP6

bandarusridhar1 — Fri, 06 Oct 2017 12:20:47 GMT

@Winnie Philip

Check the umask permission on your Linux machines. Even you can remove all the info under /appl/hadoop/yarn/local/usercache/* on your each node where you have resource manager so that it will recreate the file with new permissions.

Hope this helps you.

Re: Hive error on HDP6

Shelton — Fri, 06 Oct 2017 14:03:07 GMT

@Winnie Philip

Can you attach the RM logs ?

Re: Hive error on HDP6

winnie_philip — Fri, 06 Oct 2017 22:37:53 GMT

Hi Geoffrey,

Please find attatched the RM log from today.

rm-log.txt

Re: Hive error on HDP6

winnie_philip — Fri, 06 Oct 2017 22:39:18 GMT

umask on Linux is 0022

Re: Hive error on HDP6

jsensharma — Fri, 06 Oct 2017 23:17:35 GMT

@Winnie Philip

We see the Warning as :

2017-10-06 10:28:10,119 WARN  localizer.ResourceLocalizationService (ResourceLocalizationService.java:checkLocalDir(1445)) - Permissions incorrectly set for dir /usr/local/opt/hadoop/yarn/local/nmPrivate, should be rwx------, actual value = rwx-w----<br>

We will need to check the permissions for yarn local directories Which basically depends on the umask value set for the user 'yarn' at system level.

What is the umask value set for the yarn user set under the following files:

/etc/bashrc 
/etc/profile

Above is a WARNING message which can be seen when the ResourceLocalizationService checks the permission of nmPrivate, if it is not 700, it will just log a Warn message and change the permission to 700.

But in your case it is not able to change the permission.

WARN  localizer.ResourceLocalizationService (ResourceLocalizationService.java:initializeLocalDir(1277)) - Could not set permissions for local dir /appl/hadoop/yarn/local/usercache
EPERM: Operation not permitted

Please check the Node Managers' file/folder permissions is it is set to "yarn:hadoop" (like /appl/hadoop/yarn/)

# ls -ld /appl/hadoop/yarn
# chown yarn:hadoop /appl/hadoop/yarn

Re: Hive error on HDP6

winnie_philip — Fri, 06 Oct 2017 23:47:23 GMT

all 3 nodes have following lines in 2 files, how do I check umask value set for the yarn user? i logged as sudo su - yarn, and noted umask is 022

/etc/bashrc
/etc/profile

# By default, we want umask to get set. This sets it for non-login shell. # Current threshold for system reserved uid/gids is 200 # You could check uidgid reservation validity in # /usr/share/doc/setup-*/uidgid file

if [ $UID -gt 199 ] && [ "`/usr/bin/id -gn`" = "`/usr/bin/id -un`" ];

then umask 002

else umask 022

Re: Hive error on HDP6

winnie_philip — Sat, 07 Oct 2017 00:49:10 GMT

I noticed /appl/hadoop/yarn on different nodes had different owner/grp. I made them all to yarn:hadoop

before:

[yarn@vlmazgrpmaster ~]$ ls -ld /appl/hadoop/yarn drwxrwxrwx 4 root hadoop 30 Aug 24 14:10 /appl/hadoop/yarn [e1009156@vlmazgrpdata1 ~]$ ls -ld /appl/hadoop/yarn drwxrwxrwx 4 root root 30 Aug 24 14:36 /appl/hadoop/yarn [e1009156@vlmazgrpdata2 ~]$ ls -ld /appl/hadoop/yarn drwxrwxrwx 4 root hadoop 30 Aug 24 14:35 /appl/hadoop/yarn

now:

drwxrwxrwx 4 yarn hadoop 30 Aug 24 14:35 /appl/hadoop/yarn

Re: Hive error on HDP6

jsensharma — Sat, 07 Oct 2017 01:00:23 GMT

@Winnie Philip

After fixing the user:group .. is it working Or still failing

drwxrwxrwx 4 yarn hadoop 30 Aug 24 14:35 /appl/hadoop/yarn

Re: Hive error on HDP6

winnie_philip — Sat, 07 Oct 2017 01:09:31 GMT

It worked 1/3 test. it works sometimes, but not always:

I checked all nodemanager directories, they all looked same owner:grp:

ls -ld /hadoop/yarn/local drwxr-xr-x 6 yarn hadoop 78 Oct 6 12:56 /hadoop/yarn/local ls -ld /appl/hadoop/yarn/local drwxrwxrwx 5 yarn hadoop 57 Oct 6 12:56 /appl/hadoop/yarn/local ls -ld /usr/db/hadoop/yarn/local drwxr-xr-x 5 yarn hadoop 57 Oct 6 12:56 /usr/db/hadoop/yarn/local ls -ld /usr/local/opt/hadoop/yarn/local drwxr-xr-x 5 yarn hadoop 57 Oct 6 12:58 /usr/local/opt/hadoop/yarn/local

Re: Hive error on HDP6

winnie_philip — Sat, 07 Oct 2017 02:05:39 GMT

I ran the command with -R, looks like fixed it, I will watch for errors little bit, I restarted YARN also, this directory tree had mixed owner:grp.

chown yarn:hadoop -R /appl/hadoop/yarn

Re: Hive error on HDP6

jsensharma — Sat, 07 Oct 2017 02:25:26 GMT

@Winnie Philip

Great!!! to see your issue is fixed. It will be also great if you can mark this HCC thread as Answered by clicking on the "Accept" Button. That way other HCC users can quickly find the solution when they encounter the same issue.

Re: Hive error on HDP6

winnie_philip — Sat, 07 Oct 2017 02:46:17 GMT

Also, I went ahead and ran commands for all other nodemanager yarn directory chain on all nodes:

sudo chown -R yarn:hadoop -R /appl/hadoop/yarn
sudo chown -R yarn:hadoop /hadoop/yarn/
sudo chown -R yarn:hadoop /usr/db/hadoop/yarn/
sudo chown -R yarn:hadoop /usr/local/opt/hadoop/yarn/

Thank you very much for all help from everyone! I really appreciate!!

Re: Hive error on HDP6

winnie_philip — Sat, 07 Oct 2017 02:49:39 GMT

Thank you Geoffery

Re: Hive error on HDP6

winnie_philip — Sat, 07 Oct 2017 02:51:09 GMT

Thank you Sridhar!