Created 10-05-2017 08:36 PM
I'm keep getting permission error while loading data to Hive tables and/or querying tables. I granted full(r+w+x) permission to appl/hadoop/yarn/local/usercache in all of my cluster nodes. I'm still getting errors. I have 3 nodes cluster (HDP6).
Application application_1507209932823_0008 failed 2 times due to AM Container for appattempt_1507209932823_0008_000002 exited with exitCode: -1000 For more detailed output, check the application tracking page: http://mymaster.fisdev.local:8088/cluster/app/application_1507209932823_0008 Then click on links to logs of each attempt. Diagnostics: Could not set permissions for local dir /appl/hadoop/yarn/local/usercache Failing this attempt. Failing the application.
Created 10-06-2017 06:00 PM
After fixing the user:group .. is it working Or still failing
drwxrwxrwx 4 yarn hadoop 30 Aug 24 14:35 /appl/hadoop/yarn
.
Created 10-06-2017 05:49 PM
I noticed /appl/hadoop/yarn on different nodes had different owner/grp. I made them all to yarn:hadoop
before:
[yarn@vlmazgrpmaster ~]$ ls -ld /appl/hadoop/yarn drwxrwxrwx 4 root hadoop 30 Aug 24 14:10 /appl/hadoop/yarn [e1009156@vlmazgrpdata1 ~]$ ls -ld /appl/hadoop/yarn drwxrwxrwx 4 root root 30 Aug 24 14:36 /appl/hadoop/yarn [e1009156@vlmazgrpdata2 ~]$ ls -ld /appl/hadoop/yarn drwxrwxrwx 4 root hadoop 30 Aug 24 14:35 /appl/hadoop/yarn
now:
drwxrwxrwx 4 yarn hadoop 30 Aug 24 14:35 /appl/hadoop/yarn
Created 10-06-2017 06:00 PM
After fixing the user:group .. is it working Or still failing
drwxrwxrwx 4 yarn hadoop 30 Aug 24 14:35 /appl/hadoop/yarn
.
Created 10-06-2017 06:09 PM
It worked 1/3 test. it works sometimes, but not always:
I checked all nodemanager directories, they all looked same owner:grp:
ls -ld /hadoop/yarn/local drwxr-xr-x 6 yarn hadoop 78 Oct 6 12:56 /hadoop/yarn/local ls -ld /appl/hadoop/yarn/local drwxrwxrwx 5 yarn hadoop 57 Oct 6 12:56 /appl/hadoop/yarn/local ls -ld /usr/db/hadoop/yarn/local drwxr-xr-x 5 yarn hadoop 57 Oct 6 12:56 /usr/db/hadoop/yarn/local ls -ld /usr/local/opt/hadoop/yarn/local drwxr-xr-x 5 yarn hadoop 57 Oct 6 12:58 /usr/local/opt/hadoop/yarn/local
Created 10-06-2017 07:05 PM
I ran the command with -R, looks like fixed it, I will watch for errors little bit, I restarted YARN also, this directory tree had mixed owner:grp.
chown yarn:hadoop -R /appl/hadoop/yarn
Created 10-06-2017 07:25 PM
Great!!! to see your issue is fixed. It will be also great if you can mark this HCC thread as Answered by clicking on the "Accept" Button. That way other HCC users can quickly find the solution when they encounter the same issue.
Created 10-06-2017 07:46 PM
Also, I went ahead and ran commands for all other nodemanager yarn directory chain on all nodes:
Thank you very much for all help from everyone! I really appreciate!!