I keep getting this access denied even though i'm changing the permission of the directory of /opt/yarn/nm/usercache/
this is encountered upon doing some select query with joins
AccessDeniedException : /opt/yarn/nm/usercache/user/appcache/application_123123_00/blockmgr-clel77710/0a/shuffle_5_1_0.index
How can I fix it permanently?
May be the permissions are messed up can you try the following steps and see whether it helps.
If you keep seeing this, check in the bash profile on all nodes, if the umask is set to something else. It should be 0022.