Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Hdfs acl giving permission denied error for spark job

Hdfs acl giving permission denied error for spark job

Expert Contributor

I have a user user1 with group group1

I use a third party tool which writes spark event logs to a directory in hdfs. Currently using the user home directory to write the logs temporarily.

I created this directory like:

Hdfs dfs -mkdir /user/user1/sparkeventlogs

Right now this is created under user1:group2

Changed ownership to the right group:

Hdfs dfs -chown -R user1:group1 /user/user1/sparkeventlogs

I also added ACL to the above directory using setfacl command and when I do getfacl it gives me correct user and group assigned. User and group both have rwx permissions

Now when the job is run, it the giving permission denied with a user who ran it under that AD group saying

User1:group2 drwx———

When actually it is

User1:group1 drwxrwx—-

We have ranger enabled but I don’t have access to it

Thanks for you help

1 REPLY 1
Highlighted

Re: Hdfs acl giving permission denied error for spark job

Expert Contributor

@PJ

Since you have ranger enabled, its possible that your permission is denied at Ranger end. I would definitely check the Ranger Audit logs for any events for the users and see if we are hitting the permission denied in there.

Also I would add a ranger hdfs policy to allow user user1 write access to /user/user1/sparkeventlogs once I validate it was Ranger who was blocking the permissions.

Don't have an account?
Coming from Hortonworks? Activate your account here