question Re: ACLs are enabled and applied but not working in Support Questions

ACLs are enabled and applied but not working

muthukumar_siva — Mon, 08 Jan 2018 13:11:01 GMT

Dear all,

I have enabled ACL on the ambari console and restarted the required services and I'm able to set the permissions for specific group as well. But when they try to execute it is not working. Need your suggestions. My HDP version is 2.4 and hadoop 2.7.

getfacl permission on the folder and file is:

$ hdfs dfs -getfacl -R /abc/month=12/
# file: /abc/month=12
# owner: abiuser
# group: dfsusers
user::rwx
group::r-x
group:data_team:r--
mask::r-x
other::---
default:user::rwx
default:group::r-x
default:group:data_team:r-x
default:mask::r-x
default:other::---

# file: /abc/month=12/file1.bcsf
# owner: abiuser
# group: dfsusers
user::rwx
group::r--
group:data_team:r--
mask::r--
other::---

user A and B are part of data_team, when they try to read the file we are getting the below error.

$ hadoop fs -ls /abc/month=12
ls: Permission denied: user=A, access=EXECUTE, inode="/abc/month=12":abiuser:dfsusers:drwxrwx---

Appreciate any suggestion / help?

Thank you

Re: ACLs are enabled and applied but not working

Shelton — Mon, 08 Jan 2018 16:54:30 GMT

@Muthukumar S

Please, could you explain the steps you did? To reproduce your scenario can you elaborate is user A and B =abiuser? what is the relation between dfsusers and data_team.

I have implemented numerous variations of permissions and I don't see why this shouldn't work

Re: ACLs are enabled and applied but not working

muthukumar_siva — Wed, 10 Jan 2018 17:27:12 GMT

@Geoffrey Shelton Okot

1. ACL feature is enabled by adding the below entry in custom hdfs-site.xml file and restarted the required services from ambari console.

<property>
<name>dfs.namenode.acls.enabled</name>
<value>true</value>
</property>

2. I gave sample as A and B user and they have been added to the group data_team (on Linux level), they are not abiuser. abiuser is the owner of the file. dfsusers is the group of that file (/abc/month=12/file1.bcsf). ACL permission added for the group data_team using the below command.

hdfs dfs -setfacl -m -R group:data_team:r-x /abc/month=12/

3. Above setup is done, but still user A and B not able to read or access the files where ACL permission been given.

Re: ACLs are enabled and applied but not working

sandeepksaini — Wed, 10 Jan 2018 18:17:22 GMT

@Muthukumar S

As suggested by Geoffery enable your setting for ACL to make it working. Furthermore,this is the link from HDP site regarding ACLs. It covers all basics how to enable, setup and check ACLs on Hadoop.

Re: ACLs are enabled and applied but not working

Shelton — Thu, 11 Jan 2018 06:36:11 GMT

@Muthukumar S

I have tried to reproduce your environment as below. HDP 2.6.2 Ambari 2.5.2, I don't think the version difference is an issue. Created group data_team,dfusers and users abisuer,usera and userb,please try to follow the steps I used to understand and compare with your own. I set the dfs.namenode.acls.enabled to true using Ambari which is the recommended way.

Created groups and users

[root@nakuru ~]# groupadd data_team 
[root@nakuru ~]# useradd -G data_team usera 
[root@nakuru ~]# useradd -G data_team userb 
[root@nakuru ~]# groupadd dfsusers 
[root@nakuru ~]# useradd abiuser

Switched to user abiuser belonging to group dfsusers and created a file file1.txt with the below contents

[root@nakuru ~]# su - abiuser 
[abiuser@nakuru ~]$ vi file1.txt 
/*contents*/
I have enabled ACL on the ambari console and restarted the required services and I'm able to set the permissions for specific group as well. But when they try to execute it is not working. Need your suggestions. My HDP version is 2.4 and hadoop 2.7.

Checked the saved file

[abiuser@nakuru ~]$ ls -al 
-rw-r--r-- 1 abiuser abiuser 250 Jan 10 22:24 file1.txt

Enabled ACL (custom hdfs-site.xml) through Ambari.

dfs.namenode.acls.enabled=true

Restart all stale configs in my case

HDFS

YARN

MapReduces2

Atlas

As hdfs user created the directory and change ownership and permission

[hdfs@nakuru ~]$ hdfs dfs -mkdir -p /abc/month=12 
[hdfs@nakuru ~]$ hdfs dfs -chown -R abiuser:dfsusers /abc/month=12

Validate the above

[hdfs@nakuru ~]$ hdfs dfs -ls /abc 
Found 1 items drwxr-xr-x - abiuser dfsusers 0 2018-01-10 22:40 /abc/month=12

Copy the file1.txt from local to hdfs

[abiuser@nakuru ~]$ hdfs dfs -put file1.txt /abc/month=12 
[abiuser@nakuru ~]$ hdfs dfs -ls /abc/month=12 
Found 1 items -rw-r--r-- 3 abiuser dfsusers 250 2018-01-10 22:46 /abc/month=12/file1.txt

Now see the ACL's on the file

[abiuser@nakuru ~]$ hdfs dfs -getfacl -R /abc/month=12/ 
# file: /abc/month=12 
# owner: abiuser 
# group: dfsusers 
user::rwx 
group::r-x 
other::r-x 
# file: /abc/month=12/file1.txt 
# owner: abiuser 
# group: dfsusers 
user::rw- 
group::r-- 
other::r--

Now set the ACL rwx for usera and userb as the file owner abiuser

[abiuser@nakuru ~]$ hdfs dfs -setfacl -m user:usera:rwx /abc/month=12/file1.txt 
[abiuser@nakuru ~]$ hdfs dfs -setfacl -m user:userb:rwx /abc/month=12/file1.txt

Validate the above ACL's for the file1.txt

[abiuser@nakuru ~]$ hdfs dfs -getfacl -R /abc/month=12/ 
# file: /abc/month=12 
# owner: abiuser 
# group: dfsusers 
user::rwx 
group::r-x 
other::r-x 
# file: /abc/month=12/file1.txt 
# owner: abiuser 
# group: dfsusers 
user::rw- 
user:usera:rwx 
user:userb:rwx 
group::r-- 
mask::rwx 
other::r--

See if usera can read the file

[root@nakuru ~]# su - usera 
[usera@nakuru ~]$ hdfs dfs -cat /abc/month=12/file1.txt 

I have enabled ACL on the ambari console and restarted the required services and I'm able to set the permissions for specific group as well. But when they try to execute it is not working. Need your suggestions. My HDP version is 2.4 and hadoop 2.7.

I get exactly the contents of the file1.txt

See if userb can read the file

[root@nakuru ~]# su - userb 
[userb@nakuru ~]$ hdfs dfs -cat /abc/month=12/file1.txt 

I have enabled ACL on the ambari console and restarted the required services and I'm able to set the permissions for specific group as well. But when they try to execute it is not working. Need your suggestions. My HDP version is 2.4 and hadoop 2.7.

Voila , that answers your question the file owner and group is abiuser:dfsuser but usera and userb from a different group data_team can successfully read the file1.txt

Could you Accept the answer by Clicking on Accept button below, if this answers your problem that would be great help to Community users to find solution quickly.

Re: ACLs are enabled and applied but not working

muthukumar_siva — Thu, 11 Jan 2018 10:29:34 GMT

@Geoffrey Shelton Okot

First of all thanks for your time and outputs, samething been done with only one difference. I have given acl permission for the group data_team with r-x instead of individual users. In future there will be a requirement for other users to get only read access which I can do by just adding them to the group data_team in Linux. Hope this also should work. Below is the command I have used.

hdfs dfs -setfacl -m -R group:data_team:r-x /abc/month=12

Could you create different file with abiuser as owner and dfsusers as group and add ACL for the group data_team with just read permission?

Thank you.

Re: ACLs are enabled and applied but not working

muthukumar_siva — Thu, 11 Jan 2018 10:31:50 GMT

@Sandeep Kumar

Yes, I have referred those documents already and set as required. Problem is it is not allowing the user to read the file which got a proper permission in ACL. You may go through my initial postings with the steps.

Thank you

Re: ACLs are enabled and applied but not working

Shelton — Fri, 12 Jan 2018 00:00:54 GMT

@Muthukumar S

I have successfully reproduced as you request "created a different file with abiuser as owner and dfsusers as group and add ACL for the group data_team with just read permission?"

Created file acltest2.txt as user abiuser see contents

[root@nakuru ~]# su - abiuser
[abiuser@nakuru ~]$ vi acltest2.txt
Could you create different file with abiuser as owner and dfsusers as group and add ACL for the group data_team with just read permission?
Thank you.

Check the file

[abiuser@nakuru ~]$ ls -al
-rw-r--r--   1 abiuser dfsusers  151 Jan 11 13:00 acltest2.txt

Copied the file to hdfs

[abiuser@nakuru ~]$ hdfs dfs -put  acltest2.txt /abc/month=12

Confirmation of file in HDFS note user and group

[abiuser@nakuru ~]$ hdfs dfs -ls  /abc/month=12
Found 2 items
-rw-r--r--   3 abiuser dfsusers        151 2018-01-11 13:00 /abc/month=12/acltest2.txt
-rw-r--r--   3 abiuser dfsusers        249 2018-01-11 12:38 /abc/month=12/file1.txt

Set the ACL for group data_team [readonly] where usera and userb belong

[abiuser@nakuru ~]$ hdfs dfs -setfacl -m group:data_team:r-- /abc/month=12/acltest2.txt

Changed to usera

[root@nakuru ~]# su - usera

Successfully read the file as usera

[usera@nakuru ~]$ hdfs dfs -cat /abc/month=12/actest2.txt 
Could you create different file with abiuser as owner and dfsusers as group and add ACL for the group data_team with just read permission? Thank you.

Now lets check the ACL's

[usera@nakuru ~]$ hdfs dfs -getfacl -R /abc/month=12/ 
# file: /abc/month=12 
# owner: abiuser 
# group: dfsusers 
user::rwx 
group::r-x 
other::r-x 
# file: /abc/month=12/acltest2.txt 
# owner: abiuser 
# group: dfsusers 
user::rw- 
group::r-- 
group:data_team:r-- 
group:dfsusers:r-- 
mask::r-- 
other::r-- 
# file: /abc/month=12/file1.txt 
# owner: abiuser 
# group: dfsusers 
user::rw- 
group::r-- 
other::r--

Hope that answers your issue where did you encounter the problem is there a step you missed?

Please accept and close this thread

Re: ACLs are enabled and applied but not working

Shelton — Mon, 15 Jan 2018 06:59:06 GMT

@Muthukumar S
Can you revert whether this issue was resolved if so then accept and close the thread.

Re: ACLs are enabled and applied but not working

muthukumar_siva — Mon, 15 Jan 2018 23:14:02 GMT

@Geoffrey Shelton Okot

Sorry was stuck up in few issues and missed to reply. Yes the steps you have mentioned all followed. I was getting the error which i have shown in my first post. Hence i initialized this thread and you have provided the same steps which i have followed. Not sure what is wrong or some bug ?

When ran with user A or B who are part of data_team in ACL. 😞

"$ hadoop fs -ls /abc/month=12

ls: Permission denied: user=A, access=EXECUTE, inode="/abc/month=12":abiuser:dfsusers:drwxrwx---"

Re: ACLs are enabled and applied but not working

Shelton — Tue, 16 Jan 2018 04:21:34 GMT

@Muthukumar S

I am sure if it was a BUG then, hortonworks would have notified its customers having said that, it might sound trivial but try to go over your code, personally I don't see the issue with HDP 2.4 but if I may ask why haven't you upgraded?

[root@nakuru ~]# su - usera
[usera@nakuru ~]$ id
uid=1024(usera) gid=1024(usera) groups=1024(usera),507(data_team)
[usera@nakuru ~]$  hadoop fs -ls /abc/month=12
Found 2 items
-rw-r--r--+  3 abiuser dfsusers        151 2018-01-11 13:00 /abc/month=12/acltest2.txt
-rw-r--r--   3 abiuser dfsusers        249 2018-01-11 12:38 /abc/month=12/file1.txt
[usera@nakuru ~]$ hdfs dfs -getfacl -R /abc/month=12/
# file: /abc/month=12
# owner: abiuser
# group: dfsusers
user::rwx
group::r-x
other::r-x
# file: /abc/month=12/acltest2.txt
# owner: abiuser
# group: dfsusers
user::rw-
group::r--
group:data_team:r--
group:dfsusers:r--
mask::r--
other::r--
# file: /abc/month=12/file1.txt
# owner: abiuser
# group: dfsusers
user::rw-
group::r--
other::r--
[usera@nakuru ~]$

Since I reproduced your use case and provided the solution, I think its better you accept and close the thread. The hortonworks demo HDFS ACLS: fine-grained permissions for hdfs files in hadoop was delivered using HDP 2.4 , so at times when I get in such a situation I ask a friend to crosscheck my code you might have forgotten something.
Cheers !

Re: ACLs are enabled and applied but not working

muthukumar_siva — Tue, 16 Jan 2018 10:24:53 GMT

@Geoffrey Shelton Okot

Thanks will close the thread. Yes the steps are verified multiple times and we end up with that error. We have not subscribed for even hortonworks basic support, because of this risk we have not upgraded. In case we stuck up with some issues there is no one to help. Client is aware of this.

Re: ACLs are enabled and applied but not working

Shelton — Tue, 16 Jan 2018 15:59:51 GMT

@Muthukumar S

Okay cheers I will try to build an HDP 2.4 on VM what Ambari you ambari version. I hate to leave unfinished work.

Will update you

Re: ACLs are enabled and applied but not working

muthukumar_siva — Tue, 16 Jan 2018 17:29:21 GMT

@Geoffrey Shelton Okot

I think HDP 2.4 is not downloadable from hortonworks site? Because we will be setting up new environment in which we will install the latest version and only latest one is downloadable. May be there will be someother link for 2.4. Even I think it might be a bug on the version. There is no hint found for this error apart from the regular steps you have provided. Below are details you have asked for.

Ambari Server

$ rpm -qa | grep -i ambari
ambari-server-2.2.1.0-161.x86_64

$ rpm -qa | grep -i hadoop
hadoop_2_4_0_0_169-mapreduce-2.7.1.2.4.0.0-169.el6.x86_64
hadoop_2_4_0_0_169-yarn-2.7.1.2.4.0.0-169.el6.x86_64
hadoop_2_4_0_0_169-libhdfs-2.7.1.2.4.0.0-169.el6.x86_64
hadoop_2_4_0_0_169-2.7.1.2.4.0.0-169.el6.x86_64
hadoop_2_4_0_0_169-hdfs-2.7.1.2.4.0.0-169.el6.x86_64

$ rpm -qa | grep -i ambari
ambari-metrics-monitor-2.2.1.0-161.x86_64
ambari-metrics-collector-2.2.1.0-161.x86_64
ambari-agent-2.2.1.0-161.x86_64
ambari-metrics-hadoop-sink-2.2.1.0-161.x86_64

$ rpm -qa | grep -i hdp
hdp-select-2.4.0.0-169.el6.noarch

Re: ACLs are enabled and applied but not working

Shelton — Tue, 16 Jan 2018 20:09:57 GMT

@Muthukumar S

HDP 2.4 is still downloadable are https://hortonworks.com/downloads/#data-platform HDP Downloads Click view all locate Hortonworks Data Platform Archive and expand on the right of your screen

See attached screen.

I will download the sandbox and reproduce your use case.

Re: ACLs are enabled and applied but not working

muthukumar_siva — Wed, 17 Jan 2018 18:19:58 GMT

@Geoffrey Shelton Okot

Thank you very much for the information.