Created 01-12-2018 11:19 AM
Hi Guys,
I have a group "Marketing" and it has 3 users
1. Mark1
2. Mark2
3. Mark3
and other group is "Account" and it has also 3 users
1. AC1
2. AC2
3. AC3
my questions are :
1. How i get the list of all user in any group ? I have tried below command it give me all users and all groups but not specifically.
"cat /etc/passwd | awk -F':' '{ print $1}' | xargs -n1 groups" .
2. I want to set the ACL in such way, In first scenario, User "AC1" (group: "Account") will have the Right "Read/Write/Execute" in Group "Marketing".
3. In second scenario, User Mark1 not able to copy the file into "Account" user.
Please guide me in details.
NOTE : I have been using Amazon Machine.
Thanks,
Created 01-12-2018 09:57 PM
Prerequisite for question 1,2 and 3
I am assuming you are creating the ACL's from scratch, below are steps to prepare the groups and users
Create the 2 groups
# groupadd Marketing # groupadd Account
Add the 3 users to Marketing group
# useradd -G Marketing Mark1 # useradd -G Marketing Mark2 # useradd -G Marketing Mark3
Add 3 users to Accounting group
# useradd -G Account AC1 # useradd -G Account AC2 # useradd -G Account AC3
Answer to question 1
There are 2 variations to get the all memebers of a group in linux the 2 versions of the command are below grep 'Account' /etc/group awk -F':' '/Marketing/{print $4}' /etc/group
Expected output
[root@nakuru ~]# grep 'Account' /etc/group Account:x:1029:AC1,AC2,AC3 [root@nakuru ~]# awk -F':' '/Marketing/{print $4}' /etc/group Mark1,Mark2,Mark3
To enable ACL's in HDP you need to set the dfs.namenode.acls.enabled to true using Ambari in custom hdfs-site.xml which is the recommended way. And restart all stale service typicall HDFS,MapReduce,YARN,ATLAS in my case see attached screenshot
Answer to question 2
Task Set user "AC1" (group: "Account") to have "Read/Write/Execute" privilege in Group "Marketing".
This will entail creating a file in hdfs with owner Mark1or 2 or 3 and group Marketing, as root switch to any user in group Marketing. First create a directory in hdfs and change the ownership to Mark1 and group Marketing
As hdfs user created the directory and change ownership and permission
# su - hdfs [hdfs@nakuru ~] $ hdfs dfs -mkdir -p /marketing/acldemo [hdfs@nakuru ~]$ hdfs dfs -chown -R Mark1:marketing /marketing/acldemo
Validate the above commands were successful.
[hdfs@nakuru ~]$ hdfs dfs -ls /marketing Found 1 items drwxr-xr-x - Mark1 marketing 0 2018-01-12 21:54 /marketing/acldemo
Get the current ACL
[hdfs@nakuru ~]$ hdfs dfs -getfacl -R /marketing/acldemo # file: /marketing/acldemo # owner: Mark1 # group: marketing user::rwx group::r-x other::r-x
I removed the r-x for other to be sure and revalidate note the others now had no r-x
[Mark1@nakuru ~]$ hdfs dfs -chmod 750 /marketing/acldemo [Mark1@nakuru ~]$ hdfs dfs -getfacl -R /marketing/acldemo # file: /marketing/acldemo # owner: Mark1 # group: marketing user::rwx group::r-x other::---
Switch to user Mark1 create a local file and copy to
hdfs # su - Mark1 [Mark1@nakuru ~]$ echo "This is Hussain testing ACL ser "AC1" (group: "Account") will have the Right Read/Write/Execute in Group "Marketing"" > test1.txt [Mark1@nakuru ~]$ ls -al -rw-r--r-- 1 Mark1 Marketing 113 Jan 12 21:51 test1.txt
Copy the above file to hdfs in previously created directory and check that it was successfully copied to hdfs
[Mark1@nakuru ~]$ hdfs dfs -put test1.txt /marketing/acldemo [Mark1@nakuru ~]$ hdfs dfs -ls /marketing/acldemo Found 1 items -rw-r--r-- 3 Mark1 marketing 113 2018-01-12 22:05 /marketing/acldemo/test1.txt
Testing
Switched to user AC1 in group Account to see if he could read the file, it failed that's normal
[root@nakuru ~]# su AC1 [AC1@nakuru root]$ hdfs dfs -cat /marketing/acldemo/test1.txt cat: Permission denied: user=AC1, access=EXECUTE, inode="/marketing/acldemo/test1.txt":Mark1:marketing:drwxr-x-
Change the ACL for user AC1 of group Account to have rwx as you requested
[Mark1@nakuru ~]$ hdfs dfs -setfacl -m user:AC1:rwx /marketing/acldemo
Check the new ACL,note now the user ACI now has rwx on the file test1
[Mark1@nakuru ~]$ hdfs dfs -getfacl /marketing/acldemo/test1.txt # file: /marketing/acldemo/test1.txt # owner: Mark1 # group: marketing user::rw- user:AC1:rwx group::r-- mask::rwx other::r--
Switch to user AC1 and test that user AC1 can now read the file.
[root@nakuru ~]# su AC1 [AC1@nakuru root]$ hdfs dfs -cat /marketing/acldemo/test1.txt This is Hussain testing ACL ser AC1 (group: Account) will have the Right Read/Write/Execute in Group Marketing
SUCCESS !
Answer to question 3
User Mark1 of Marketing should not able to copy the file into "Account" user, create directory and change ownership to any user in Account group
[root@nakuru ~]# su - hdfs [hdfs@nakuru ~]$ hdfs dfs -mkdir -p /Account/acldemo2 [hdfs@nakuru ~]$ hdfs dfs -chown AC1:Account /Account/acldemo2
Get the ACL of newly created directory, note the 3 octets (other is r-x)
[root@nakuru ~]# su AC1 [AC1@nakuru root]$ hdfs dfs -getfacl /Account/acldemo2 # file: /Account/acldemo2 # owner: AC1 # group: Account user::rwx group::r-x other::r-x
Test with user Mark1 can't copy a file to the directory /Account/acldemo2 from local
[root@nakuru ~]# su - Mark1 [Mark1@nakuru ~]$ hdfs dfs -put test1.txt /Account/acldemo2 put: Permission denied: user=Mark1, access=WRITE, inode="/Account/acldemo2/test1.txt._COPYING_":AC1:Account:drwxr-xr-x
The above is quite straightforward Mark1 belongs to Marketing and doesn't have any permissions on this directory, I hope that's what you meant?
Please if that answers your question then please Accept the answer by Clicking on Accept button below, That would be a great help to Community users to find a solution quickly for these kinds of ACL issues.
Created 01-12-2018 09:57 PM
Prerequisite for question 1,2 and 3
I am assuming you are creating the ACL's from scratch, below are steps to prepare the groups and users
Create the 2 groups
# groupadd Marketing # groupadd Account
Add the 3 users to Marketing group
# useradd -G Marketing Mark1 # useradd -G Marketing Mark2 # useradd -G Marketing Mark3
Add 3 users to Accounting group
# useradd -G Account AC1 # useradd -G Account AC2 # useradd -G Account AC3
Answer to question 1
There are 2 variations to get the all memebers of a group in linux the 2 versions of the command are below grep 'Account' /etc/group awk -F':' '/Marketing/{print $4}' /etc/group
Expected output
[root@nakuru ~]# grep 'Account' /etc/group Account:x:1029:AC1,AC2,AC3 [root@nakuru ~]# awk -F':' '/Marketing/{print $4}' /etc/group Mark1,Mark2,Mark3
To enable ACL's in HDP you need to set the dfs.namenode.acls.enabled to true using Ambari in custom hdfs-site.xml which is the recommended way. And restart all stale service typicall HDFS,MapReduce,YARN,ATLAS in my case see attached screenshot
Answer to question 2
Task Set user "AC1" (group: "Account") to have "Read/Write/Execute" privilege in Group "Marketing".
This will entail creating a file in hdfs with owner Mark1or 2 or 3 and group Marketing, as root switch to any user in group Marketing. First create a directory in hdfs and change the ownership to Mark1 and group Marketing
As hdfs user created the directory and change ownership and permission
# su - hdfs [hdfs@nakuru ~] $ hdfs dfs -mkdir -p /marketing/acldemo [hdfs@nakuru ~]$ hdfs dfs -chown -R Mark1:marketing /marketing/acldemo
Validate the above commands were successful.
[hdfs@nakuru ~]$ hdfs dfs -ls /marketing Found 1 items drwxr-xr-x - Mark1 marketing 0 2018-01-12 21:54 /marketing/acldemo
Get the current ACL
[hdfs@nakuru ~]$ hdfs dfs -getfacl -R /marketing/acldemo # file: /marketing/acldemo # owner: Mark1 # group: marketing user::rwx group::r-x other::r-x
I removed the r-x for other to be sure and revalidate note the others now had no r-x
[Mark1@nakuru ~]$ hdfs dfs -chmod 750 /marketing/acldemo [Mark1@nakuru ~]$ hdfs dfs -getfacl -R /marketing/acldemo # file: /marketing/acldemo # owner: Mark1 # group: marketing user::rwx group::r-x other::---
Switch to user Mark1 create a local file and copy to
hdfs # su - Mark1 [Mark1@nakuru ~]$ echo "This is Hussain testing ACL ser "AC1" (group: "Account") will have the Right Read/Write/Execute in Group "Marketing"" > test1.txt [Mark1@nakuru ~]$ ls -al -rw-r--r-- 1 Mark1 Marketing 113 Jan 12 21:51 test1.txt
Copy the above file to hdfs in previously created directory and check that it was successfully copied to hdfs
[Mark1@nakuru ~]$ hdfs dfs -put test1.txt /marketing/acldemo [Mark1@nakuru ~]$ hdfs dfs -ls /marketing/acldemo Found 1 items -rw-r--r-- 3 Mark1 marketing 113 2018-01-12 22:05 /marketing/acldemo/test1.txt
Testing
Switched to user AC1 in group Account to see if he could read the file, it failed that's normal
[root@nakuru ~]# su AC1 [AC1@nakuru root]$ hdfs dfs -cat /marketing/acldemo/test1.txt cat: Permission denied: user=AC1, access=EXECUTE, inode="/marketing/acldemo/test1.txt":Mark1:marketing:drwxr-x-
Change the ACL for user AC1 of group Account to have rwx as you requested
[Mark1@nakuru ~]$ hdfs dfs -setfacl -m user:AC1:rwx /marketing/acldemo
Check the new ACL,note now the user ACI now has rwx on the file test1
[Mark1@nakuru ~]$ hdfs dfs -getfacl /marketing/acldemo/test1.txt # file: /marketing/acldemo/test1.txt # owner: Mark1 # group: marketing user::rw- user:AC1:rwx group::r-- mask::rwx other::r--
Switch to user AC1 and test that user AC1 can now read the file.
[root@nakuru ~]# su AC1 [AC1@nakuru root]$ hdfs dfs -cat /marketing/acldemo/test1.txt This is Hussain testing ACL ser AC1 (group: Account) will have the Right Read/Write/Execute in Group Marketing
SUCCESS !
Answer to question 3
User Mark1 of Marketing should not able to copy the file into "Account" user, create directory and change ownership to any user in Account group
[root@nakuru ~]# su - hdfs [hdfs@nakuru ~]$ hdfs dfs -mkdir -p /Account/acldemo2 [hdfs@nakuru ~]$ hdfs dfs -chown AC1:Account /Account/acldemo2
Get the ACL of newly created directory, note the 3 octets (other is r-x)
[root@nakuru ~]# su AC1 [AC1@nakuru root]$ hdfs dfs -getfacl /Account/acldemo2 # file: /Account/acldemo2 # owner: AC1 # group: Account user::rwx group::r-x other::r-x
Test with user Mark1 can't copy a file to the directory /Account/acldemo2 from local
[root@nakuru ~]# su - Mark1 [Mark1@nakuru ~]$ hdfs dfs -put test1.txt /Account/acldemo2 put: Permission denied: user=Mark1, access=WRITE, inode="/Account/acldemo2/test1.txt._COPYING_":AC1:Account:drwxr-xr-x
The above is quite straightforward Mark1 belongs to Marketing and doesn't have any permissions on this directory, I hope that's what you meant?
Please if that answers your question then please Accept the answer by Clicking on Accept button below, That would be a great help to Community users to find a solution quickly for these kinds of ACL issues.
Created 01-14-2018 10:56 PM
Did it resolve your sitaution?
Created on 01-16-2018 11:19 AM - edited 08-18-2019 12:58 AM
Thanks a lot @Geoffrey Shelton Okot for your brief answer. Sorry for late reply!
I am unable to see the All user of the Group. please see the attache image :
Created 01-16-2018 11:53 AM
I see you are failing on the namenode. Whats your cluster setup (node distribution) single or multinode cluster ? Typical your user should have been created on the gateway node.
Please revert
Created 01-16-2018 11:57 AM
I have been using amazon machine. node detail :
1. ResourceManager
2. HiveServer
3. ResourceManager
4. Node2
5. Node1 ( I have added this node into cluster)
Thanks
Created 01-16-2018 01:01 PM
Do you plan to have Resource Manager HA only and not a Namenode HA? How many physical servers in AWS do you plan to deploy? Here is a typical setup looks like this
1.Gateway aka edge node
2.Master nodes best is more than one for NN HA and RM HA etc
3.Slave node aka data nodes (As many as possible)
Are you using a blueprint to deploy in AWS?
Created 01-16-2018 01:30 PM
Currently "Resource Manager High Availability" just on ResourceManager.I can add AdditionalResourceManager on "NameNode"
Everything set already on AWS machine. I am using it for exam learning.
Created 01-16-2018 01:34 PM
Ok good go ahead and do the setup and most probably we could do a remote session to check the ACL stuff.
Please keep me posted.