question Re: HDFS folder permission denied, but the user is in the owner group. in Support Questions

HDFS folder permission denied, but the user is in the owner group.

Seaport — Tue, 14 Jan 2020 05:05:57 GMT

I am using HDP. The inode in the following code is a managed hive table.

# id zeppelin
uid=1017(zeppelin) gid=1003(hadoop) groups=1003(hadoop),1005(zeppelin)
# sudo -u zeppelin hadoop fs -ls /warehouse/tablespace/managed/hive/test1
ls: Permission denied: user=zeppelin, access=READ_EXECUTE, inode="/warehouse/tablespace/managed/hive/test1":hive:hadoop:drwxrwx---

The user zeppelin is in hadoop group, which has full permisison on the hdfs folder. So why do I get the permission error?

Re: HDFS folder permission denied, but the user is in the owner group.

EricL — Tue, 14 Jan 2020 06:17:45 GMT

@Seaport ,

Can you try:

hdfs groups zeppelin

Or run "id zeppelin" on the active NN host?

Re: HDFS folder permission denied, but the user is in the owner group.

Seaport — Tue, 14 Jan 2020 16:05:03 GMT

# hdfs groups zeppelin
zeppelin : hadoop zeppelin

On the name node,

# id zeppelin
uid=1018(zeppelin) gid=1003(hadoop) groups=1003(hadoop),1005(zeppelin)

Re: HDFS folder permission denied, but the user is in the owner group.

Seaport — Tue, 14 Jan 2020 17:49:20 GMT

I might have found the reason.

I ran the following command as hdfs, which is the superuser of hdfs.

$ hadoop fs -getfacl /warehouse/tablespace/managed/hive/test1
# file: /warehouse/tablespace/managed/hive/test1
# owner: hive
# group: hadoop
user::rwx
user:hive:rwx
group::---
mask::rwx
other::---
default:user::rwx
default:user:hive:rwx
default:group::---
default:mask::rwx
default:other::---

The output, as I understand, shows that the group owner has no permission on the folder. My guess is that, HDP Hive uses ACL to limit direct access to files behind managed tables. HDP Hive tries to force accessing to managed tables only through Hive.

Re: HDFS folder permission denied, but the user is in the owner group.

EricL — Tue, 14 Jan 2020 22:06:46 GMT

@Seaport

Great, thanks for sharing! So try to use "hdfs dfs -setfacl" to update it and see how it goes.

Re: HDFS folder permission denied, but the user is in the owner group.

Seaport — Wed, 15 Jan 2020 01:09:07 GMT

I tried the following command

# sudo -u hdfs hadoop fs -setfacl -m g::rx /warehouse/tablespace/managed/hive/test1

But I got the error

-setfacl: Invalid type of acl in <aclSpec> :g::rx

The acl spec is to modify the owning group permission to rx.

Any suggestion?

Re: HDFS folder permission denied, but the user is in the owner group.

EricL — Wed, 15 Jan 2020 03:14:16 GMT

@Seaport,

Please refer to documentation here:
https://hadoop.apache.org/docs/r2.4.1/hadoop-project-dist/hadoop-common/FileSystemShell.html#setfacl

Maybe try:

sudo -u hdfs hadoop fs -setfacl -m group:hadoop:r-x /warehouse/tablespace/managed/hive/test1

Re: HDFS folder permission denied, but the user is in the owner group.

Shelton — Wed, 15 Jan 2020 07:43:15 GMT

@Seaport

As the permission is with the zeppelin user [other] you will need to do that at a user level, remember fine-grained security ONLY give what is necessary !!

$ hdfs dfs -getfacl /warehouse/tablespace/managed/hive # file: /warehouse/tablespace/managed/hive # owner: hive # group: hadoop user::rwx group::--- other::--- default:user::rwx default:user:hive:rwx default:group::--- default:mask::rwx default:other::---

The command below will set [ r-x } bits to the correct ACL you can change to rwx if you wish

hdfs dfs -setfacl -R -m user:zeppelin:r-x /warehouse/tablespace/managed/hive

Thereafter the zeppelin user can

[zeppelin~]$ hdfs dfs -ls /warehouse/tablespace/managed/hive Found 3 items drwxrwx---+ - hive hadoop 0 2018-12-12 23:42 /warehouse/tablespace/managed/hive/information_schema.db drwxrwx---+ - hive hadoop 0 2018-12-12 23:41 /warehouse/tablespace/managed/hive/sys.db drwxrwx---+ - hive hadoop 0 2020-01-15 00:20 /warehouse/tablespace/managed/hive/zepp.db

The earlier error is gone

ls: Permission denied: user=zeppelin, access=READ_EXECUTE, inode="/warehouse/tablespace/managed/hive":hive:hadoop:drwx------

Happy hadooping

Re: HDFS folder permission denied, but the user is in the owner group.

Seaport — Wed, 15 Jan 2020 18:02:34 GMT

@Shelton @EricL Thank you both.

the correct ACL spec is group::r-x

Now the following command works.

sudo -u zeppelin hadoop fs -ls /warehouse/tablespace/managed/hive/test1

From what I just ran into, I feel that, by design, Hive takes extra effort to prevent users from accessing managed table files directly. I will follow that design and access Hive managed table only through Hive.