How to can access the HDFS using my local linux user(permission denied)

[#1]

I'm using CDH 6.3.2. When trying to create directories in hdfs, I'm having permission denied like this one:

[devuser@master01 ~]$ hdfs dfs -mkdir /dev
mkdir: Permission denied: user=devuser, access=WRITE, inode="/":hdfs:supergroup:drwxr-xr-x

How can I add my user on hdfs? and how can my user run jobs as well?

[#2 · Accepted solution]

Hi, Mondi!

The msg you are getting is telling that user devuser does not have permission to write to the root (/) HDFS folder.  This is normal, and just like a Linux FS.  If you create a folder in HDFS and make your local unix account the owner, you can use that folder as you.

If you want to create objects in hdfs /, you need to become the superuser, hdfs.  Example:

# create your folder
$ sudo -u hdfs hdfs dfs -mkdir /user/devuser
# make devuser the owner:
$ sudo -u hdfs hdfs -chown devuser /user/devuser

You first elevate to hdfs, then run the hdfs dfs command.  This wil create your personal directory in /user and make you the owner.  Now, when you run hdfs commands as devuser, you will have permission to modify objects in that folder.

CAREFUL: elevating to hdfs is like becoming root in linux.  You can do a lot of damage if you make a mistake.

[#3 · Accepted solution]

Hi, Mondi!

The msg you are getting is telling that user devuser does not have permission to write to the root (/) HDFS folder.  This is normal, and just like a Linux FS.  If you create a folder in HDFS and make your local unix account the owner, you can use that folder as you.

If you want to create objects in hdfs /, you need to become the superuser, hdfs.  Example:

# create your folder
$ sudo -u hdfs hdfs dfs -mkdir /user/devuser
# make devuser the owner:
$ sudo -u hdfs hdfs -chown devuser /user/devuser

You first elevate to hdfs, then run the hdfs dfs command.  This wil create your personal directory in /user and make you the owner.  Now, when you run hdfs commands as devuser, you will have permission to modify objects in that folder.

CAREFUL: elevating to hdfs is like becoming root in linux.  You can do a lot of damage if you make a mistake.

[#4]

Hi @hughatoclc  thanks for this. this is a big help. however, how can I make my user also in hdfs supergroup? so that I can freely explore the hdfs and also can I run jobs using this as well?

[#5 · Accepted solution]

@Mondi 

just update dfs.cluster.administrators with the admin username you want in hdfs config (and restart hdfs,yarn,MR2,... services)

ex: dfs.cluster.administrators = hdfs,ops

you can also use a HDFS administrators group (only one administrator group) using dfs.permissions.superusergroup

ex : dfs.permissions.superusergroup = operations

to verify config has been updated, once services restarted

hdfs getconf -confKey dfs.cluster.administrators

or

hdfs getconf -confKey dfs.permissions.superusergroup