HDFS Permissions Get Reverted On Their Own

[#1]

The users of our cluster have requested us to create an HDFS directory for them to store their hive tables under /projects/hawpm in HDFS. So i have given their directory 755 permissions using hdfs dfs -chmod, since they use Tableau with the native Hadoop Hive connection to extract their data. which works. However, after almost an hour the permissions are set back to 700 automatically and any changes of ownership to the directory are also being reverted. I do not have sticky bits enabled on this directory. What could be causing my permissions to get reverted like this?

Please note that this directory given does not replace the Hive metadata directory /apps/hive/warehouse which is also being used.

drwx------   - hawpm       hawpm                 0 2017-10-25 09:29 /projects/hawpm

[#2]

Is there also a way I can check what reverted those changes? I cannot find anything in the name node logs.

[#3]

@Naif Massoud

The best way to findout who and what changed the HDFS directory permission will be to see the HDFS audit logs.

# tail -f /var/log/hadoop/hdfs/hdfs-audit.log

.

Example Output: (please check the "ugi" attribute to know who changed it)

2017-11-21 13:22:08,695 INFO FSNamesystem.audit: allowed=true    ugi=hdfs (auth:SIMPLE)    ip=/172.23.116.41    cmd=setPermission    src=/projects/hawpm    dst=null    perm=hdfs:hdfs:rwxrwxrwx    proto=rpc    callerContext=CLI

.

[#4]

Hi @Jay Kumar SenSharma,
Thank you for the quick reply. I have checked the logs as you proposed but unfortunately, I only see the permission changes I have made but not what reverted them.

Could it be that there is a certain configuration causing this?

[#5]

Is it possible to lock the directory /projects/hawpm that nothing can change its permissions until it's unlocked again?

[#6]

@Naif Massoud,

Do you have any ranger policies on the folder. Also check if any jobs are running which can change the permissions of the folder.

For this question 'Is it possible to lock the directory /projects/hawpm that nothing can change its permissions until it's unlocked again?'

You can create a policy in ranger with the permissions you want to set and give access to only users/group you want .

Thanks,

Aditya

[#7]

@Aditya Sirna

I do not have Ranger policies set for that folder. I am only using POSIX on it. I cannot see anything in the audit logs that shows what changes the permissions back to 700 instead of mine which were 755. it even resets back the owner if I change it from hawpm.

The underlying directories /projects/hawpm/* however have 755 permissions set which is the default set from the umask 022 in Ambari. I really find it strange that this parent directory keeps reverting back the permissions.

Cheers,

Naif

[#8]

Another question:

When someone adds a new metadata directory for Hive (other than /apps/hive/warehouse), does it have to be added somewhere in the Hive configs in Ambari? How does Hive understand that it can also find tables there?

Does it make sense to have something like this in case both directories are used:

hive.metastore.warehouse.dir = /apps/hive/warehouse, /projects/