HADOOP_CLIENT_OPTS environment variable don't enforce fs.permissions.umask-mode

[#1]

Hi,

I wanted to change from a specific user a different from the default umask mode. Thus I've established the hadoop client options environment variable this way:

  export HADOOP_CLIENT_OPTS="-D fs.permissions.umask-mode=007"

I've notice that it don't enforce the umask mode and the default umask mode (022) is still in use creating files with permission mask rw-r--r--. However, if I execute the following hadoop fs -put command it use the new umask mode specified and the new file is created as expected with permissions rw-rw---- instead of rw-r--r--:

  hadoop fs -Dfs.permissions.umask-mode=007 -put file /directory

I wouldn't like to change all the commands that the user execute and it's better to establish the environment variable because it's transparent to the user.

Why the options in HADOOP_CLIENT_OPTS aren't taken into account?

The instalation where it all ocurrs is a Cloudera Enterprise 5.7.1 with CDH 5.7.1.

Thank you.

[#2]

Here's the code in question. Looks like it should be working. Are you seeing it not take effect immediately after the export or are you doing something between exporting the variable and running the hadoop command?

[#3]

Hi Cloudera, thank you for your reply.

As you said it should be working but surprisingly it's not. No, I'm doing nothing between declaring the environment variable and launching the hadoop command, and I did it in the same shell session. May be you can check it in your lab. The only thing I've noted is that the java commands launched underneath are slightly different:

When I launch the "hadoop fs -ls" command setting the environment variable HADOOP_CLIENT_OPTS, I see this:

/usr/java/jdk1.7.0_67-cloudera/bin/java -Xmx1000m -Dhadoop.log.dir=/xxx/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hadoop/logs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/xxx/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hadoop -Dhadoop.id.str= -Dhadoop.root.logger=INFO,console -Djava.library.path=/xxx/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Stack=true -Dfs.permissions.umask-mode=007 -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.fs.FsShell -ls

When I launch the "hadoop fs -Dfs.permissions.umask-mode=007 -ls" command, I see this:

/usr/java/jdk1.7.0_67-cloudera/bin/java -Xmx1000m -Dhadoop.log.dir=/xxx/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hadoop/logs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/xxx/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hadoop -Dhadoop.id.str= -Dhadoop.root.logger=INFO,console -Djava.library.path=/xxx/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Stack=true -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.fs.FsShell -Dfs.permissions.umask-mode=007 -ls

As you can see the declaration of the umask property goes before the FsShell class in the first case and after the main class in the second case. I guess that it could be ultimately the cause of the different behavior.

May be the code should be changed from "$HADOOP_OPTS $CLASS" to "$CLASS $HADOOP_OPTS ?

[#4]

The issue is that you're trying to pass an FS-specific option, which needs to go after the class name at run time whereas the HADOOP_OPTS are more general and need to be passed before the class. Unfortunately, it doesn't look like there's a way to do this at the shell.

[#5]

😞