question NiFi PutHDFS Processor doesn't use "fs.permissions.umask-mode" in Support Questions

NiFi PutHDFS Processor doesn't use "fs.permissions.umask-mode"

kei_miyauchi_ — Wed, 05 Sep 2018 09:39:58 GMT

Hi,

I try to put a file into HDFS via PutHDFS Processor. At least the flow works, but "fs.permissons.umask-mode" on hdfs-site.xml is ignored.
After some investigation, I found PutHDFS Processor refers deprecated property "dfs.umaskmode" which is now replaced by "fs.permissions.umask-mode".
I set "dfs.umaskmode" on hdfs-site.xml by myself, but still the Processor doesn't set permission properly.
"Permission umask" property on PutHDFS processor works, but I don't wanna set it for each Processor.
How can I let the processor use "fs.permissions.umask-mode"? Any workaround will do.

Re: NiFi PutHDFS Processor doesn't use "fs.permissions.umask-mode"

jts — Thu, 06 Sep 2018 04:20:53 GMT

@Kei Miyauchi

With core-site.xml and hdfs-site.xml being provided in the "Hadoop Configuration Resources" property, that config is passed to the hadoop client that PutHDFS uses to send data to HDFS. However, in the code it looks like if the "Permissions umask" property is not set, then PutHDFS will use a default umask of "18", which is pulled from FsPermission.java from hadoop-common.

Unfortunately, I don't think there's a workaround. The "Permissions umask" property doesn't support EL, so for now you would have to set the umask explicitly via the property.

I created bug NIFI-5575 to track the issue.

Re: NiFi PutHDFS Processor doesn't use "fs.permissions.umask-mode"

kei_miyauchi_ — Thu, 06 Sep 2018 09:31:54 GMT

It's a pity that there's no workaround. Okay now I'll set "Permission umask" for each processor until the issue resolved.
Thank you for the reply and advice. I'll watch the issue you've created.

Re: NiFi PutHDFS Processor doesn't use "fs.permissions.umask-mode"

jts — Fri, 07 Sep 2018 04:03:59 GMT

I apologize that I couldn't think of a workaround, and that you'll have to set "Permission umask" for each processor. After NIFI-5575 is resolved, it will be included in a future HDF release, and you should be able to update your flow to remove the specific settings in each processor.