Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Datanode data directory permissions

Solved Go to solution

Datanode data directory permissions

New Contributor

Since documents say set directory permission to 700 for datanode data directories( owned by hdfs), and Yarn/Node Manager/Containers will run as mapred userid, then how do mapred programs access blocks , based on my understanding mapred user wont be able to do ls -ltr ${dfs.datanode.data.dir} ? OR this is somehow driven by how file has permission set within hdfs?

 

Thanks for the help!!

 

 

Regards,

Dev

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Datanode data directory permissions

Master Guru
A few points:

1. Remote/non-short-circuit-local block reads are done via the DN IPC port (50010 or 1004), and not via local filesystem, so HDFS clients of any form will never require local permission access to the block files. The DN reads and streams the data. Same applies for writes.
2. Local short circuit reads, if enabled and applicable, are done in a secure manner via use of Unix Socket Domains. The DN opens the file and passes the file descriptor to the client, bypassing the need for clients to be able to read block file paths directly.
2 REPLIES 2

Re: Datanode data directory permissions

Master Guru
A few points:

1. Remote/non-short-circuit-local block reads are done via the DN IPC port (50010 or 1004), and not via local filesystem, so HDFS clients of any form will never require local permission access to the block files. The DN reads and streams the data. Same applies for writes.
2. Local short circuit reads, if enabled and applicable, are done in a secure manner via use of Unix Socket Domains. The DN opens the file and passes the file descriptor to the client, bypassing the need for clients to be able to read block file paths directly.

Re: Datanode data directory permissions

New Contributor

Thanks!