I recommend not setting this in core-site.xml, and instead setting it on the command line invocation specifically for the DistCp command that needs to communicate with the unsecured cluster. Setting it in core-site.xml means that all RPC connections for any application are eligible for fallback to simple authentication. This potentially expands the attack surface for man-in-the-middle attacks. Here is an example of overriding the setting on the command line while running DistCp: hadoop distcp -D ipc.client.fallback-to-simple-auth-allowed=true hdfs://nn1:8020/foo/bar hdfs://nn2:8020/bar/foo The command must be run while logged into the secured cluster, not the unsecured cluster. ... View more

If there is no existing documentation covering the JournalNode, then here is my recommendation. Bootstrap the new server by copying the contents of dfs.journalnode.edits.dir from an existing JournalNode. Start JournalNode on the new server. Reconfigure the NameNodes to include the new server in dfs.namenode.shared.edits.dir. Restart standby NN and verify it remains healthy. Restart active NN and verify it remains healthy. Reconfigure the NameNodes to remove the old server from dfs.namenode.shared.edits.dir. Restart standby NN and verify it remains healthy. Restart active NN and verify it remains healthy. Some might note that during the copy in step 1, it's possible that additional transactions are being logged concurrently, so the copy might be out-of-date immediately. This is not a problem though. The JournalNode is capable of "catching up" by synchronizing data from other running JournalNodes. In fact, step 1 is really just an optimization of this "catching up". ... View more

ipc.server.tcpnodelay controls use of Nagle's algorithm on any server component that makes use of Hadoop's common RPC framework. That means that full deployment of a change in this setting would require a restart of any component that uses that common RPC framework. That's a broad set of components, including all HDFS, YARN and MapReduce daemons. It probably also includes other components in the wider ecosystem. ... View more

The ACLs specified in the hadoop-policy.xml file refer to Hadoop service-level authorization. http://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-common/ServiceLevelAuth.html These ACLs are enforced on Hadoop RPC service calls. These ACLs are not applicable to access through WebHDFS. In order to fully control authorization to HDFS files, use HDFS permissions and ACLs. http://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-hdfs/HdfsPermissionsGuide.html Permissions and ACLs applied to directories and files are enforced for all means of access to the file system. Other potential solutions are to use Knox or Ranger. ... View more

There is currently no way for a newly created directory in HDFS to set its group from the primary group of the creating user automatically. Instead, it always follows the rule quoted in the question: the group is the group of the parent directory. One way I've handled this in the past is first to create an intermediate directory and then explicitly change its group to the user's primary group, using chmod on the shell or setOwner in the Java APIs. Then, additional files and directories created by the process would use this as the destination directory. For example, a MapReduce job could specify its output directory under this intermediate directory, and then the output files created by that MapReduce job would have the desired group. ... View more

Does this question refer to Hadoop Service Level Authorization? http://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-common/ServiceLevelAuth.html If so, then there is no need to restart the NameNode to make changes in service-level ACLs take effect. Instead, an admin can run this command: hdfs dfsadmin -refreshServiceAcl More documentation on this command is available here: http://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-hdfs/HDFSCommands.html#dfsadmin There is similar functionality for YARN too: http://hadoop.apache.org/docs/r2.7.1/hadoop-yarn/hadoop-yarn-site/YarnCommands.html#rmadmin Another way to manage this is to declare a single "hadoopaccess" group for use in the service-level ACL definitions. Whenever a new set of users needs access, they would be added to this group. This shifts the management effort to an AD/LDAP administrator. Different IT shops would likely make a different trade-off between managing it that way or managing it in the service-level authorization policy files. Both approaches are valid, and it depends on the operator's preference. ... View more