Hello, I have an application that copies data to HDFS, but is failing due to the datanode being excluded. See snippet: private void copyFileToHdfs(FileSystem hdfs, Path localFilePath, Path hdfsFilePath) throws IOException, InterruptedException {
log.info("Copying " + localFilePath + " to " + hdfsFilePath);
hdfs.copyFromLocalFile(localFilePath, hdfsFilePath);
} However, when I try to execute, I get: org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /user/dev/workflows/test.jar could only be replicated to 0 nodes instead of minReplication (=1). There are 1 datanode(s) running and 1 node(s) are excluded in this operation.
at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1583)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getNewBlockTargets(FSNamesystem.java:3109)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3033)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:725)
HDFS commands work fine. I can create, modify, delete. Copying data is the only problem. I've also attempted to check the ports by telnet'ing. I can telnet 8020, but not 50010. I assume this is the root of the issue and why the single datanode is being excluded. I attempted to add an iptable firewall rule but I still am running into the same issue. Any help is appreciated.
... View more
