I am trying to load a simple file to HDP Hadoop cluster using HDFS client and I got the following exception. Exception in thread "main" org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /home/maria_dev/read_write_hdfs_example.txt could only be replicated to 0 nodes instead of minReplication (=1). There are 1 datanode(s) running and 1 node(s) are excluded in this operation. I looked into the namenode logs and enabled debug log level for NetworkTopology and BlockPlacementPolicy components. After enabling logs, I found that the data node 172.18.0.2:50010 is being excluded and since I am running only one datanode, it is unable to find second replica. 2023-04-28 06:00:22,188 DEBUG net.NetworkTopology (NetworkTopology.java:chooseRandom(780)) - Choosing random from 1 available nodes on node /default- rack, scope=/default-rack, excludedScope=null, excludeNodes=[] 2023-04-28 06:00:22,188 DEBUG net.NetworkTopology (NetworkTopology.java:chooseRandom(796)) - chooseRandom returning 172.18.0.2:50010 2023-04-28 06:00:22,189 INFO hdfs.StateChange (FSNamesystem.java:logAllocatedBlock(3866)) - BLOCK* allocate blk_1073743107_2310, replicas=172.18.0.2 :50010 for /home/maria_dev/read_write_hdfs_example.txt 2023-04-28 06:00:24,972 INFO BlockStateChange (BlockManager.java:computeReplicationWorkForBlocks(1653)) - BLOCK* neededReplications = 0, pendingRepl ications = 0. 2023-04-28 06:00:24,986 INFO destination.HDFSAuditDestination (HDFSAuditDestination.java:logJSON(179)) - Flushing HDFS audit. Event Size:2 2023-04-28 06:00:25,060 INFO hdfs.StateChange (FSNamesystem.java:completeFile(3759)) - DIR* completeFile: /spark2-history/.e3751543-0a05-4c2b-af27-f 3a7c02666b2 is closed by DFSClient_NONMAPREDUCE_-282543677_1 2023-04-28 06:00:27,974 INFO BlockStateChange (BlockManager.java:computeReplicationWorkForBlocks(1653)) - BLOCK* neededReplications = 0, pendingRepl ications = 0. 2023-04-28 06:00:27,988 INFO provider.BaseAuditHandler (BaseAuditHandler.java:logStatus(310)) - Audit Status Log: name=hdfs.async.batch.hdfs, interv al=01:00.313 minutes, events=23, succcessCount=23, totalEvents=878, totalSuccessCount=876, totalDeferredCount=2 2023-04-28 06:00:27,988 INFO destination.HDFSAuditDestination (HDFSAuditDestination.java:logJSON(179)) - Flushing HDFS audit. Event Size:3 2023-04-28 06:00:30,975 INFO BlockStateChange (BlockManager.java:computeReplicationWorkForBlocks(1653)) - BLOCK* neededReplications = 0, pendingRepl ications = 0. 2023-04-28 06:00:33,976 INFO BlockStateChange (BlockManager.java:computeReplicationWorkForBlocks(1653)) - BLOCK* neededReplications = 0, pendingRepl ications = 0. 2023-04-28 06:00:35,073 INFO hdfs.StateChange (FSNamesystem.java:completeFile(3759)) - DIR* completeFile: /spark2-history/.9aa45e6b-dff7-4f6d-b25d-a 662930c6797 is closed by DFSClient_NONMAPREDUCE_-282543677_1 2023-04-28 06:00:36,976 INFO BlockStateChange (BlockManager.java:computeReplicationWorkForBlocks(1653)) - BLOCK* neededReplications = 0, pendingRepl ications = 0. 2023-04-28 06:00:36,989 INFO destination.HDFSAuditDestination (HDFSAuditDestination.java:logJSON(179)) - Flushing HDFS audit. Event Size:3 2023-04-28 06:00:39,977 INFO BlockStateChange (BlockManager.java:computeReplicationWorkForBlocks(1653)) - BLOCK* neededReplications = 0, pendingRepl ications = 0. 2023-04-28 06:00:42,977 INFO BlockStateChange (BlockManager.java:computeReplicationWorkForBlocks(1653)) - BLOCK* neededReplications = 0, pendingRepl ications = 0. 2023-04-28 06:00:43,251 DEBUG net.NetworkTopology (NetworkTopology.java:chooseRandom(780)) - Choosing random from 0 available nodes on node /default- rack, scope=/default-rack, excludedScope=null, excludeNodes=[172.18.0.2:50010] 2023-04-28 06:00:43,251 DEBUG net.NetworkTopology (NetworkTopology.java:chooseRandom(796)) - chooseRandom returning null 2023-04-28 06:00:43,251 DEBUG blockmanagement.BlockPlacementPolicy (BlockPlacementPolicyDefault.java:chooseLocalRack(547)) - Failed to choose from lo cal rack (location = /default-rack); the second replica is not found, retry choosing ramdomly org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy$NotEnoughReplicasException: at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseRandom(BlockPlacementPolicyDefault.java:701) at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseRandom(BlockPlacementPolicyDefault.java:622) at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseLocalRack(BlockPlacementPolicyDefault.java:529) at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseLocalStorage(BlockPlacementPolicyDefault.java:489) Please help troubleshooting the issue further.
... View more
