Hi Timothy,
Here are some logs which might give you some insights, today I have eliminated Networking and Dual NiC issues, as both the clusters are the in the same Subnet and there is no dual NiC for these VMs and all possible traffic is seamlessly flowing back and forth.
Name Node Logs:
2020-08-13 16:01:26,120 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for mapred/principle@user.queue (auth:KERBEROS)
2020-08-13 16:01:26,126 INFO SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager: Authorization successful for mapred/principle@user.queue (auth:KERBEROS) for protocol=interface user.queue.user.queue.user.queue
2020-08-13 16:01:28,672 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for principle@user.queue (auth:KERBEROS)
2020-08-13 16:01:28,679 INFO SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager: Authorization successful for principle@user.queue (auth:KERBEROS) for protocol=interface user.queue.user.queue.user.queue
2020-08-13 16:01:28,704 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocateBlock: /user/abc/puthdfs_test/.user.queue. user.queue.64.55-1545405130172 blk_1075343453_1603222{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-53536364-33f4-40d6-85c2-508abf7ff023:NORMAL:00.00.64.58:50010|RBW], ReplicaUnderConstruction[[DISK]DS-abba7d97-925a-4299-af86-b58fef9aaa12:NORMAL:00.00.64.84:50010|RBW], ReplicaUnderConstruction[[DISK]DS-286b28e8-d035-4b8c-a2dd-aabb08666234:NORMAL:00.00.64.56:50010|RBW]]}
2020-08-13 16:01:28,727 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocateBlock: /user/abc/puthdfs_test/.user.queue. user.queue.64.55-1545405130172 blk_1075343454_1603223{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-abba7d97-925a-4299-af86-b58fef9aaa12:NORMAL:00.00.64.84:50010|RBW], ReplicaUnderConstruction[[DISK]DS-286b28e8-d035-4b8c-a2dd-aabb08666234:NORMAL:00.00.64.56:50010|RBW], ReplicaUnderConstruction[[DISK]DS-d6f56418-6e18-4317-a8ec-4a5b15757728:NORMAL:00.00.64.57:50010|RBW]]}
2020-08-13 16:01:28,734 WARN org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy: Failed to place enough replicas, still in need of 1 to reach 3 (unavailableStorages=[], storagePolicy=BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]}, newBlock=true) For more information, please enable DEBUG log level on user.queue.user.queue.user.queue.BlockPlacementPolicy and user.queue.user.queue.NetworkTopology
2020-08-13 16:01:28,735 WARN org.apache.hadoop.hdfs.protocol.BlockStoragePolicy: Failed to place enough replicas: expected size is 1 but only 0 storage types can be selected (replication=3, selected=[], unavailable=[DISK], removed=[DISK], policy=BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]})
DataNode Logs:
2020-08-13 16:00:41,154 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving user.queue.64.55-1545405130172:blk_1075343452_1603221 src: /00.00.64.58:55510 dest: /00.00.64.57:50010
2020-08-13 16:00:41,213 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /00.00.64.58:55510, dest: /00.00.64.57:50010, bytes: 56, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1029630366_107, offset: 0, srvID: cb4e7a77-f5d6-49a5-abab-58d060602ec7, blockid: user.queue.64.55-1545405130172:blk_1075343452_1603221, duration: 54439548
2020-08-13 16:00:41,214 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: user.queue.64.55-1545405130172:blk_1075343452_1603221, type=HAS_DOWNSTREAM_IN_PIPELINE terminating
2020-08-13 16:00:45,149 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Scheduling blk_1075343452_1603221 file /data/dfs/dn/current/user.queue.64.55-1545405130172/current/finalized/subdir24/subdir112/blk_1075343452 for deletion
2020-08-13 16:00:45,149 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Deleted user.queue.64.55-1545405130172 blk_1075343452_1603221 file /data/dfs/dn/current/user.queue.64.55-1545405130172/current/finalized/subdir24/subdir112/blk_1075343452
2020-08-13 16:01:28,743 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: user.queue.user.queue:50010:DataXceiver error processing unknown operation src: /00.00.64.67:59988 dst: /00.00.64.57:50010
java.io.IOException:
at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.DataTransferSaslUtil.readSaslMessage(DataTransferSaslUtil.java:217)
at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.doSaslHandshake(SaslDataTransferServer.java:364)
at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.getEncryptedStreams(SaslDataTransferServer.java:178)
at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.receive(SaslDataTransferServer.java:110)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:193)
at java.lang.Thread.run(Thread.java:748)
2020-08-13 16:01:41,210 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving user.queue.64.55-1545405130172:blk_1075343458_1603227 src: /00.00.64.56:58556 dest: /00.00.64.57:50010
2020-08-13 16:01:41,223 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /00.00.64.56:58556, dest: /00.00.64.57:50010, bytes: 56, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_1081910632_107, offset: 0, srvID: cb4e7a77-f5d6-49a5-abab-58d060602ec7, blockid: user.queue.64.55-1545405130172:blk_1075343458_1603227, duration: 8787325
2020-08-13 16:01:41,225 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: user.queue.64.55-1545405130172:blk_1075343458_1603227, type=HAS_DOWNSTREAM_IN_PIPELINE terminating
2020-08-13 16:01:45,151 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Scheduling blk_1075343458_1603227 file /data/dfs/dn/current/user.queue.64.55-1545405130172/current/finalized/subdir24/subdir112/blk_1075343458 for deletion
2020-08-13 16:01:45,151 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Deleted user.queue.64.55-1545405130172 blk_1075343458_1603227 file /data/dfs/dn/current/user.queue.64.55-1545405130172/current/finalized/subdir24/subdir112/blk_1075343458
2020-08-13 16:02:13,278 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving user.queue.64.55-1545405130172:blk_1075343459_1603228 src: /00.00.64.55:43446 dest: /00.00.64.57:50010
Thanks
David
