Datanode low number of blocks

Noel_0317 — Tue, 01 Aug 2023 08:56:49 GMT

Hi,

We have 3 datanodes but the other 1 is not having expected number of blocks compared to the other 2.

Also, the count of under-replicated blocks is high.

Datanode low block logs:

2023-08-01 07:31:30,134 DEBUG datanode.DataNode: isDatanode=false, isClient=true, isTransfer=false 2023-08-01 07:31:30,134 DEBUG datanode.DataNode: writeBlock receive buf size 530480 tcp no delay true 2023-08-01 07:31:30,134 INFO datanode.DataNode: Receiving BP-1419792351-10.128.20.54-1579223720671:blk_1122742296_49001972 src: /10.128.66.223:58748 dest: /10.128.64.127:9866 2023-08-01 07:31:30,134 DEBUG datanode.DataNode: BlockReceiver: BP-1419792351-10.128.20.54-1579223720671:blk_1122742296_49001972 storageType=DISK, inAddr=/10.128.66.223:58748, myAddr=/10.128.64.127:9866 stage=PIPELINE_SETUP_CREATE, newGs=0, minBytesRcvd=0, maxBytesRcvd=0 clientname=DFSClient_NONMAPREDUCE_378619176_53, srcDataNode=:0, datanode=hdfs-datanode-0.hdfs-datanode.metering.svc.cluster.local:9866 requestedChecksum=DataChecksum(type=CRC32C, chunkSize=512) cachingStrategy=CachingStrategy(dropBehind=null, readahead=null) allowLazyPersist=false, pinning=false, isClient=true, isDatanode=false, responseInterval=30000, storageID= 2023-08-01 07:31:30,135 DEBUG datanode.DataNode: writeTo blockfile is /hadoop/dfs/data/current/BP-1419792351-10.128.20.54-1579223720671/current/rbw/blk_1122742296 of size 0 2023-08-01 07:31:30,135 DEBUG datanode.DataNode: writeTo metafile is /hadoop/dfs/data/current/BP-1419792351-10.128.20.54-1579223720671/current/rbw/blk_1122742296_49001972.meta of size 0 2023-08-01 07:31:30,136 DEBUG datanode.DataNode: PacketResponder: BP-1419792351-10.128.20.54-1579223720671:blk_1122742296_49001972, type=LAST_IN_PIPELINE: seqno=-2 waiting for local datanode to finish w rite. 2023-08-01 07:31:30,137 DEBUG datanode.DataNode: Receiving one packet for block BP-1419792351-10.128.20.54-1579223720671:blk_1122742296_49001972: PacketHeader with packetLen=31560 header data: offsetInBlock: 0 seqno: 0 lastPacketInBlock: false dataLen: 31308 2023-08-01 07:31:30,138 DEBUG datanode.DataNode: PacketResponder: BP-1419792351-10.128.20.54-1579223720671:blk_1122742296_49001972, type=LAST_IN_PIPELINE: enqueue Packet(seqno=0, lastPacketInBlock=false, offsetInBlock=31308, ackEnqueueNanoTime=1058942549958316, ackStatus=SUCCESS) 2023-08-01 07:31:30,138 DEBUG datanode.DataNode: PacketResponder: BP-1419792351-10.128.20.54-1579223720671:blk_1122742296_49001972, type=LAST_IN_PIPELINE, replyAck=seqno: 0 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0 2023-08-01 07:31:30,138 DEBUG datanode.DataNode: PacketResponder: BP-1419792351-10.128.20.54-1579223720671:blk_1122742296_49001972, type=LAST_IN_PIPELINE: seqno=-2 waiting for local datanode to finish write. 2023-08-01 07:31:30,140 DEBUG datanode.DataNode: Receiving one packet for block BP-1419792351-10.128.20.54-1579223720671:blk_1122742296_49001972: PacketHeader with packetLen=4 header data: offsetInBlock: 31308 seqno: 1 lastPacketInBlock: true dataLen: 0

But the other 2 datanodes are having the same format of logs:

2023-08-01 07:35:47,430 DEBUG datanode.VolumeScanner: start scanning block BP-1419792351-10.128.20.54-1579223720671:blk_1097852465_24111684 2023-08-01 07:35:47,430 DEBUG datanode.DataNode: block=BP-1419792351-10.128.20.54-1579223720671:blk_1097852465_24111684, replica=FinalizedReplica, blk_1097852465_24111684, FINALIZED getNumBytes() = 26874 getBytesOnDisk() = 26874 getVisibleLength()= 26874 getVolume() = /hadoop/dfs/data getBlockURI() = file:/hadoop/dfs/data/current/BP-1419792351-10.128.20.54-1579223720671/current/finalized/subdir15/subdir6/blk_1097852465

bash-4.2$ hdfs fsck / | grep 'Under replicated' | grep /##/##/datasource_##/dt=2023-07-28/20230728_064016_04281_deje9_d6514f86-c17b-439c-8a7b-062141a490c9 /##/##/datasource_##/dt=2023-07-28/20230728_064016_04281_deje9_d6514f86-c17b-439c-8a7b-062141a490c9: Under replicated BP-1419792351-10.128.20.54-1579223720671:blk_1122552021_48811695. Target Replicas is 3 but found 2 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).

Is there a way to fix the high number of under-replicated blocks?

Re: Datanode low number of blocks

rki_ — Fri, 04 Aug 2023 07:36:35 GMT

@Noel_0317 Do you have rack awareness configured for the Datanodes?

Also, check for any disk-level issues on the datanode.

Try enabling Debug for block placement :

log4j.logger.org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy=DEBUG
log4j.logger.org.apache.hadoop.hdfs.protocol.BlockStoragePolicy=DEBUG

question Datanode low number of blocks in Support Questions

Datanode low number of blocks

Re: Datanode low number of blocks