hdfs fsck command returns under replicated blocks

Question

Hello,We have a 9 node HDP 3.1.5 cluster with erasure coding enabled. The hdfs fsck command returns under replicated blocks though the filesystem under path '/' is HEALTHY. Please advise how to reconstruct the replication for the under replicated blocks. Please find the hdfs fsck report below: [hdfs@hostname (prod) ~] $ hdfs fsck /Connecting to namenode via http://namenode:50070/fsck?ugi=hdfs&path=%2FFSCK started by hdfs (auth:SIMPLE) from /10.19.224.62 for path / at Wed May 26 22:48:22 PDT 2021/app/warehouse/db/data/eventdate=20210516/00000-21028-5a7dd08a-8c9e-4f62-b920-b6265292da8c-00001.orc:  Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850161008_890690. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s)./app/warehouse/db/data/eventdate=20210408/00047-789-396cdf80-9b0b-4411-a6e2-a1094cfb223e-00001.orc:  Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850438512_871703. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s)./app/warehouse/db/data/eventdate=20210408/00088-26191-cc73b880-2802-445f-bac6-39b3ca3f1c53-00001.orc:  Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850100896_894680. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s)./app/warehouse/db/data/eventdate=20210408/00090-26193-db7533fa-dd20-4b52-b17b-83ad0b5fdd42-00001.orc:  Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850100944_894675. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s)./app/warehouse/db/data/eventdate=20210408/00091-26194-5ecbd72f-3f3b-4ca1-a215-1d896a7b570f-00001.orc:  Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850100960_894672. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s)./app/warehouse/db/data/eventdate=20210408/00094-26197-bfa46627-e1be-4a4d-8dd9-bd6836375ddf-00001.orc:  Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850100912_894678. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s)./app/warehouse/db/data/eventdate=20210408/00113-11202-28809f0a-5a6b-4346-b7bf-ff049aee4a56-00001.orc:  Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850270336_883296. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s)./app/warehouse/db/data/eventdate=20210408/00115-11204-2170dec5-7a14-4e80-a1a6-e519cd59ff22-00001.orc:  Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850270352_883295. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s)./app/warehouse/db/data/eventdate=20210408/00116-11205-b7a5330a-5eb9-42d4-92fa-38e02a30a037-00001.orc:  Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850270368_883292. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s)./app/warehouse/db/data/eventdate=20210515/00021-763-9f83eb53-60d9-4b30-882e-883ae9d1ef47-00001.orc:  Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850438528_871700. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s)./app/warehouse/db/data/eventdate=20210515/00031-773-c9fdedf9-eb4f-4630-83a5-4305f4662dbd-00001.orc:  Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850438496_871705. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s)./app/warehouse/db/data/eventdate=20210515/00034-776-eb151fed-ce23-4ff8-a2d2-18c39826a91d-00001.orc:  Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850438480_871704. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s)./app/warehouse/db/data/eventdate=20210516/00002-32558-22756b91-fe4d-43c4-9470-4369d168bee9-00001.orc:  Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850023616_899554. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s)./app/warehouse/db/data/eventdate=20210516/00004-32560-c8192106-f0ba-4d57-8642-c8fc23939598-00001.orc:  Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850023584_899557. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s)./app/warehouse/db/data/eventdate=20210516/00005-32561-96ea33e1-5073-4d5a-98f4-730d3e90d5a5-00001.orc:  Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850023536_899565. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s)./app/warehouse/db/data/eventdate=20210516/00008-32564-9039e48b-620d-416a-8cb4-d3bdc4b0e3b4-00001.orc:  Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850023488_899569. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s)./app/warehouse/db/data/eventdate=20210516/00011-32567-92b48f62-42db-45da-8e18-6d1fa9d388f4-00001.orc:  Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850023632_899553. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s)./app/warehouse/db/data/eventdate=20210516/00013-32569-df9e2f0e-0692-4a84-a533-a7be15387b2f-00001.orc:  Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850023712_899546. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s)./app/warehouse/db/data/eventdate=20210516/00014-32570-e42b7b90-e0f4-49fd-81bd-d8f659714271-00001.orc:  Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850023472_899573. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s)./app/warehouse/db/data/eventdate=20210516/00017-32573-256ac3a6-f376-4efc-99b9-8963207725b8-00001.orc:  Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850023568_899560. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s)./app/warehouse/db/data/eventdate=20210516/00020-32576-821aeae6-a95b-48d1-b6e0-705321b61841-00001.orc:  Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850023680_899550. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s)./app/warehouse/db/data/eventdate=20210516/00021-32577-49044a0b-368a-442c-a6de-c9fa7f671382-00001.orc:  Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850023520_899567. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s)./app/warehouse/db/data/eventdate=20210516/00022-32578-1667fa90-ed74-4883-8eca-f259226e1486-00001.orc:  Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850023552_899562. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s)./app/warehouse/db/data/eventdate=20210516/00024-32580-3b03fcb0-ff13-4c70-a777-bc1bc944c67e-00001.orc:  Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850023504_899566. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s)./app/warehouse/db/data/eventdate=20210516/00026-32582-80ae351e-3bcd-48bd-89bb-5e72e95a3cd8-00001.orc:  Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850023456_899572. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s)./app/warehouse/db/data/eventdate=20210516/00034-20524-aa2e7a9a-4aaa-4c4d-be53-fe5e45984b51-00001.orc:  Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850163360_890520. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s)./app/warehouse/db/data/eventdate=20210516/00049-20539-277be8a4-ed39-469f-86c0-4e1044a21b37-00001.orc:  Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850163312_890525. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s)./app/warehouse/db/data/eventdate=20210516/00053-20543-f04e67ea-a60d-4430-a263-dac69b52911e-00001.orc:  Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850163280_890524. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s)./app/warehouse/db/data/eventdate=20210516/00056-20546-1f18e9f0-b70b-4952-af4c-8a3b4658b5fb-00001.orc:  Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850163136_890534. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s)./app/warehouse/db/data/eventdate=20210516/00060-20550-70622ca7-9527-477c-ab8b-6fd88817afbe-00001.orc:  Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850163328_890522. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s)./app/warehouse/db/data/eventdate=20210516/00066-20556-57201abf-cc0a-43e0-b5a7-3195602e5e20-00001.orc:  Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850163344_890519. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s)./app/warehouse/db/data/eventdate=20210516/00069-20559-f133cea8-686d-44f9-b3d9-ee1e3e13ea68-00001.orc:  Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850163376_890518. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s)./app/warehouse/db/data/eventdate=20210516/00072-20562-f13ac26e-87ed-42ae-a41a-e7d7a26372e9-00001.orc:  Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850163264_890526. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).Status: HEALTHY Number of data-nodes:	9 Number of racks:		1 Total dirs:			8858 Total symlinks:		0Replicated Blocks: Total size:	2240783571508 B (Total open files size: 18753 B) Total files:	209261 (Files currently being written: 3) Total blocks (validated):	215843 (avg. block size 10381543 B) (Total open file blocks (not validated): 3) Minimally replicated blocks:	215843 (100.0 %) Over-replicated blocks:	0 (0.0 %) Under-replicated blocks:	2 (9.2659943E-4 %) Mis-replicated blocks:		0 (0.0 %) Default replication factor:	3 Average block replication:	3.0000556 Missing blocks:		0 Corrupt blocks:		0 Missing replicas:		2 (3.088598E-4 %)Erasure Coded Block Groups: Total size:	3742386435459 B (Total open files size: 20054224 B) Total files:	162106 (Files currently being written: 112) Total block groups (validated):	163330 (avg. block group size 22913037 B) (Total open file block groups (not validated): 109) Minimally erasure-coded block groups:	163330 (100.0 %) Over-erasure-coded block groups:	1 (6.122574E-4 %) Under-erasure-coded block groups:	40 (0.024490295 %) Unsatisfactory placement block groups:	0 (0.0 %) Average block group size:	5.6923347 Missing block groups:		0 Corrupt block groups:		0 Missing internal blocks:	46 (0.004947444 %)FSCK ended at Wed May 26 22:48:38 PDT 2021 in 16072 millisecondsThe filesystem under path '/' is HEALTHY

ChethanYM · Answer

Hi,

Under replicated blocks mean these are the blocks that haven't meet their target replication for the file they belong to.

Can you use below script to fix this and see if this helps:

su - <$hdfs_user>

bash-4.1$ hdfs fsck / | grep 'Under replicated' | awk -F':' '{print $1}' >> /tmp/under_replicated_files

-bash-4.1$ for hdfsfile in `cat /tmp/under_replicated_files`; do echo "Fixing $hdfsfile :" ; hadoop fs -setrep 3 $hdfsfile; done

Please check this below cloudera community article for the details:

https://community.cloudera.com/t5/Community-Articles/Fix-Under-replicated-blocks-in-HDFS-manually/ta-p/244746

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.