Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

hdfs fsck command returns under replicated blocks

avatar
New Contributor

Hello,

We have a 9 node HDP 3.1.5 cluster with erasure coding enabled. The hdfs fsck command returns under replicated blocks though the filesystem under path '/' is HEALTHY.

 

Please advise how to reconstruct the replication for the under replicated blocks. Please find the hdfs fsck report below:

 

[hdfs@hostname (prod) ~] $ hdfs fsck /
Connecting to namenode via http://namenode:50070/fsck?ugi=hdfs&path=%2F
FSCK started by hdfs (auth:SIMPLE) from /10.19.224.62 for path / at Wed May 26 22:48:22 PDT 2021

/app/warehouse/db/data/eventdate=20210516/00000-21028-5a7dd08a-8c9e-4f62-b920-b6265292da8c-00001.orc:  Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850161008_890690. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).

/app/warehouse/db/data/eventdate=20210408/00047-789-396cdf80-9b0b-4411-a6e2-a1094cfb223e-00001.orc:  Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850438512_871703. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).

/app/warehouse/db/data/eventdate=20210408/00088-26191-cc73b880-2802-445f-bac6-39b3ca3f1c53-00001.orc:  Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850100896_894680. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).

/app/warehouse/db/data/eventdate=20210408/00090-26193-db7533fa-dd20-4b52-b17b-83ad0b5fdd42-00001.orc:  Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850100944_894675. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).

/app/warehouse/db/data/eventdate=20210408/00091-26194-5ecbd72f-3f3b-4ca1-a215-1d896a7b570f-00001.orc:  Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850100960_894672. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).

/app/warehouse/db/data/eventdate=20210408/00094-26197-bfa46627-e1be-4a4d-8dd9-bd6836375ddf-00001.orc:  Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850100912_894678. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).

/app/warehouse/db/data/eventdate=20210408/00113-11202-28809f0a-5a6b-4346-b7bf-ff049aee4a56-00001.orc:  Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850270336_883296. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).

/app/warehouse/db/data/eventdate=20210408/00115-11204-2170dec5-7a14-4e80-a1a6-e519cd59ff22-00001.orc:  Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850270352_883295. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).

/app/warehouse/db/data/eventdate=20210408/00116-11205-b7a5330a-5eb9-42d4-92fa-38e02a30a037-00001.orc:  Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850270368_883292. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).

/app/warehouse/db/data/eventdate=20210515/00021-763-9f83eb53-60d9-4b30-882e-883ae9d1ef47-00001.orc:  Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850438528_871700. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).

/app/warehouse/db/data/eventdate=20210515/00031-773-c9fdedf9-eb4f-4630-83a5-4305f4662dbd-00001.orc:  Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850438496_871705. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).

/app/warehouse/db/data/eventdate=20210515/00034-776-eb151fed-ce23-4ff8-a2d2-18c39826a91d-00001.orc:  Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850438480_871704. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).

/app/warehouse/db/data/eventdate=20210516/00002-32558-22756b91-fe4d-43c4-9470-4369d168bee9-00001.orc:  Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850023616_899554. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).

/app/warehouse/db/data/eventdate=20210516/00004-32560-c8192106-f0ba-4d57-8642-c8fc23939598-00001.orc:  Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850023584_899557. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).

/app/warehouse/db/data/eventdate=20210516/00005-32561-96ea33e1-5073-4d5a-98f4-730d3e90d5a5-00001.orc:  Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850023536_899565. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).

/app/warehouse/db/data/eventdate=20210516/00008-32564-9039e48b-620d-416a-8cb4-d3bdc4b0e3b4-00001.orc:  Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850023488_899569. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).

/app/warehouse/db/data/eventdate=20210516/00011-32567-92b48f62-42db-45da-8e18-6d1fa9d388f4-00001.orc:  Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850023632_899553. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).

/app/warehouse/db/data/eventdate=20210516/00013-32569-df9e2f0e-0692-4a84-a533-a7be15387b2f-00001.orc:  Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850023712_899546. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).

/app/warehouse/db/data/eventdate=20210516/00014-32570-e42b7b90-e0f4-49fd-81bd-d8f659714271-00001.orc:  Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850023472_899573. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).

/app/warehouse/db/data/eventdate=20210516/00017-32573-256ac3a6-f376-4efc-99b9-8963207725b8-00001.orc:  Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850023568_899560. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).

/app/warehouse/db/data/eventdate=20210516/00020-32576-821aeae6-a95b-48d1-b6e0-705321b61841-00001.orc:  Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850023680_899550. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).

/app/warehouse/db/data/eventdate=20210516/00021-32577-49044a0b-368a-442c-a6de-c9fa7f671382-00001.orc:  Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850023520_899567. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).

/app/warehouse/db/data/eventdate=20210516/00022-32578-1667fa90-ed74-4883-8eca-f259226e1486-00001.orc:  Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850023552_899562. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).

/app/warehouse/db/data/eventdate=20210516/00024-32580-3b03fcb0-ff13-4c70-a777-bc1bc944c67e-00001.orc:  Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850023504_899566. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).

/app/warehouse/db/data/eventdate=20210516/00026-32582-80ae351e-3bcd-48bd-89bb-5e72e95a3cd8-00001.orc:  Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850023456_899572. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).

/app/warehouse/db/data/eventdate=20210516/00034-20524-aa2e7a9a-4aaa-4c4d-be53-fe5e45984b51-00001.orc:  Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850163360_890520. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).

/app/warehouse/db/data/eventdate=20210516/00049-20539-277be8a4-ed39-469f-86c0-4e1044a21b37-00001.orc:  Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850163312_890525. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).

/app/warehouse/db/data/eventdate=20210516/00053-20543-f04e67ea-a60d-4430-a263-dac69b52911e-00001.orc:  Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850163280_890524. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).

/app/warehouse/db/data/eventdate=20210516/00056-20546-1f18e9f0-b70b-4952-af4c-8a3b4658b5fb-00001.orc:  Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850163136_890534. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).

/app/warehouse/db/data/eventdate=20210516/00060-20550-70622ca7-9527-477c-ab8b-6fd88817afbe-00001.orc:  Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850163328_890522. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).

/app/warehouse/db/data/eventdate=20210516/00066-20556-57201abf-cc0a-43e0-b5a7-3195602e5e20-00001.orc:  Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850163344_890519. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).

/app/warehouse/db/data/eventdate=20210516/00069-20559-f133cea8-686d-44f9-b3d9-ee1e3e13ea68-00001.orc:  Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850163376_890518. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).

/app/warehouse/db/data/eventdate=20210516/00072-20562-f13ac26e-87ed-42ae-a41a-e7d7a26372e9-00001.orc:  Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850163264_890526. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).


Status: HEALTHY
 Number of data-nodes:	9
 Number of racks:		1
 Total dirs:			8858
 Total symlinks:		0

Replicated Blocks:
 Total size:	2240783571508 B (Total open files size: 18753 B)
 Total files:	209261 (Files currently being written: 3)
 Total blocks (validated):	215843 (avg. block size 10381543 B) (Total open file blocks (not validated): 3)
 Minimally replicated blocks:	215843 (100.0 %)
 Over-replicated blocks:	0 (0.0 %)
 Under-replicated blocks:	2 (9.2659943E-4 %)
 Mis-replicated blocks:		0 (0.0 %)
 Default replication factor:	3
 Average block replication:	3.0000556
 Missing blocks:		0
 Corrupt blocks:		0
 Missing replicas:		2 (3.088598E-4 %)

Erasure Coded Block Groups:
 Total size:	3742386435459 B (Total open files size: 20054224 B)
 Total files:	162106 (Files currently being written: 112)
 Total block groups (validated):	163330 (avg. block group size 22913037 B) (Total open file block groups (not validated): 109)
 Minimally erasure-coded block groups:	163330 (100.0 %)
 Over-erasure-coded block groups:	1 (6.122574E-4 %)
 Under-erasure-coded block groups:	40 (0.024490295 %)
 Unsatisfactory placement block groups:	0 (0.0 %)
 Average block group size:	5.6923347
 Missing block groups:		0
 Corrupt block groups:		0
 Missing internal blocks:	46 (0.004947444 %)
FSCK ended at Wed May 26 22:48:38 PDT 2021 in 16072 milliseconds


The filesystem under path '/' is HEALTHY

 

 

4 REPLIES 4

avatar
Super Collaborator

Hi,

 

Under replicated blocks mean these are the blocks that haven't meet their target replication for the file they belong to.

Can you use below script to fix this and see if this helps:

 

su - <$hdfs_user>

 

bash-4.1$ hdfs fsck / | grep 'Under replicated' | awk -F':' '{print $1}' >> /tmp/under_replicated_files

 

-bash-4.1$ for hdfsfile in `cat /tmp/under_replicated_files`; do echo "Fixing $hdfsfile :" ;  hadoop fs -setrep 3 $hdfsfile; done

 

Please check this below cloudera community article for the details:

https://community.cloudera.com/t5/Community-Articles/Fix-Under-replicated-blocks-in-HDFS-manually/ta...

 

 

avatar
New Contributor

Hi @ChethanYM 

 

Thank you for your response. I tried to run it, but since Erasure replication is enabled on the cluster, it errors out.

Fixing /app/warehouse/db/data/eventdate=20210408/00090-26193-db7533fa-dd20-4b52-b17b-83ad0b5fdd42-00001.orc :
Did not set replication for: /app/warehouse/db/data/eventdate=20210408/00090-26193-db7533fa-dd20-4b52-b17b-83ad0b5fdd42-00001.orc, because it's an erasure coded file.

avatar
Super Collaborator

Hello,

seems to be you cannot change the originally created files with an EC policy to regular files with replication factor. The only way to do that is to re-write the files with EC policy so that they will then get created with default replication factor of 3.

 

Please go through the below document for more details:

https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html

avatar
Community Manager

@nnandula did you resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. 

 

Screen Shot 2019-08-06 at 1.54.47 PM.png

 

 


Cy Jervis, Manager, Community Program
Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.