hdfs fsck command returns under replicated blocks

nnandula — Fri, 28 May 2021 10:14:24 GMT

Hello,

We have a 9 node HDP 3.1.5 cluster with erasure coding enabled. The hdfs fsck command returns under replicated blocks though the filesystem under path '/' is HEALTHY.

Please advise how to reconstruct the replication for the under replicated blocks. Please find the hdfs fsck report below:

[hdfs@hostname (prod) ~] $ hdfs fsck / Connecting to namenode via http://namenode:50070/fsck?ugi=hdfs&path=%2F FSCK started by hdfs (auth:SIMPLE) from /10.19.224.62 for path / at Wed May 26 22:48:22 PDT 2021 /app/warehouse/db/data/eventdate=20210516/00000-21028-5a7dd08a-8c9e-4f62-b920-b6265292da8c-00001.orc: Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850161008_890690. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s). /app/warehouse/db/data/eventdate=20210408/00047-789-396cdf80-9b0b-4411-a6e2-a1094cfb223e-00001.orc: Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850438512_871703. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s). /app/warehouse/db/data/eventdate=20210408/00088-26191-cc73b880-2802-445f-bac6-39b3ca3f1c53-00001.orc: Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850100896_894680. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s). /app/warehouse/db/data/eventdate=20210408/00090-26193-db7533fa-dd20-4b52-b17b-83ad0b5fdd42-00001.orc: Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850100944_894675. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s). /app/warehouse/db/data/eventdate=20210408/00091-26194-5ecbd72f-3f3b-4ca1-a215-1d896a7b570f-00001.orc: Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850100960_894672. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s). /app/warehouse/db/data/eventdate=20210408/00094-26197-bfa46627-e1be-4a4d-8dd9-bd6836375ddf-00001.orc: Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850100912_894678. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s). /app/warehouse/db/data/eventdate=20210408/00113-11202-28809f0a-5a6b-4346-b7bf-ff049aee4a56-00001.orc: Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850270336_883296. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s). /app/warehouse/db/data/eventdate=20210408/00115-11204-2170dec5-7a14-4e80-a1a6-e519cd59ff22-00001.orc: Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850270352_883295. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s). /app/warehouse/db/data/eventdate=20210408/00116-11205-b7a5330a-5eb9-42d4-92fa-38e02a30a037-00001.orc: Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850270368_883292. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s). /app/warehouse/db/data/eventdate=20210515/00021-763-9f83eb53-60d9-4b30-882e-883ae9d1ef47-00001.orc: Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850438528_871700. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s). /app/warehouse/db/data/eventdate=20210515/00031-773-c9fdedf9-eb4f-4630-83a5-4305f4662dbd-00001.orc: Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850438496_871705. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s). /app/warehouse/db/data/eventdate=20210515/00034-776-eb151fed-ce23-4ff8-a2d2-18c39826a91d-00001.orc: Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850438480_871704. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s). /app/warehouse/db/data/eventdate=20210516/00002-32558-22756b91-fe4d-43c4-9470-4369d168bee9-00001.orc: Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850023616_899554. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s). /app/warehouse/db/data/eventdate=20210516/00004-32560-c8192106-f0ba-4d57-8642-c8fc23939598-00001.orc: Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850023584_899557. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s). /app/warehouse/db/data/eventdate=20210516/00005-32561-96ea33e1-5073-4d5a-98f4-730d3e90d5a5-00001.orc: Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850023536_899565. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s). /app/warehouse/db/data/eventdate=20210516/00008-32564-9039e48b-620d-416a-8cb4-d3bdc4b0e3b4-00001.orc: Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850023488_899569. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s). /app/warehouse/db/data/eventdate=20210516/00011-32567-92b48f62-42db-45da-8e18-6d1fa9d388f4-00001.orc: Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850023632_899553. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s). /app/warehouse/db/data/eventdate=20210516/00013-32569-df9e2f0e-0692-4a84-a533-a7be15387b2f-00001.orc: Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850023712_899546. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s). /app/warehouse/db/data/eventdate=20210516/00014-32570-e42b7b90-e0f4-49fd-81bd-d8f659714271-00001.orc: Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850023472_899573. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s). /app/warehouse/db/data/eventdate=20210516/00017-32573-256ac3a6-f376-4efc-99b9-8963207725b8-00001.orc: Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850023568_899560. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s). /app/warehouse/db/data/eventdate=20210516/00020-32576-821aeae6-a95b-48d1-b6e0-705321b61841-00001.orc: Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850023680_899550. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s). /app/warehouse/db/data/eventdate=20210516/00021-32577-49044a0b-368a-442c-a6de-c9fa7f671382-00001.orc: Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850023520_899567. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s). /app/warehouse/db/data/eventdate=20210516/00022-32578-1667fa90-ed74-4883-8eca-f259226e1486-00001.orc: Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850023552_899562. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s). /app/warehouse/db/data/eventdate=20210516/00024-32580-3b03fcb0-ff13-4c70-a777-bc1bc944c67e-00001.orc: Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850023504_899566. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s). /app/warehouse/db/data/eventdate=20210516/00026-32582-80ae351e-3bcd-48bd-89bb-5e72e95a3cd8-00001.orc: Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850023456_899572. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s). /app/warehouse/db/data/eventdate=20210516/00034-20524-aa2e7a9a-4aaa-4c4d-be53-fe5e45984b51-00001.orc: Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850163360_890520. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s). /app/warehouse/db/data/eventdate=20210516/00049-20539-277be8a4-ed39-469f-86c0-4e1044a21b37-00001.orc: Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850163312_890525. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s). /app/warehouse/db/data/eventdate=20210516/00053-20543-f04e67ea-a60d-4430-a263-dac69b52911e-00001.orc: Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850163280_890524. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s). /app/warehouse/db/data/eventdate=20210516/00056-20546-1f18e9f0-b70b-4952-af4c-8a3b4658b5fb-00001.orc: Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850163136_890534. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s). /app/warehouse/db/data/eventdate=20210516/00060-20550-70622ca7-9527-477c-ab8b-6fd88817afbe-00001.orc: Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850163328_890522. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s). /app/warehouse/db/data/eventdate=20210516/00066-20556-57201abf-cc0a-43e0-b5a7-3195602e5e20-00001.orc: Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850163344_890519. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s). /app/warehouse/db/data/eventdate=20210516/00069-20559-f133cea8-686d-44f9-b3d9-ee1e3e13ea68-00001.orc: Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850163376_890518. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s). /app/warehouse/db/data/eventdate=20210516/00072-20562-f13ac26e-87ed-42ae-a41a-e7d7a26372e9-00001.orc: Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850163264_890526. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s). Status: HEALTHY Number of data-nodes: 9 Number of racks: 1 Total dirs: 8858 Total symlinks: 0 Replicated Blocks: Total size: 2240783571508 B (Total open files size: 18753 B) Total files: 209261 (Files currently being written: 3) Total blocks (validated): 215843 (avg. block size 10381543 B) (Total open file blocks (not validated): 3) Minimally replicated blocks: 215843 (100.0 %) Over-replicated blocks: 0 (0.0 %) Under-replicated blocks: 2 (9.2659943E-4 %) Mis-replicated blocks: 0 (0.0 %) Default replication factor: 3 Average block replication: 3.0000556 Missing blocks: 0 Corrupt blocks: 0 Missing replicas: 2 (3.088598E-4 %) Erasure Coded Block Groups: Total size: 3742386435459 B (Total open files size: 20054224 B) Total files: 162106 (Files currently being written: 112) Total block groups (validated): 163330 (avg. block group size 22913037 B) (Total open file block groups (not validated): 109) Minimally erasure-coded block groups: 163330 (100.0 %) Over-erasure-coded block groups: 1 (6.122574E-4 %) Under-erasure-coded block groups: 40 (0.024490295 %) Unsatisfactory placement block groups: 0 (0.0 %) Average block group size: 5.6923347 Missing block groups: 0 Corrupt block groups: 0 Missing internal blocks: 46 (0.004947444 %) FSCK ended at Wed May 26 22:48:38 PDT 2021 in 16072 milliseconds The filesystem under path '/' is HEALTHY

Re: hdfs fsck command returns under replicated blocks

ChethanYM — Mon, 31 May 2021 03:49:11 GMT

Hi,

Under replicated blocks mean these are the blocks that haven't meet their target replication for the file they belong to.

Can you use below script to fix this and see if this helps:

su - <$hdfs_user>

bash-4.1$ hdfs fsck / | grep 'Under replicated' | awk -F':' '{print $1}' >> /tmp/under_replicated_files

-bash-4.1$ for hdfsfile in `cat /tmp/under_replicated_files`; do echo "Fixing $hdfsfile :" ; hadoop fs -setrep 3 $hdfsfile; done

Please check this below cloudera community article for the details:

https://community.cloudera.com/t5/Community-Articles/Fix-Under-replicated-blocks-in-HDFS-manually/ta-p/244746

Re: hdfs fsck command returns under replicated blocks

nnandula — Mon, 31 May 2021 18:53:00 GMT

Hi @ChethanYM

Thank you for your response. I tried to run it, but since Erasure replication is enabled on the cluster, it errors out.

Fixing /app/warehouse/db/data/eventdate=20210408/00090-26193-db7533fa-dd20-4b52-b17b-83ad0b5fdd42-00001.orc : Did not set replication for: /app/warehouse/db/data/eventdate=20210408/00090-26193-db7533fa-dd20-4b52-b17b-83ad0b5fdd42-00001.orc, because it's an erasure coded file.

Re: hdfs fsck command returns under replicated blocks

ChethanYM — Tue, 01 Jun 2021 02:00:56 GMT

Hello,

seems to be you cannot change the originally created files with an EC policy to regular files with replication factor. The only way to do that is to re-write the files with EC policy so that they will then get created with default replication factor of 3.

Please go through the below document for more details:

https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html

Re: hdfs fsck command returns under replicated blocks

cjervis — Mon, 07 Jun 2021 19:32:24 GMT

@nnandula did you resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future.

question Re: hdfs fsck command returns under replicated blocks in Support Questions

hdfs fsck command returns under replicated blocks

Re: hdfs fsck command returns under replicated blocks

Re: hdfs fsck command returns under replicated blocks

Re: hdfs fsck command returns under replicated blocks

Re: hdfs fsck command returns under replicated blocks