Hello,
We have a 9 node HDP 3.1.5 cluster with erasure coding enabled. The hdfs fsck command returns under replicated blocks though the filesystem under path '/' is HEALTHY.
Please advise how to reconstruct the replication for the under replicated blocks. Please find the hdfs fsck report below:
[hdfs@hostname (prod) ~] $ hdfs fsck /
Connecting to namenode via http://namenode:50070/fsck?ugi=hdfs&path=%2F
FSCK started by hdfs (auth:SIMPLE) from /10.19.224.62 for path / at Wed May 26 22:48:22 PDT 2021
/app/warehouse/db/data/eventdate=20210516/00000-21028-5a7dd08a-8c9e-4f62-b920-b6265292da8c-00001.orc: Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850161008_890690. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).
/app/warehouse/db/data/eventdate=20210408/00047-789-396cdf80-9b0b-4411-a6e2-a1094cfb223e-00001.orc: Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850438512_871703. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).
/app/warehouse/db/data/eventdate=20210408/00088-26191-cc73b880-2802-445f-bac6-39b3ca3f1c53-00001.orc: Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850100896_894680. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).
/app/warehouse/db/data/eventdate=20210408/00090-26193-db7533fa-dd20-4b52-b17b-83ad0b5fdd42-00001.orc: Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850100944_894675. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).
/app/warehouse/db/data/eventdate=20210408/00091-26194-5ecbd72f-3f3b-4ca1-a215-1d896a7b570f-00001.orc: Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850100960_894672. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).
/app/warehouse/db/data/eventdate=20210408/00094-26197-bfa46627-e1be-4a4d-8dd9-bd6836375ddf-00001.orc: Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850100912_894678. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).
/app/warehouse/db/data/eventdate=20210408/00113-11202-28809f0a-5a6b-4346-b7bf-ff049aee4a56-00001.orc: Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850270336_883296. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).
/app/warehouse/db/data/eventdate=20210408/00115-11204-2170dec5-7a14-4e80-a1a6-e519cd59ff22-00001.orc: Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850270352_883295. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).
/app/warehouse/db/data/eventdate=20210408/00116-11205-b7a5330a-5eb9-42d4-92fa-38e02a30a037-00001.orc: Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850270368_883292. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).
/app/warehouse/db/data/eventdate=20210515/00021-763-9f83eb53-60d9-4b30-882e-883ae9d1ef47-00001.orc: Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850438528_871700. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).
/app/warehouse/db/data/eventdate=20210515/00031-773-c9fdedf9-eb4f-4630-83a5-4305f4662dbd-00001.orc: Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850438496_871705. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).
/app/warehouse/db/data/eventdate=20210515/00034-776-eb151fed-ce23-4ff8-a2d2-18c39826a91d-00001.orc: Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850438480_871704. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).
/app/warehouse/db/data/eventdate=20210516/00002-32558-22756b91-fe4d-43c4-9470-4369d168bee9-00001.orc: Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850023616_899554. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).
/app/warehouse/db/data/eventdate=20210516/00004-32560-c8192106-f0ba-4d57-8642-c8fc23939598-00001.orc: Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850023584_899557. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).
/app/warehouse/db/data/eventdate=20210516/00005-32561-96ea33e1-5073-4d5a-98f4-730d3e90d5a5-00001.orc: Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850023536_899565. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).
/app/warehouse/db/data/eventdate=20210516/00008-32564-9039e48b-620d-416a-8cb4-d3bdc4b0e3b4-00001.orc: Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850023488_899569. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).
/app/warehouse/db/data/eventdate=20210516/00011-32567-92b48f62-42db-45da-8e18-6d1fa9d388f4-00001.orc: Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850023632_899553. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).
/app/warehouse/db/data/eventdate=20210516/00013-32569-df9e2f0e-0692-4a84-a533-a7be15387b2f-00001.orc: Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850023712_899546. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).
/app/warehouse/db/data/eventdate=20210516/00014-32570-e42b7b90-e0f4-49fd-81bd-d8f659714271-00001.orc: Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850023472_899573. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).
/app/warehouse/db/data/eventdate=20210516/00017-32573-256ac3a6-f376-4efc-99b9-8963207725b8-00001.orc: Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850023568_899560. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).
/app/warehouse/db/data/eventdate=20210516/00020-32576-821aeae6-a95b-48d1-b6e0-705321b61841-00001.orc: Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850023680_899550. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).
/app/warehouse/db/data/eventdate=20210516/00021-32577-49044a0b-368a-442c-a6de-c9fa7f671382-00001.orc: Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850023520_899567. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).
/app/warehouse/db/data/eventdate=20210516/00022-32578-1667fa90-ed74-4883-8eca-f259226e1486-00001.orc: Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850023552_899562. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).
/app/warehouse/db/data/eventdate=20210516/00024-32580-3b03fcb0-ff13-4c70-a777-bc1bc944c67e-00001.orc: Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850023504_899566. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).
/app/warehouse/db/data/eventdate=20210516/00026-32582-80ae351e-3bcd-48bd-89bb-5e72e95a3cd8-00001.orc: Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850023456_899572. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).
/app/warehouse/db/data/eventdate=20210516/00034-20524-aa2e7a9a-4aaa-4c4d-be53-fe5e45984b51-00001.orc: Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850163360_890520. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).
/app/warehouse/db/data/eventdate=20210516/00049-20539-277be8a4-ed39-469f-86c0-4e1044a21b37-00001.orc: Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850163312_890525. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).
/app/warehouse/db/data/eventdate=20210516/00053-20543-f04e67ea-a60d-4430-a263-dac69b52911e-00001.orc: Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850163280_890524. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).
/app/warehouse/db/data/eventdate=20210516/00056-20546-1f18e9f0-b70b-4952-af4c-8a3b4658b5fb-00001.orc: Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850163136_890534. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).
/app/warehouse/db/data/eventdate=20210516/00060-20550-70622ca7-9527-477c-ab8b-6fd88817afbe-00001.orc: Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850163328_890522. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).
/app/warehouse/db/data/eventdate=20210516/00066-20556-57201abf-cc0a-43e0-b5a7-3195602e5e20-00001.orc: Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850163344_890519. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).
/app/warehouse/db/data/eventdate=20210516/00069-20559-f133cea8-686d-44f9-b3d9-ee1e3e13ea68-00001.orc: Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850163376_890518. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).
/app/warehouse/db/data/eventdate=20210516/00072-20562-f13ac26e-87ed-42ae-a41a-e7d7a26372e9-00001.orc: Under replicated BP-350338832-10.19.224.63-1582213708410:blk_-9223372036850163264_890526. Target Replicas is 9 but found 8 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).
Status: HEALTHY
Number of data-nodes: 9
Number of racks: 1
Total dirs: 8858
Total symlinks: 0
Replicated Blocks:
Total size: 2240783571508 B (Total open files size: 18753 B)
Total files: 209261 (Files currently being written: 3)
Total blocks (validated): 215843 (avg. block size 10381543 B) (Total open file blocks (not validated): 3)
Minimally replicated blocks: 215843 (100.0 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 2 (9.2659943E-4 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 3
Average block replication: 3.0000556
Missing blocks: 0
Corrupt blocks: 0
Missing replicas: 2 (3.088598E-4 %)
Erasure Coded Block Groups:
Total size: 3742386435459 B (Total open files size: 20054224 B)
Total files: 162106 (Files currently being written: 112)
Total block groups (validated): 163330 (avg. block group size 22913037 B) (Total open file block groups (not validated): 109)
Minimally erasure-coded block groups: 163330 (100.0 %)
Over-erasure-coded block groups: 1 (6.122574E-4 %)
Under-erasure-coded block groups: 40 (0.024490295 %)
Unsatisfactory placement block groups: 0 (0.0 %)
Average block group size: 5.6923347
Missing block groups: 0
Corrupt block groups: 0
Missing internal blocks: 46 (0.004947444 %)
FSCK ended at Wed May 26 22:48:38 PDT 2021 in 16072 milliseconds
The filesystem under path '/' is HEALTHY
