Skip to main content
mike_bronson7
mike_bronson7
Gold ⭐️⭐️⭐️
October 26, 2019
Solved

under-replicated blocks + why we get this warning on new scratch installtion?

We installed new ambari cluster with the following details ( we moved to redhat 7.5 instead 7.2 )

 

Redhat – 7.5
HDP version – 2.6.4
Ambari – 2.6.2

 

After we complete the installation , we notice about very strange behavior ( please note that this is new cluster )

 

On HDFS status summary, I see the following messages about under-replicated blocks

 

We see under replicated blocks is 12 ( while its should be 0 on new installation )

 

Any suggestion – why this ?

I just want to say that this behavior not appears on redhat 7.2

Shelton
SheltonAnswer
Gold ⭐️⭐️⭐️
October 27, 2019

@mike_bronson7 

Surely you can use that hdfs fsck / -delete but remember it will be put in the trash !!! 

View in thread

6 replies

Shelton
Shelton
Gold ⭐️⭐️⭐️
October 26, 2019

@mike_bronson7 

Under replicated blocks

There are a couple of potential source of the problem that triggers this alert! The HDP versions earlier than HDP 3.x all use the standard default 3 replication factor for reasons you know well , the ability to rebuild the data in whatever case as opposed to the new Erasure coding policies in Hadoop 3.0.

Secondly, the cluster will rebalance itself if you gave it time 🙂 

Having said that the first question is how many data nodes were set up in this new cluster and did you enable rack awareness? 


This usually means that some files are “asking” for a specific number of target replicas that are not present or not being able to get the replica. So the question is, how i know which files are asking for a number of replicas that are not available?

The first option is use hdfs fsck:

$ hdfs fsck / -storagepolicies

****** **************output *********************
Connecting to namenode via http://xxx.com:50070/fsck?ugi=hdfs&storagepolicies=1&path=%2F
FSCK started by hdfs (auth:SIMPLE) from /192.168.0.94 for path / at Sat Oct 26 23:03:24 CEST 2019

/user/zeppelin/notebook/2EC24FF9U/note.json:
Under replicated BP-2067995211-192.168.0.101-1537740712051:blk_1073751507_10767.
Target Replicas is 3 but found 1 live replica(s), 0 decommissioned replica(s) and 0 decommissioning replica(s).

******

Change the replication

$ hdfs dfs -setrep -w 1 /user/zeppelin/notebook/2EC24FF9U/note.json
Replication 1 set: /user/zeppelin/notebook/2EC24FF9U/note.json
Waiting for /user/zeppelin/notebook/2EC24FF9U/note.json ... done


You also need to check dfs.replication in hdfs-site.xml the default is configured to be 3. Note that it turns out that if you upload files through Ambari, the file actually gets the replication factor of 3.

HTH 

Translate
    mike_bronson7
    mike_bronson7Author
    Gold ⭐️⭐️⭐️
    October 27, 2019

    Dear Shelton

    this are the results that we get from

     hdfs fsck / -storagepolicies


    FSCK started by hdfs (auth:SIMPLE) from /192.9.200.217 for path / at Sun Oct 27 05:49:31 UTC 2019
    ..................
    /hdp/apps/2.6.4.0-91/hive/hive.tar.gz: CORRUPT blockpool BP-2095386762-192.9.201.8-1571956239762 block blk_1073741831

    /hdp/apps/2.6.4.0-91/hive/hive.tar.gz: MISSING 1 blocks of total size 106475099 B..
    /hdp/apps/2.6.4.0-91/mapreduce/hadoop-streaming.jar: CORRUPT blockpool BP-2095386762-192.9.201.8-1571956239762 block blk_1073741834

    /hdp/apps/2.6.4.0-91/mapreduce/hadoop-streaming.jar: MISSING 1 blocks of total size 105758 B..
    /hdp/apps/2.6.4.0-91/mapreduce/mapreduce.tar.gz: CORRUPT blockpool BP-2095386762-192.9.201.8-1571956239762 block blk_1073741825

    /hdp/apps/2.6.4.0-91/mapreduce/mapreduce.tar.gz: CORRUPT blockpool BP-2095386762-192.9.201.8-1571956239762 block blk_1073741826

    /hdp/apps/2.6.4.0-91/mapreduce/mapreduce.tar.gz: MISSING 2 blocks of total size 212360343 B..
    /hdp/apps/2.6.4.0-91/pig/pig.tar.gz: CORRUPT blockpool BP-2095386762-192.9.201.8-1571956239762 block blk_1073741829

    /hdp/apps/2.6.4.0-91/pig/pig.tar.gz: CORRUPT blockpool BP-2095386762-192.9.201.8-1571956239762 block blk_1073741830

    /hdp/apps/2.6.4.0-91/pig/pig.tar.gz: MISSING 2 blocks of total size 135018554 B..
    /hdp/apps/2.6.4.0-91/slider/slider.tar.gz: CORRUPT blockpool BP-2095386762-192.9.201.8-1571956239762 block blk_1073741828

    /hdp/apps/2.6.4.0-91/slider/slider.tar.gz: MISSING 1 blocks of total size 47696340 B..
    /hdp/apps/2.6.4.0-91/spark2/spark2-hdp-yarn-archive.tar.gz: CORRUPT blockpool BP-2095386762-192.9.201.8-1571956239762 block blk_1073741832

    /hdp/apps/2.6.4.0-91/spark2/spark2-hdp-yarn-archive.tar.gz: CORRUPT blockpool BP-2095386762-192.9.201.8-1571956239762 block blk_1073741833

    /hdp/apps/2.6.4.0-91/spark2/spark2-hdp-yarn-archive.tar.gz: MISSING 2 blocks of total size 189992674 B..
    /hdp/apps/2.6.4.0-91/tez/tez.tar.gz: CORRUPT blockpool BP-2095386762-192.9.201.8-1571956239762 block blk_1073741827

    /hdp/apps/2.6.4.0-91/tez/tez.tar.gz: MISSING 1 blocks of total size 53236968 B......
    /user/ambari-qa/.staging/job_1571958926657_0001/job.jar: Under replicated BP-2095386762-192.9.201.8-1571956239762:blk_1073741864_1131. Target Replicas is 10 but found 5 live replica(s), 0 decommissioned replica(s) and 0 decommissioning replica(s).
    .
    /user/ambari-qa/.staging/job_1571958926657_0001/job.split: Under replicated BP-2095386762-192.9.201.8-1571956239762:blk_1073741865_1132. Target Replicas is 10 but found 5 live replica(s), 0 decommissioned replica(s) and 0 decommissioning replica(s).
    ...Status: CORRUPT

     

    yes we check the replication factor - yes its 3

     

    based on that results , can we just delete the corrupted blocks?

    Translate
    Michael-Bronson
    mike_bronson7
    mike_bronson7Author
    Gold ⭐️⭐️⭐️
    October 27, 2019

    may I return to my first question

    until using redhat 7.2 , every thing was ok , after each scratch installation we never seen that

    but when we jump to redhat 7.5

    then every cluster that created was with corrupted files - any HINT - why ?

    Translate
    Michael-Bronson
    Terms & ConditionsAccessibility statement