Support Questions
Find answers, ask questions, and share your expertise

I am using hortonworks 2.4.2 my hdfs bloacks are under replicated more than 300 blocks and also problem in hbase regions some regions are failed how to resolve the issue

Explorer
11 REPLIES 11

Contributor
@Arul kumar

What is your cluster size? Are there any datanodes that are not functioning? Please provide more details/logs for the error you are observing on hbase side.

Explorer

My cluster size :totaly 8 nodes volume :5.7 TB

Master mem size:128GB, slaves each node:64 GB Default replication:3

My problem is each nodes are not balanced

Total dfs used:200 GB,Non dfs used:512 GB

NOTE:all the data nodes are healthy

Contributor

"/user/hdfs/.staging/job_1485198116716_0827/libjars/slf4j-api-1.6.1.jar: Under replicated BP-1853671431-172.31.1.228-1441814978539:blk_1080194405_6456388. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s) and 0 decommissioning replica(s)."

This indicates there are 6 replicas available but the replication factor for the specified block is 10. Since there are not enough datanodes in your cluster to meet the replication factor of 10, it shows them as underreplicated blocks. Also, Please provide more details/logs for the error you are observing on hbase side.

Explorer

Take a look in Ambari (if you're running Ambari); under the HDFS service, what does it say regarding "DataNodes" and "DataNodes status"?

Explorer

datanode.png kindly find the datanode status snap,it is rinning normally but under replicated block morethan 250

@Arul kumar

it seem NN is having only 1.6 GB RAM, i think the slowness on underlying storage might be causing this issue.

Super Mentor

@Arul kumar

Please try running the following HDFS commands to explicitly replicate those blocks:

# hdfs fsck / | grep 'Under replicated' | awk -F':' '{print $1}' >> /tmp/all_under_replicated_files 

# for hdfsfile in `cat /tmp/all_under_replicated_files`; do echo "Fixing $hdfsfile :" ;  hadoop fs -setrep 3 $hdfsfile; done

.

Super Mentor

@Arul kumar

Also, Regarding the hbase regions server failure, Can you please share the logs to see if ther are any ERRORS/WARNING.

Explorer

DEBUG [1678582822@qtp-59433821-6382] client.ConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=hbase:meta, metaLocation=, attempt=1 of 35 failed; retrying after sleep of 200 because: HRegionInfo was null in SocialMediaInsightsDetail_New, row=keyvalues={SocialMediaInsightsDetail_New,99999999,1485087298913.d8af4925ac301421a8aa33744d4c8055./info:seqnumDuringOpen/1485180641494/Put/vlen=8/seqid=0, SocialMediaInsightsDetail_New,99999999,1485087298913.d8af4925ac301421a8aa33744d4c8055./info:server/1485180641494/Put/vlen=13/seqid=0, SocialMediaInsightsDetail_New,99999999,1485087298913.d8af4925ac301421a8aa33744d4c8055./info:serverstartcode/1485180641494/Put/vlen=8/seqid=0}