Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

I am using hortonworks 2.4.2 my hdfs bloacks are under replicated more than 300 blocks and also problem in hbase regions some regions are failed how to resolve the issue

I am using hortonworks 2.4.2 my hdfs bloacks are under replicated more than 300 blocks and also problem in hbase regions some regions are failed how to resolve the issue

New Contributor
11 REPLIES 11

Re: I am using hortonworks 2.4.2 my hdfs bloacks are under replicated more than 300 blocks and also problem in hbase regions some regions are failed how to resolve the issue

Contributor
@Arul kumar

What is your cluster size? Are there any datanodes that are not functioning? Please provide more details/logs for the error you are observing on hbase side.

Re: I am using hortonworks 2.4.2 my hdfs bloacks are under replicated more than 300 blocks and also problem in hbase regions some regions are failed how to resolve the issue

New Contributor

My cluster size :totaly 8 nodes volume :5.7 TB

Master mem size:128GB, slaves each node:64 GB Default replication:3

My problem is each nodes are not balanced

Total dfs used:200 GB,Non dfs used:512 GB

NOTE:all the data nodes are healthy

Re: I am using hortonworks 2.4.2 my hdfs bloacks are under replicated more than 300 blocks and also problem in hbase regions some regions are failed how to resolve the issue

Contributor

"/user/hdfs/.staging/job_1485198116716_0827/libjars/slf4j-api-1.6.1.jar: Under replicated BP-1853671431-172.31.1.228-1441814978539:blk_1080194405_6456388. Target Replicas is 10 but found 6 live replica(s), 0 decommissioned replica(s) and 0 decommissioning replica(s)."

This indicates there are 6 replicas available but the replication factor for the specified block is 10. Since there are not enough datanodes in your cluster to meet the replication factor of 10, it shows them as underreplicated blocks. Also, Please provide more details/logs for the error you are observing on hbase side.

Re: I am using hortonworks 2.4.2 my hdfs bloacks are under replicated more than 300 blocks and also problem in hbase regions some regions are failed how to resolve the issue

New Contributor

Take a look in Ambari (if you're running Ambari); under the HDFS service, what does it say regarding "DataNodes" and "DataNodes status"?

Re: I am using hortonworks 2.4.2 my hdfs bloacks are under replicated more than 300 blocks and also problem in hbase regions some regions are failed how to resolve the issue

New Contributor

datanode.png kindly find the datanode status snap,it is rinning normally but under replicated block morethan 250

Re: I am using hortonworks 2.4.2 my hdfs bloacks are under replicated more than 300 blocks and also problem in hbase regions some regions are failed how to resolve the issue

@Arul kumar

it seem NN is having only 1.6 GB RAM, i think the slowness on underlying storage might be causing this issue.

Re: I am using hortonworks 2.4.2 my hdfs bloacks are under replicated more than 300 blocks and also problem in hbase regions some regions are failed how to resolve the issue

Super Mentor

@Arul kumar

Please try running the following HDFS commands to explicitly replicate those blocks:

# hdfs fsck / | grep 'Under replicated' | awk -F':' '{print $1}' >> /tmp/all_under_replicated_files 

# for hdfsfile in `cat /tmp/all_under_replicated_files`; do echo "Fixing $hdfsfile :" ;  hadoop fs -setrep 3 $hdfsfile; done

.

Re: I am using hortonworks 2.4.2 my hdfs bloacks are under replicated more than 300 blocks and also problem in hbase regions some regions are failed how to resolve the issue

Super Mentor

@Arul kumar

Also, Regarding the hbase regions server failure, Can you please share the logs to see if ther are any ERRORS/WARNING.

Re: I am using hortonworks 2.4.2 my hdfs bloacks are under replicated more than 300 blocks and also problem in hbase regions some regions are failed how to resolve the issue

New Contributor

DEBUG [1678582822@qtp-59433821-6382] client.ConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=hbase:meta, metaLocation=, attempt=1 of 35 failed; retrying after sleep of 200 because: HRegionInfo was null in SocialMediaInsightsDetail_New, row=keyvalues={SocialMediaInsightsDetail_New,99999999,1485087298913.d8af4925ac301421a8aa33744d4c8055./info:seqnumDuringOpen/1485180641494/Put/vlen=8/seqid=0, SocialMediaInsightsDetail_New,99999999,1485087298913.d8af4925ac301421a8aa33744d4c8055./info:server/1485180641494/Put/vlen=13/seqid=0, SocialMediaInsightsDetail_New,99999999,1485087298913.d8af4925ac301421a8aa33744d4c8055./info:serverstartcode/1485180641494/Put/vlen=8/seqid=0}

Don't have an account?
Coming from Hortonworks? Activate your account here