Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

hbase hbck reports inconsistency (No HDFS region dir) after hbase config change and restart

avatar
Explorer

Hello everyone,

 

So after I changed a configuration i.e. hbase.client.scanner.caching, restarting hbase worked fine but deploying client configuration failed. Immediately after that, hbase hbck started reporting inconsistencies. Number of inconsistencies is equal to total number of regions in hbase. Following are the inconsistencies

ERROR: Region { meta => RA_x_x_x,9ee27,1512059462720.f5d4c63ece4f64773daa21501a78c186., hdfs => null, deployed => x.x.x.x,60020,1515057853133;RA_x_x_x,9ee27,1512059462720.f5d4c63ece4f64773daa21501a78c186., replicaId => 0 } found in META, but not in HDFS, and deployed on x.x.x.x,60020,1515057853133
ERROR: Region { meta => RA_x_x_x,9a428,1513610210132.f591ee286128ba597513492d31f4dd17., hdfs => null, deployed => x.x.x.x,60020,1515057853133;RA_x_x_x,9a428,1513610210132.f591ee286128ba597513492d31f4dd17., replicaId => 0 } found in META, but not in HDFS, and deployed on x.x.x.x,60020,1515057853133

The same repeated for all the tables and all the corresponding regions. I stumbled upon

 

https://community.cloudera.com/t5/Storage-Random-Access-HDFS/hbase-hbck-reports-inconsistency-immedi...

 

and tried all the things but the final solution doesnt work. Also, hbase hbck -repair doesnt do anything for me and exits with following output and does most of the commands.

 

root@x.x.x.x:/var/log/hbase# sudo -u hbase hbase hbck -repair
2018-01-04 14:54:50,011 INFO  [main] Configuration.deprecation: fs.default.name is deprecated. Instead, use fs.defaultFS
HBaseFsck command line options: -repair
root@x.x.x.x:/var/log/hbase# 

I tried various other commands but to no avail. Now i cant delete the hbase data as there are more than 50 tables are present and too much of data loss will happen. Also, all the hbase client are able to read/write data to hbase without any issue but still hbase hbck reports inconsistencies. If anyone has faced anything familiar and knows how to go around this, it would be great help.

 

Not sure if this is anywhere related but the deploying client configuration also fails with

+ echo 'using 90 as PRIORITY'
+ echo 'using  as RUNNER_PROGRAM'
+ echo 'using  as RUNNER_ARGS'
+ echo 'using /usr/sbin/update-alternatives as UPDATE_ALTERNATIVES'
+ set -x
+ set -e
+ [[ 5 == -1 ]]
+ echo 'Deploying service client configs to /etc/hbase/conf.cloudera.hbase'
+ [[ -n /opt/cloudera/parcels ]]
+ [[ /opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/hadoop-0.20-mapreduce = /opt/cloudera/parcels/* ]]
+ CDH_MR1_HOME=/opt/cloudera/parcels/CDH/lib/hadoop-0.20-mapreduce
+ [[ -n /opt/cloudera/parcels ]]
+ [[ /opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/hadoop-mapreduce = /opt/cloudera/parcels/* ]]
+ CDH_MR2_HOME=/opt/cloudera/parcels/CDH/lib/hadoop-mapreduce
+ perl -pi -e 's#{{CDH_MR1_HOME}}#/opt/cloudera/parcels/CDH/lib/hadoop-0.20-mapreduce#g' /run/cloudera-scm-agent/process/ccdeploy_hbase-conf_etchbaseconf.cloudera.hbase_-6731751354269369376/hbase-conf/__cloudera_generation__ /run/cloudera-scm-agent/process/ccdeploy_hbase-conf_etchbaseconf.cloudera.hbase_-6731751354269369376/hbase-conf/__cloudera_metadata__ /run/cloudera-scm-agent/process/ccdeploy_hbase-conf_etchbaseconf.cloudera.hbase_-6731751354269369376/hbase-conf/core-site.xml /run/cloudera-scm-agent/process/ccdeploy_hbase-conf_etchbaseconf.cloudera.hbase_-6731751354269369376/hbase-conf/hbase-env.sh /run/cloudera-scm-agent/process/ccdeploy_hbase-conf_etchbaseconf.cloudera.hbase_-6731751354269369376/hbase-conf/hbase-site.xml /run/cloudera-scm-agent/process/ccdeploy_hbase-conf_etchbaseconf.cloudera.hbase_-6731751354269369376/hbase-conf/hdfs-site.xml /run/cloudera-scm-agent/process/ccdeploy_hbase-conf_etchbaseconf.cloudera.hbase_-6731751354269369376/hbase-conf/jaas.conf /run/cloudera-scm-agent/process/ccdeploy_hbase-conf_etchbaseconf.cloudera.hbase_-6731751354269369376/hbase-conf/log4j.properties /run/cloudera-scm-agent/process/ccdeploy_hbase-conf_etchbaseconf.cloudera.hbase_-6731751354269369376/hbase-conf/ssl-client.xml
Unescaped left brace in regex is illegal here in regex; marked by <-- HERE in m/{{ <-- HERE CDH_MR1_HOME}}/ at -e line 1.
1 REPLY 1

avatar
Explorer

After debugging further, it seems that hbase hbck is not working properly. It is taking up incorrect hbase rootdir. Excerpt from hbck output

 

2018-01-05 11:28:44,051 INFO  [main] util.HBaseFsck: Loading region directories from HDFS
ERROR: Version file does not exist in root dir hdfs://nameservice1/tmp/hbase-hbase/hbase

2018-01-05 11:28:44,162 INFO  [main] util.HBaseFsck: Loading region information from HDFS

The rootdir being picked up is /tmp/hbase-{user running hbck command}/hbase. I am not sure why hbck is checking there instead of defined rootdir '/hbase'. Maybe due to this any of the hbck options commands are not running.