Hello everyone,
So after I changed a configuration i.e. hbase.client.scanner.caching, restarting hbase worked fine but deploying client configuration failed. Immediately after that, hbase hbck started reporting inconsistencies. Number of inconsistencies is equal to total number of regions in hbase. Following are the inconsistencies
ERROR: Region { meta => RA_x_x_x,9ee27,1512059462720.f5d4c63ece4f64773daa21501a78c186., hdfs => null, deployed => x.x.x.x,60020,1515057853133;RA_x_x_x,9ee27,1512059462720.f5d4c63ece4f64773daa21501a78c186., replicaId => 0 } found in META, but not in HDFS, and deployed on x.x.x.x,60020,1515057853133
ERROR: Region { meta => RA_x_x_x,9a428,1513610210132.f591ee286128ba597513492d31f4dd17., hdfs => null, deployed => x.x.x.x,60020,1515057853133;RA_x_x_x,9a428,1513610210132.f591ee286128ba597513492d31f4dd17., replicaId => 0 } found in META, but not in HDFS, and deployed on x.x.x.x,60020,1515057853133The same repeated for all the tables and all the corresponding regions. I stumbled upon
https://community.cloudera.com/t5/Storage-Random-Access-HDFS/hbase-hbck-reports-inconsistency-immedi...
and tried all the things but the final solution doesnt work. Also, hbase hbck -repair doesnt do anything for me and exits with following output and does most of the commands.
root@x.x.x.x:/var/log/hbase# sudo -u hbase hbase hbck -repair
2018-01-04 14:54:50,011 INFO [main] Configuration.deprecation: fs.default.name is deprecated. Instead, use fs.defaultFS
HBaseFsck command line options: -repair
root@x.x.x.x:/var/log/hbase#
I tried various other commands but to no avail. Now i cant delete the hbase data as there are more than 50 tables are present and too much of data loss will happen. Also, all the hbase client are able to read/write data to hbase without any issue but still hbase hbck reports inconsistencies. If anyone has faced anything familiar and knows how to go around this, it would be great help.
Not sure if this is anywhere related but the deploying client configuration also fails with
+ echo 'using 90 as PRIORITY'
+ echo 'using as RUNNER_PROGRAM'
+ echo 'using as RUNNER_ARGS'
+ echo 'using /usr/sbin/update-alternatives as UPDATE_ALTERNATIVES'
+ set -x
+ set -e
+ [[ 5 == -1 ]]
+ echo 'Deploying service client configs to /etc/hbase/conf.cloudera.hbase'
+ [[ -n /opt/cloudera/parcels ]]
+ [[ /opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/hadoop-0.20-mapreduce = /opt/cloudera/parcels/* ]]
+ CDH_MR1_HOME=/opt/cloudera/parcels/CDH/lib/hadoop-0.20-mapreduce
+ [[ -n /opt/cloudera/parcels ]]
+ [[ /opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/hadoop-mapreduce = /opt/cloudera/parcels/* ]]
+ CDH_MR2_HOME=/opt/cloudera/parcels/CDH/lib/hadoop-mapreduce
+ perl -pi -e 's#{{CDH_MR1_HOME}}#/opt/cloudera/parcels/CDH/lib/hadoop-0.20-mapreduce#g' /run/cloudera-scm-agent/process/ccdeploy_hbase-conf_etchbaseconf.cloudera.hbase_-6731751354269369376/hbase-conf/__cloudera_generation__ /run/cloudera-scm-agent/process/ccdeploy_hbase-conf_etchbaseconf.cloudera.hbase_-6731751354269369376/hbase-conf/__cloudera_metadata__ /run/cloudera-scm-agent/process/ccdeploy_hbase-conf_etchbaseconf.cloudera.hbase_-6731751354269369376/hbase-conf/core-site.xml /run/cloudera-scm-agent/process/ccdeploy_hbase-conf_etchbaseconf.cloudera.hbase_-6731751354269369376/hbase-conf/hbase-env.sh /run/cloudera-scm-agent/process/ccdeploy_hbase-conf_etchbaseconf.cloudera.hbase_-6731751354269369376/hbase-conf/hbase-site.xml /run/cloudera-scm-agent/process/ccdeploy_hbase-conf_etchbaseconf.cloudera.hbase_-6731751354269369376/hbase-conf/hdfs-site.xml /run/cloudera-scm-agent/process/ccdeploy_hbase-conf_etchbaseconf.cloudera.hbase_-6731751354269369376/hbase-conf/jaas.conf /run/cloudera-scm-agent/process/ccdeploy_hbase-conf_etchbaseconf.cloudera.hbase_-6731751354269369376/hbase-conf/log4j.properties /run/cloudera-scm-agent/process/ccdeploy_hbase-conf_etchbaseconf.cloudera.hbase_-6731751354269369376/hbase-conf/ssl-client.xml
Unescaped left brace in regex is illegal here in regex; marked by <-- HERE in m/{{ <-- HERE CDH_MR1_HOME}}/ at -e line 1.
