Hi All,
The HBase shell count command run on a table is giving a different count than the custom map reduce row counter job ran on snapshot of same table.
This is a custom row counter written by us which is similar to HBase default map reduce row counter, the only difference is that our job first creates the snapshot and then run the map reduce task on snapshot and not on directly table, after finishing the count on snapshot, it deletes the snapshot programmatically.
Snapshot creation is programmatic done through the HBaseAdmin.snapshot() API.
When we analyzed the map reduce job log, we understood that it is gathering extra regions while creating the snapshot.
From master web UI (also hbase hbck) the number of regions allocated for the tables TABLE1 and TABLE2 are 10 and 4 respectively, however in the snapshot creation logs it is showing 18 and 7 regions respectively.
The same Map Reduce job ran on same tables in prod giving a correct count which is matching with the HBase shell count command. Only in cob we are observing this erroneous behavior.
There is a one way replication enabled from prod to cob.
Also there is no filter condition added in scanner in map reduce job, it is plain select count(*) kind of query.
The command used for running the custom the map reduce counter job is –
java -cp ${HADOOP_CLASSPATH}:${HADOOP_CONF_DIR}:${HBASE_CONF_DIR} com.XXX.XXX.XXX.RowCounterCustom -libjars ${LIBJAR} TABLE1
We have observed that there are 7 .regioninfo files for table TABLE1 whereas 4 regions are only reported by hbck command. And suspect that this might be the root cause of issue. When it creates the snapshot it is gathering information from 7 regions instead of 4 regions and that might be the reason it is counting extra rows.
This issue is non producible on lower envs.
We want to know is there any bug reported in HBase where regions count not matching between number of regions reported by hbck command and number of actual .regioninfo files of table on hdfs.
Please let us know what could be the reason of not processing of appropriate number of regions leading to incorrect row count in COB.
--------------------------------------------------------------------------
Please see below for ref the description of TABLE1 –
hbase(main):001:0> desc 'TABLE1'
Table TABLE1 is ENABLED
TABLE1, {TABLE_ATTRIBUTES => {coprocessor$1 => '|org.apache.phoenix.coprocessor.ScanRegionObserver|805306366|', coprocessor$2 => '|org.apache.
phoenix.coprocessor.UngroupedAggregateRegionObserver|805306366|', coprocessor$3 => '|org.apache.phoenix.coprocessor.GroupedAggregateRegionObserver|805306366|
', coprocessor$4 => '|org.apache.phoenix.coprocessor.ServerCachingEndpointImpl|805306366|', coprocessor$5 => '|org.apache.phoenix.hbase.index.Indexer|8053063
66|index.builder=org.apache.phoenix.index.PhoenixIndexBuilder,org.apache.hadoop.hbase.index.codec.class=org.apache.phoenix.index.PhoenixIndexCodec', coproces
sor$6 => '|org.apache.hadoop.hbase.regionserver.LocalIndexSplitter|805306366|'}
COLUMN FAMILIES DESCRIPTION
{NAME => '0', DATA_BLOCK_ENCODING => 'FAST_DIFF', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '1', COMPRESSION => 'NONE', VERSIONS => '2', TTL => 'FOREVER', M
IN_VERSIONS => '0', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}
1 row in 0.2790 seconds
--------------------------------------------------------------------------
