Solved
Hbase Flush/writes not working for one of the column families in HBase table
Hi All,
- I have two Cloudera clusters and each cluster has one HBase table .
- I first write to cluster1 and then get my data replicated to cluster2 with WAL replication.
- Cluster1(has 3 Region servers) and cluster2(has 80 Region servers).
- I am using
org.apache.hadoop.hbase.spark.HBaseContext.bulkPut to write data to Hbase.
- My table has two column families and writes/flushes are successful for one column family in source cluster(cluster1) and the other column family is not logging any errors/warnings to Flush/write failures in region server logs.
- One weird issue here is, my writes are reaching WAL successfully(both Column Family data) and they are replicating successfully in destination cluster(cluster2) and I can see data in both Column families in destination
- whereas in source, I believe the writes are reaching WAL only, somehow it is not being picked up by memstore and hence not being flushed to one of the column families.(It works for one column family and not working for the other).
- I tried to write data to that column family manually from hbase shell (put command) and it works fine without any issues.
What I have tried so far to fix this:
- hbase hbck -details , no inconsistencies found.
- Used hbck2 tool to fix hdfs filesystem for Hbase tables/hdfs directories
- Dropped the table in source, exported a snapshot from destination cluster which has data for both column families and tried to rerun my batch job. Still the writes are going to one column family only, other one is not getting any write requests.
- tried to tune HBase write performance by changing several parameters from link below. No luck.
- Not sure if its a bug in my current Hbase version. It was working just fine for over an year now. ( Hbase version :2.1.0), CDH 6.2.1

