HBase Region Replica(Read Replica) doesn't work as expected.

[#1]

Hi there,

I am working on enabling the hbase read replica but it turns out to not work as expected.

The hbase version is 1.2.0-cdh5.10.0 and I did almost exactlly as described in the following documentation.

https://www.cloudera.com/documentation/enterprise/5-10-x/topics/admin_hbase_read_replicas.html

Things go well except that I cannot get the right value from secondary region replicas after I put a new row. It seems that the WAL replication doesn't work when data is put into the primary replica and all other replicas remain empty(I put the data, waited for 1 hour, and got nothing).

hbase(main):012:0> put 'myTable', 'r2', 'myCF:c', 'v2'
0 row(s) in 0.0850 seconds

hbase(main):013:0> get 'myTable', 'r2', {CONSISTENCY => 'TIMELINE', REGION_REPLICA_ID => 2}
COLUMN                                           CELL                                                                                                                                          
0 row(s) in 0.0070 seconds

hbase(main):020:0> !desc 'myTable'
Table myTable is ENABLED 
myTable, {TABLE_ATTRIBUTES => {REGION_REPLICATION => '3'} 
COLUMN FAMILIES DESCRIPTION 
{NAME => 'myCF', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIO
NS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}

Following is my configuration

<property>
<name>hbase.regionserver.storefile.refresh.period</name>
<value>30000</value>
</property>
<property>
<name>hbase.ipc.client.allowsInterrupt</name>
<value>true</value>
<description>Whether to enable interruption of RPC threads at the client. The default value of true is
required to enable Primary RegionServers to access other RegionServers in secondary mode. </description>
</property>
<property>
<name>hbase.client.primaryCallTimeout.get</name>
<value>10</value>
</property>
<property>
<name>hbase.client.primaryCallTimeout.multiget</name>
<value>10</value>
</property>
<property>
<name>hbase.region.replica.wait.for.primary.flush</name>
<value>false</value>
</property>
<property>
<name>hbase.region.replica.replication.memstore.enabled</name>
<value>true</value>
</property>
 <property>
 <name>hbase.region.replica.replication.enabled</name>
 <value>true</value>
 </property>
 <property>
 <name>hbase.replication</name>
 <value>true</value>
 </property>

Per the CDH documentation, the writes should be replicated to secondary replicas by hbase replication feature but it didn't. A peer named 'region_replica_replication' is not found for replication.

I didn't find a way to set REGION_MEMSTORE_REPLICATION to false as well.

Please kindly share your thoughts to me. Any inputs will be helpful.

Best regards,

Ryan

[#2 · Accepted solution]

Finally I found the root cause myself.

I just checked the checkbox which is only applied to region server.

Enable Replication To Secondary Region Replicas 
hbase.region.replica.replication.enabled

It should be a defect of Cloudera Manager because this configuration is required by HMaster as well. It works when I add this configuration into 'Master Default Group / Advanced'.

Best regards,

Ryan