HBase Replication Error - ERROR: No meta znode available
hello.
After registering the target cluster as a peer in the replication source cluster
Enabling replication on a table results in "ERROR: No meta znode available" error.
>list_peers
PEER_ID CLUSTER_KEY ENDPOINT_CLASSNAME STATE REPLICATE_ALL NAMESPACES TABLE_CFS BANDWIDTH SERIAL
A_dev1 A-hadoop-dev01,A-hadoop-dev02,A-hadoop-dev03:2181:/hbase-unsecure ENABLED true 0 false
Indexer_myIndexer un01,nn01,nn02:2181:/ngdata/sep/hbase-slave/Indexer_myIndexer ENABLED true 0 false
hbase:032:0> enable_table_replication 'emp'
ERROR: No meta znode available
For usage try 'help "enable_table_replication"'
Took 73.6962 seconds
....
2023-07-20 15:25:44,586 INFO [ReadOnlyZKClient-A-hadoop-dev01:2181,A-hadoop-dev02:2181,A-hadoop-dev03:2181@0x7a2a7492-EventThread] zookeeper.ClientCnxn: EventThread shut down for session: 0x200001a125a0611
2023-07-20 15:25:44,630 DEBUG [ReadOnlyZKClient-un01:2181,nn01:2181,nn02:2181@0x00d66502-SendThread(nn01:2181)] zookeeper.ClientCnxn: Reading reply sessionid:0x2050181ac97c8f9, packet:: clientPath:/ngdata/sep/hbase-slave/Indexer_myIndexer serverPath:/ngdata/sep/hbase-slave/Indexer_myIndexer finished:false header:: 3,8 replyHeader:: 3,90207463322,0 request:: '/ngdata/sep/hbase-slave/Indexer_myIndexer,F response:: v{'rs,'hbaseid}
2023-07-20 15:25:44,833 DEBUG [ReadOnlyZKClient-un01:2181,nn01:2181,nn02:2181@0x00d66502-SendThread(nn01:2181)] zookeeper.ClientCnxn: Reading reply sessionid:0x2050181ac97c8f9, packet:: clientPath:/ngdata/sep/hbase-slave/Indexer_myIndexer serverPath:/ngdata/sep/hbase-slave/Indexer_myIndexer finished:false header:: 4,8 replyHeader:: 4,90207463322,0 request:: '/ngdata/sep/hbase-slave/Indexer_myIndexer,F response:: v{'rs,'hbaseid}
2023-07-20 15:25:45,138 DEBUG [ReadOnlyZKClient-un01:2181,nn01:2181,nn02:2181@0x00d66502-SendThread(nn01:2181)] zookeeper.ClientCnxn: Reading reply sessionid:0x2050181ac97c8f9, packet:: clientPath:/ngdata/sep/hbase-slave/Indexer_myIndexer serverPath:/ngdata/sep/hbase-slave/Indexer_myIndexer finished:false header:: 5,8 replyHeader:: 5,90207463322,0 request:: '/ngdata/sep/hbase-slave/Indexer_myIndexer,F response:: v{'rs,'hbaseid}
2023-07-20 15:25:45,645 DEBUG [ReadOnlyZKClient-un01:2181,nn01:2181,nn02:2181@0x00d66502-SendThread(nn01:2181)] zookeeper.ClientCnxn: Reading reply sessionid:0x2050181ac97c8f9, packet:: clientPath:/ngdata/sep/hbase-slave/Indexer_myIndexer serverPath:/ngdata/sep/hbase-slave/Indexer_myIndexer finished:false header:: 6,8 replyHeader:: 6,90207463322,0 request:: '/ngdata/sep/hbase-slave/Indexer_myIndexer,F response:: v{'rs,'hbaseid}
2023-07-20 15:25:46,651 DEBUG [ReadOnlyZKClient-un01:2181,nn01:2181,nn02:2181@0x00d66502-SendThread(nn01:2181)] zookeeper.ClientCnxn: Reading reply sessionid:0x2050181ac97c8f9, packet:: clientPath:/ngdata/sep/hbase-slave/Indexer_myIndexer serverPath:/ngdata/sep/hbase-slave/Indexer_myIndexer finished:false header:: 7,8 replyHeader:: 7,90207463322,0 request:: '/ngdata/sep/hbase-slave/Indexer_myIndexer,F response:: v{'rs,'hbaseid}
2023-07-20 15:25:48,310 DEBUG [ReadOnlyZKClient-localhost:2181@0x4270b142-SendThread(localhost:2181)] zookeeper.ClientCnxn: Got ping response for sessionid: 0x1000004d23ce006 after 1ms
2023-07-20 15:25:48,658 DEBUG [ReadOnlyZKClient-un01:2181,nn01:2181,nn02:2181@0x00d66502-SendThread(nn01:2181)] zookeeper.ClientCnxn: Reading reply sessionid:0x2050181ac97c8f9, packet:: clientPath:/ngdata/sep/hbase-slave/Indexer_myIndexer serverPath:/ngdata/sep/hbase-slave/Indexer_myIndexer finished:false header:: 8,8 replyHeader:: 8,90207463322,0 request:: '/ngdata/sep/hbase-slave/Indexer_myIndexer,F response:: v{'rs,'hbaseid}
2023-07-20 15:25:48,661 DEBUG [main] client.RpcRetryingCallerImpl: =row 'emp' on table 'hbase:meta' at null, see https://s.apache.org/timeout, exception=java.io.IOException: No meta znode available
at org.apache.hadoop.hbase.client.ZKAsyncRegistry.getMetaRegionLocation(ZKAsyncRegistry.java:139)
at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getMetaRegionLocation$5(ZKAsyncRegistry.java:205)
at org.apache.hadoop.hbase.util.FutureUtils.lambda$addListener$0(FutureUtils.java:65)
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:174)
at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:342)
at java.lang.Thread.run(Thread.java:748)
There is another cluster with the same specifications and version, and it operates normally in that cluster.
The variable settings between the two clusters are almost identical.
- version : HBase 2.2.3 / Hadoop 3.1.1
How can i fix the error?
The hbase hbck2 tool is difficult to use.
