Created 07-22-2023 05:59 AM
hello.
After registering the target cluster as a peer in the replication source cluster
Enabling replication on a table results in "ERROR: No meta znode available" error.
>list_peers
PEER_ID CLUSTER_KEY ENDPOINT_CLASSNAME STATE REPLICATE_ALL NAMESPACES TABLE_CFS BANDWIDTH SERIAL
A_dev1 A-hadoop-dev01,A-hadoop-dev02,A-hadoop-dev03:2181:/hbase-unsecure ENABLED true 0 false
Indexer_myIndexer un01,nn01,nn02:2181:/ngdata/sep/hbase-slave/Indexer_myIndexer ENABLED true 0 false
hbase:032:0> enable_table_replication 'emp'
ERROR: No meta znode available
For usage try 'help "enable_table_replication"'
Took 73.6962 seconds
....
2023-07-20 15:25:44,586 INFO [ReadOnlyZKClient-A-hadoop-dev01:2181,A-hadoop-dev02:2181,A-hadoop-dev03:2181@0x7a2a7492-EventThread] zookeeper.ClientCnxn: EventThread shut down for session: 0x200001a125a0611
2023-07-20 15:25:44,630 DEBUG [ReadOnlyZKClient-un01:2181,nn01:2181,nn02:2181@0x00d66502-SendThread(nn01:2181)] zookeeper.ClientCnxn: Reading reply sessionid:0x2050181ac97c8f9, packet:: clientPath:/ngdata/sep/hbase-slave/Indexer_myIndexer serverPath:/ngdata/sep/hbase-slave/Indexer_myIndexer finished:false header:: 3,8 replyHeader:: 3,90207463322,0 request:: '/ngdata/sep/hbase-slave/Indexer_myIndexer,F response:: v{'rs,'hbaseid}
2023-07-20 15:25:44,833 DEBUG [ReadOnlyZKClient-un01:2181,nn01:2181,nn02:2181@0x00d66502-SendThread(nn01:2181)] zookeeper.ClientCnxn: Reading reply sessionid:0x2050181ac97c8f9, packet:: clientPath:/ngdata/sep/hbase-slave/Indexer_myIndexer serverPath:/ngdata/sep/hbase-slave/Indexer_myIndexer finished:false header:: 4,8 replyHeader:: 4,90207463322,0 request:: '/ngdata/sep/hbase-slave/Indexer_myIndexer,F response:: v{'rs,'hbaseid}
2023-07-20 15:25:45,138 DEBUG [ReadOnlyZKClient-un01:2181,nn01:2181,nn02:2181@0x00d66502-SendThread(nn01:2181)] zookeeper.ClientCnxn: Reading reply sessionid:0x2050181ac97c8f9, packet:: clientPath:/ngdata/sep/hbase-slave/Indexer_myIndexer serverPath:/ngdata/sep/hbase-slave/Indexer_myIndexer finished:false header:: 5,8 replyHeader:: 5,90207463322,0 request:: '/ngdata/sep/hbase-slave/Indexer_myIndexer,F response:: v{'rs,'hbaseid}
2023-07-20 15:25:45,645 DEBUG [ReadOnlyZKClient-un01:2181,nn01:2181,nn02:2181@0x00d66502-SendThread(nn01:2181)] zookeeper.ClientCnxn: Reading reply sessionid:0x2050181ac97c8f9, packet:: clientPath:/ngdata/sep/hbase-slave/Indexer_myIndexer serverPath:/ngdata/sep/hbase-slave/Indexer_myIndexer finished:false header:: 6,8 replyHeader:: 6,90207463322,0 request:: '/ngdata/sep/hbase-slave/Indexer_myIndexer,F response:: v{'rs,'hbaseid}
2023-07-20 15:25:46,651 DEBUG [ReadOnlyZKClient-un01:2181,nn01:2181,nn02:2181@0x00d66502-SendThread(nn01:2181)] zookeeper.ClientCnxn: Reading reply sessionid:0x2050181ac97c8f9, packet:: clientPath:/ngdata/sep/hbase-slave/Indexer_myIndexer serverPath:/ngdata/sep/hbase-slave/Indexer_myIndexer finished:false header:: 7,8 replyHeader:: 7,90207463322,0 request:: '/ngdata/sep/hbase-slave/Indexer_myIndexer,F response:: v{'rs,'hbaseid}
2023-07-20 15:25:48,310 DEBUG [ReadOnlyZKClient-localhost:2181@0x4270b142-SendThread(localhost:2181)] zookeeper.ClientCnxn: Got ping response for sessionid: 0x1000004d23ce006 after 1ms
2023-07-20 15:25:48,658 DEBUG [ReadOnlyZKClient-un01:2181,nn01:2181,nn02:2181@0x00d66502-SendThread(nn01:2181)] zookeeper.ClientCnxn: Reading reply sessionid:0x2050181ac97c8f9, packet:: clientPath:/ngdata/sep/hbase-slave/Indexer_myIndexer serverPath:/ngdata/sep/hbase-slave/Indexer_myIndexer finished:false header:: 8,8 replyHeader:: 8,90207463322,0 request:: '/ngdata/sep/hbase-slave/Indexer_myIndexer,F response:: v{'rs,'hbaseid}
2023-07-20 15:25:48,661 DEBUG [main] client.RpcRetryingCallerImpl: =row 'emp' on table 'hbase:meta' at null, see https://s.apache.org/timeout, exception=java.io.IOException: No meta znode available
at org.apache.hadoop.hbase.client.ZKAsyncRegistry.getMetaRegionLocation(ZKAsyncRegistry.java:139)
at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getMetaRegionLocation$5(ZKAsyncRegistry.java:205)
at org.apache.hadoop.hbase.util.FutureUtils.lambda$addListener$0(FutureUtils.java:65)
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:174)
at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:342)
at java.lang.Thread.run(Thread.java:748)
There is another cluster with the same specifications and version, and it operates normally in that cluster.
The variable settings between the two clusters are almost identical.
- version : HBase 2.2.3 / Hadoop 3.1.1
How can i fix the error?
The hbase hbck2 tool is difficult to use.
Created 07-25-2023 01:54 AM
Hi @lukepoo , The 'enable_table_replication' command verifies if the table is present on the destination and tries to create it using the peer info. So there are chances that it is failing at this step. Have you tried setting the replication_scope to 1 for this table and see if replication works considering you have created the same table on the target as well.
Created 07-31-2023 08:44 AM
@lukepoo, Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future.
Regards,
Vidya Sargur,