HBase Replication Error - ERROR: No meta znode available

lukepoo — Sat, 22 Jul 2023 12:59:03 GMT

hello.

After registering the target cluster as a peer in the replication source cluster

Enabling replication on a table results in "ERROR: No meta znode available" error.

>list_peers

PEER_ID CLUSTER_KEY ENDPOINT_CLASSNAME STATE REPLICATE_ALL NAMESPACES TABLE_CFS BANDWIDTH SERIAL

A_dev1 A-hadoop-dev01,A-hadoop-dev02,A-hadoop-dev03:2181:/hbase-unsecure ENABLED true 0 false

Indexer_myIndexer un01,nn01,nn02:2181:/ngdata/sep/hbase-slave/Indexer_myIndexer ENABLED true 0 false

hbase:032:0> enable_table_replication 'emp'

ERROR: No meta znode available

For usage try 'help "enable_table_replication"'

Took 73.6962 seconds

....

2023-07-20 15:25:44,586 INFO [ReadOnlyZKClient-A-hadoop-dev01:2181,A-hadoop-dev02:2181,A-hadoop-dev03:2181@0x7a2a7492-EventThread] zookeeper.ClientCnxn: EventThread shut down for session: 0x200001a125a0611

2023-07-20 15:25:44,630 DEBUG [ReadOnlyZKClient-un01:2181,nn01:2181,nn02:2181@0x00d66502-SendThread(nn01:2181)] zookeeper.ClientCnxn: Reading reply sessionid:0x2050181ac97c8f9, packet:: clientPath:/ngdata/sep/hbase-slave/Indexer_myIndexer serverPath:/ngdata/sep/hbase-slave/Indexer_myIndexer finished:false header:: 3,8 replyHeader:: 3,90207463322,0 request:: '/ngdata/sep/hbase-slave/Indexer_myIndexer,F response:: v{'rs,'hbaseid}

2023-07-20 15:25:44,833 DEBUG [ReadOnlyZKClient-un01:2181,nn01:2181,nn02:2181@0x00d66502-SendThread(nn01:2181)] zookeeper.ClientCnxn: Reading reply sessionid:0x2050181ac97c8f9, packet:: clientPath:/ngdata/sep/hbase-slave/Indexer_myIndexer serverPath:/ngdata/sep/hbase-slave/Indexer_myIndexer finished:false header:: 4,8 replyHeader:: 4,90207463322,0 request:: '/ngdata/sep/hbase-slave/Indexer_myIndexer,F response:: v{'rs,'hbaseid}

2023-07-20 15:25:45,138 DEBUG [ReadOnlyZKClient-un01:2181,nn01:2181,nn02:2181@0x00d66502-SendThread(nn01:2181)] zookeeper.ClientCnxn: Reading reply sessionid:0x2050181ac97c8f9, packet:: clientPath:/ngdata/sep/hbase-slave/Indexer_myIndexer serverPath:/ngdata/sep/hbase-slave/Indexer_myIndexer finished:false header:: 5,8 replyHeader:: 5,90207463322,0 request:: '/ngdata/sep/hbase-slave/Indexer_myIndexer,F response:: v{'rs,'hbaseid}

2023-07-20 15:25:45,645 DEBUG [ReadOnlyZKClient-un01:2181,nn01:2181,nn02:2181@0x00d66502-SendThread(nn01:2181)] zookeeper.ClientCnxn: Reading reply sessionid:0x2050181ac97c8f9, packet:: clientPath:/ngdata/sep/hbase-slave/Indexer_myIndexer serverPath:/ngdata/sep/hbase-slave/Indexer_myIndexer finished:false header:: 6,8 replyHeader:: 6,90207463322,0 request:: '/ngdata/sep/hbase-slave/Indexer_myIndexer,F response:: v{'rs,'hbaseid}

2023-07-20 15:25:46,651 DEBUG [ReadOnlyZKClient-un01:2181,nn01:2181,nn02:2181@0x00d66502-SendThread(nn01:2181)] zookeeper.ClientCnxn: Reading reply sessionid:0x2050181ac97c8f9, packet:: clientPath:/ngdata/sep/hbase-slave/Indexer_myIndexer serverPath:/ngdata/sep/hbase-slave/Indexer_myIndexer finished:false header:: 7,8 replyHeader:: 7,90207463322,0 request:: '/ngdata/sep/hbase-slave/Indexer_myIndexer,F response:: v{'rs,'hbaseid}

2023-07-20 15:25:48,310 DEBUG [ReadOnlyZKClient-localhost:2181@0x4270b142-SendThread(localhost:2181)] zookeeper.ClientCnxn: Got ping response for sessionid: 0x1000004d23ce006 after 1ms

2023-07-20 15:25:48,658 DEBUG [ReadOnlyZKClient-un01:2181,nn01:2181,nn02:2181@0x00d66502-SendThread(nn01:2181)] zookeeper.ClientCnxn: Reading reply sessionid:0x2050181ac97c8f9, packet:: clientPath:/ngdata/sep/hbase-slave/Indexer_myIndexer serverPath:/ngdata/sep/hbase-slave/Indexer_myIndexer finished:false header:: 8,8 replyHeader:: 8,90207463322,0 request:: '/ngdata/sep/hbase-slave/Indexer_myIndexer,F response:: v{'rs,'hbaseid}

2023-07-20 15:25:48,661 DEBUG [main] client.RpcRetryingCallerImpl: =row 'emp' on table 'hbase:meta' at null, see https://s.apache.org/timeout, exception=java.io.IOException: No meta znode available

at org.apache.hadoop.hbase.client.ZKAsyncRegistry.getMetaRegionLocation(ZKAsyncRegistry.java:139)

at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getMetaRegionLocation$5(ZKAsyncRegistry.java:205)

at org.apache.hadoop.hbase.util.FutureUtils.lambda$addListener$0(FutureUtils.java:65)

at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)

at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)

at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)

at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)

at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:174)

at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:342)

at java.lang.Thread.run(Thread.java:748)

There is another cluster with the same specifications and version, and it operates normally in that cluster.

The variable settings between the two clusters are almost identical.

- version : HBase 2.2.3 / Hadoop 3.1.1

How can i fix the error?

The hbase hbck2 tool is difficult to use.

Re: HBase Replication Error - ERROR: No meta znode available

rki_ — Tue, 25 Jul 2023 08:54:42 GMT

Hi @lukepoo , The 'enable_table_replication' command verifies if the table is present on the destination and tries to create it using the peer info. So there are chances that it is failing at this step. Have you tried setting the replication_scope to 1 for this table and see if replication works considering you have created the same table on the target as well.

Re: HBase Replication Error - ERROR: No meta znode available

VidyaSargur — Mon, 31 Jul 2023 15:44:29 GMT

@lukepoo, Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future.

question Re: HBase Replication Error - ERROR: No meta znode available in Support Questions

HBase Replication Error - ERROR: No meta znode available

Re: HBase Replication Error - ERROR: No meta znode available

Re: HBase Replication Error - ERROR: No meta znode available