Support Questions

Find answers, ask questions, and share your expertise

HBase Replication Error - ERROR: No meta znode available

avatar
New Contributor

hello.

 

After registering the target cluster as a peer in the replication source cluster

Enabling replication on a table results in "ERROR: No meta znode available" error.

 

>list_peers

 PEER_ID CLUSTER_KEY ENDPOINT_CLASSNAME STATE REPLICATE_ALL NAMESPACES TABLE_CFS BANDWIDTH SERIAL

 A_dev1 A-hadoop-dev01,A-hadoop-dev02,A-hadoop-dev03:2181:/hbase-unsecure ENABLED true 0 false

 Indexer_myIndexer un01,nn01,nn02:2181:/ngdata/sep/hbase-slave/Indexer_myIndexer ENABLED true 0 false

 

hbase:032:0> enable_table_replication 'emp'

ERROR: No meta znode available

For usage try 'help "enable_table_replication"'

Took 73.6962 seconds

 

....

2023-07-20 15:25:44,586 INFO  [ReadOnlyZKClient-A-hadoop-dev01:2181,A-hadoop-dev02:2181,A-hadoop-dev03:2181@0x7a2a7492-EventThread] zookeeper.ClientCnxn: EventThread shut down for session: 0x200001a125a0611

2023-07-20 15:25:44,630 DEBUG [ReadOnlyZKClient-un01:2181,nn01:2181,nn02:2181@0x00d66502-SendThread(nn01:2181)] zookeeper.ClientCnxn: Reading reply sessionid:0x2050181ac97c8f9, packet:: clientPath:/ngdata/sep/hbase-slave/Indexer_myIndexer serverPath:/ngdata/sep/hbase-slave/Indexer_myIndexer finished:false header:: 3,8  replyHeader:: 3,90207463322,0  request:: '/ngdata/sep/hbase-slave/Indexer_myIndexer,F  response:: v{'rs,'hbaseid} 

2023-07-20 15:25:44,833 DEBUG [ReadOnlyZKClient-un01:2181,nn01:2181,nn02:2181@0x00d66502-SendThread(nn01:2181)] zookeeper.ClientCnxn: Reading reply sessionid:0x2050181ac97c8f9, packet:: clientPath:/ngdata/sep/hbase-slave/Indexer_myIndexer serverPath:/ngdata/sep/hbase-slave/Indexer_myIndexer finished:false header:: 4,8  replyHeader:: 4,90207463322,0  request:: '/ngdata/sep/hbase-slave/Indexer_myIndexer,F  response:: v{'rs,'hbaseid} 

2023-07-20 15:25:45,138 DEBUG [ReadOnlyZKClient-un01:2181,nn01:2181,nn02:2181@0x00d66502-SendThread(nn01:2181)] zookeeper.ClientCnxn: Reading reply sessionid:0x2050181ac97c8f9, packet:: clientPath:/ngdata/sep/hbase-slave/Indexer_myIndexer serverPath:/ngdata/sep/hbase-slave/Indexer_myIndexer finished:false header:: 5,8  replyHeader:: 5,90207463322,0  request:: '/ngdata/sep/hbase-slave/Indexer_myIndexer,F  response:: v{'rs,'hbaseid} 

2023-07-20 15:25:45,645 DEBUG [ReadOnlyZKClient-un01:2181,nn01:2181,nn02:2181@0x00d66502-SendThread(nn01:2181)] zookeeper.ClientCnxn: Reading reply sessionid:0x2050181ac97c8f9, packet:: clientPath:/ngdata/sep/hbase-slave/Indexer_myIndexer serverPath:/ngdata/sep/hbase-slave/Indexer_myIndexer finished:false header:: 6,8  replyHeader:: 6,90207463322,0  request:: '/ngdata/sep/hbase-slave/Indexer_myIndexer,F  response:: v{'rs,'hbaseid} 

2023-07-20 15:25:46,651 DEBUG [ReadOnlyZKClient-un01:2181,nn01:2181,nn02:2181@0x00d66502-SendThread(nn01:2181)] zookeeper.ClientCnxn: Reading reply sessionid:0x2050181ac97c8f9, packet:: clientPath:/ngdata/sep/hbase-slave/Indexer_myIndexer serverPath:/ngdata/sep/hbase-slave/Indexer_myIndexer finished:false header:: 7,8  replyHeader:: 7,90207463322,0  request:: '/ngdata/sep/hbase-slave/Indexer_myIndexer,F  response:: v{'rs,'hbaseid} 

2023-07-20 15:25:48,310 DEBUG [ReadOnlyZKClient-localhost:2181@0x4270b142-SendThread(localhost:2181)] zookeeper.ClientCnxn: Got ping response for sessionid: 0x1000004d23ce006 after 1ms

2023-07-20 15:25:48,658 DEBUG [ReadOnlyZKClient-un01:2181,nn01:2181,nn02:2181@0x00d66502-SendThread(nn01:2181)] zookeeper.ClientCnxn: Reading reply sessionid:0x2050181ac97c8f9, packet:: clientPath:/ngdata/sep/hbase-slave/Indexer_myIndexer serverPath:/ngdata/sep/hbase-slave/Indexer_myIndexer finished:false header:: 8,8  replyHeader:: 8,90207463322,0  request:: '/ngdata/sep/hbase-slave/Indexer_myIndexer,F  response:: v{'rs,'hbaseid} 

2023-07-20 15:25:48,661 DEBUG [main] client.RpcRetryingCallerImpl: =row 'emp' on table 'hbase:meta' at null, see https://s.apache.org/timeout, exception=java.io.IOException: No meta znode available

  at org.apache.hadoop.hbase.client.ZKAsyncRegistry.getMetaRegionLocation(ZKAsyncRegistry.java:139)

  at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getMetaRegionLocation$5(ZKAsyncRegistry.java:205)

  at org.apache.hadoop.hbase.util.FutureUtils.lambda$addListener$0(FutureUtils.java:65)

  at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)

  at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)

  at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)

  at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)

  at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:174)

  at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:342)

  at java.lang.Thread.run(Thread.java:748)

 

There is another cluster with the same specifications and version, and it operates normally in that cluster.

The variable settings between the two clusters are almost identical.

 

- version : HBase 2.2.3 / Hadoop 3.1.1

 

 

How can i fix the error?

The hbase hbck2 tool is difficult to use.

2 REPLIES 2

avatar
Super Collaborator

Hi @lukepoo , The 'enable_table_replication' command verifies if the table is present on the destination and tries to create it using the peer info. So there are chances that it is failing at this step. Have you tried setting the replication_scope to 1 for this table and see if replication works considering you have created the same table on the target as well.

avatar
Community Manager

@lukepoo, Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future.  

 

 



Regards,

Vidya Sargur,
Community Manager


Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.
Learn more about the Cloudera Community: