Member since
07-20-2023
2
Posts
0
Kudos Received
0
Solutions
07-22-2023
05:59 AM
hello. I am trying to CopyTable 5 minutes or 1 hour of data from a 15TB HBase Table to another cluster. But then I get hbase.ipc.CallTimeoutException and the transfer keeps failing. #1688313600000 ('2023-07-03 01:00:00') - 1688313900000 ('2023-07-03 01:05:00') : 5 minutes hbase org.apache.hadoop.hbase.mapreduce.CopyTable -Dhbase.zookeeper.quorum=A01,B01,C02 --peer.adr=dev01dev02,dev03:2181:/hbase-unsecure --starttime=1688313600000 --endtime=1688313900000 'BIG_EMP' #1688313600000 ('2023-07-03 01:00:00') - 1688317200000 ('2023-07-03 02:00:00') hbase org.apache.hadoop.hbase.mapreduce.CopyTable -Dhbase.zookeeper.quorum=A01,B01,C02 --peer.adr=dev01dev02,dev03:2181:/hbase-unsecure --starttime=1688313600000 --endtime=1688317200000 'BIG_EMP' 2023-07-20 15:56:05,731 INFO [main] mapreduce.Job: map 1% reduce 0% 2023-07-20 15:56:08,787 INFO [main] mapreduce.Job: map 2% reduce 0% 2023-07-20 15:56:12,843 INFO [main] mapreduce.Job: map 3% reduce 0% 2023-07-20 15:58:05,093 INFO [main] mapreduce.Job: Task Id : attempt_1670552835275_0148_m_000003_0, Status : FAILED Error: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=16, exceptions: 2023-07-20T06:58:04.295Z, java.net.SocketTimeoutException: callTimeout=60000, callDuration=60105: Call to A02/17.91.172.237:16020 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call[id=8,methodName=Scan], waitTime=60004, rpcTimeout=60000 row 'U022|11|1|230519-STL-131Ah-B-spl-30oC-WLTP-8th-cycle-B22-11-12-15-re' on table 'LDSM_CDGS_DATA' at region=xjfdjsfdjksdfkjfsd,1686849006313.1e8adfc0af2cf791361126e24822e63c., hostname=A02,16020,1689816082571, seqNum=860075 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.throwEnrichedException(RpcRetryingCallerWithReadReplicas.java:299) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:251) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:192) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:267) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:435) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:310) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:595) Settings related to CallTimeout are as follows. hbase.rpc.timeout=120000 hbase.client.scanner.timeout.period=120000 In this case, should I increase the timeout and test it? Or is there another way to act?
... View more
Labels:
- Labels:
-
Apache HBase
07-22-2023
05:59 AM
hello. After registering the target cluster as a peer in the replication source cluster Enabling replication on a table results in "ERROR: No meta znode available" error. >list_peers PEER_ID CLUSTER_KEY ENDPOINT_CLASSNAME STATE REPLICATE_ALL NAMESPACES TABLE_CFS BANDWIDTH SERIAL A_dev1 A-hadoop-dev01,A-hadoop-dev02,A-hadoop-dev03:2181:/hbase-unsecure ENABLED true 0 false Indexer_myIndexer un01,nn01,nn02:2181:/ngdata/sep/hbase-slave/Indexer_myIndexer ENABLED true 0 false hbase:032:0> enable_table_replication 'emp' ERROR: No meta znode available For usage try 'help "enable_table_replication"' Took 73.6962 seconds .... 2023-07-20 15:25:44,586 INFO [ReadOnlyZKClient-A-hadoop-dev01:2181,A-hadoop-dev02:2181,A-hadoop-dev03:2181@0x7a2a7492-EventThread] zookeeper.ClientCnxn: EventThread shut down for session: 0x200001a125a0611 2023-07-20 15:25:44,630 DEBUG [ReadOnlyZKClient-un01:2181,nn01:2181,nn02:2181@0x00d66502-SendThread(nn01:2181)] zookeeper.ClientCnxn: Reading reply sessionid:0x2050181ac97c8f9, packet:: clientPath:/ngdata/sep/hbase-slave/Indexer_myIndexer serverPath:/ngdata/sep/hbase-slave/Indexer_myIndexer finished:false header:: 3,8 replyHeader:: 3,90207463322,0 request:: '/ngdata/sep/hbase-slave/Indexer_myIndexer,F response:: v{'rs,'hbaseid} 2023-07-20 15:25:44,833 DEBUG [ReadOnlyZKClient-un01:2181,nn01:2181,nn02:2181@0x00d66502-SendThread(nn01:2181)] zookeeper.ClientCnxn: Reading reply sessionid:0x2050181ac97c8f9, packet:: clientPath:/ngdata/sep/hbase-slave/Indexer_myIndexer serverPath:/ngdata/sep/hbase-slave/Indexer_myIndexer finished:false header:: 4,8 replyHeader:: 4,90207463322,0 request:: '/ngdata/sep/hbase-slave/Indexer_myIndexer,F response:: v{'rs,'hbaseid} 2023-07-20 15:25:45,138 DEBUG [ReadOnlyZKClient-un01:2181,nn01:2181,nn02:2181@0x00d66502-SendThread(nn01:2181)] zookeeper.ClientCnxn: Reading reply sessionid:0x2050181ac97c8f9, packet:: clientPath:/ngdata/sep/hbase-slave/Indexer_myIndexer serverPath:/ngdata/sep/hbase-slave/Indexer_myIndexer finished:false header:: 5,8 replyHeader:: 5,90207463322,0 request:: '/ngdata/sep/hbase-slave/Indexer_myIndexer,F response:: v{'rs,'hbaseid} 2023-07-20 15:25:45,645 DEBUG [ReadOnlyZKClient-un01:2181,nn01:2181,nn02:2181@0x00d66502-SendThread(nn01:2181)] zookeeper.ClientCnxn: Reading reply sessionid:0x2050181ac97c8f9, packet:: clientPath:/ngdata/sep/hbase-slave/Indexer_myIndexer serverPath:/ngdata/sep/hbase-slave/Indexer_myIndexer finished:false header:: 6,8 replyHeader:: 6,90207463322,0 request:: '/ngdata/sep/hbase-slave/Indexer_myIndexer,F response:: v{'rs,'hbaseid} 2023-07-20 15:25:46,651 DEBUG [ReadOnlyZKClient-un01:2181,nn01:2181,nn02:2181@0x00d66502-SendThread(nn01:2181)] zookeeper.ClientCnxn: Reading reply sessionid:0x2050181ac97c8f9, packet:: clientPath:/ngdata/sep/hbase-slave/Indexer_myIndexer serverPath:/ngdata/sep/hbase-slave/Indexer_myIndexer finished:false header:: 7,8 replyHeader:: 7,90207463322,0 request:: '/ngdata/sep/hbase-slave/Indexer_myIndexer,F response:: v{'rs,'hbaseid} 2023-07-20 15:25:48,310 DEBUG [ReadOnlyZKClient-localhost:2181@0x4270b142-SendThread(localhost:2181)] zookeeper.ClientCnxn: Got ping response for sessionid: 0x1000004d23ce006 after 1ms 2023-07-20 15:25:48,658 DEBUG [ReadOnlyZKClient-un01:2181,nn01:2181,nn02:2181@0x00d66502-SendThread(nn01:2181)] zookeeper.ClientCnxn: Reading reply sessionid:0x2050181ac97c8f9, packet:: clientPath:/ngdata/sep/hbase-slave/Indexer_myIndexer serverPath:/ngdata/sep/hbase-slave/Indexer_myIndexer finished:false header:: 8,8 replyHeader:: 8,90207463322,0 request:: '/ngdata/sep/hbase-slave/Indexer_myIndexer,F response:: v{'rs,'hbaseid} 2023-07-20 15:25:48,661 DEBUG [main] client.RpcRetryingCallerImpl: =row 'emp' on table 'hbase:meta' at null, see https://s.apache.org/timeout, exception=java.io.IOException: No meta znode available at org.apache.hadoop.hbase.client.ZKAsyncRegistry.getMetaRegionLocation(ZKAsyncRegistry.java:139) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getMetaRegionLocation$5(ZKAsyncRegistry.java:205) at org.apache.hadoop.hbase.util.FutureUtils.lambda$addListener$0(FutureUtils.java:65) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:174) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:342) at java.lang.Thread.run(Thread.java:748) There is another cluster with the same specifications and version, and it operates normally in that cluster. The variable settings between the two clusters are almost identical. - version : HBase 2.2.3 / Hadoop 3.1.1 How can i fix the error? The hbase hbck2 tool is difficult to use.
... View more
Labels:
- Labels:
-
Apache HBase