Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

HBase .Meta Connection to OpenTSDB failing - OpenTSDB shutting down

HBase .Meta Connection to OpenTSDB failing - OpenTSDB shutting down

Expert Contributor

Hi All - I've kerberized HDP 2.4 (HBase version - 1.1.2.2.4.0.0-169, openTSDB version - 2.2.1) On startiong OpenTSDB, i'm getting error shown below :

What i understand (from the posts on the internet) is that this is because of connection issues between HBase & OpenTSDB

In connecting to HBase .META, the connection is failing. Any ideas on how to fix this ?

Here is the command used to start OpenTSDB

/build/tsdb tsd --zkbasedir=/hbase-secure --port=9999 --zkquorum=sandbox.hortonworks.com:2181 --cachedir=/tmp/tsd --staticroot=build/staticroot --auto-metric

Also attaching the hbase-site.xml, (the complete) opentsdb.log, opentsdb.conf

Appreciate your inputs, since i'm stuck because of this issue.

-----------------------------------------------------------------------------------------------------------------------------------------------------------------

2016-12-14 19:12:45,440 INFO [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Connecting to .META. region @ 10.0.2.15:16020 2016-12-14 19:12:45,447 INFO [AsyncHBase I/O Worker #4] HBaseClient: Channel [id: 0xd7caaba6, /10.0.2.15:53698 :> /10.0.2.15:16020] is disconnecting: [id: 0xd7caaba6, /10.0.2.15:53698 :> /10.0.2.15:16020] DISCONNECTED 2016-12-14 19:12:45,454 WARN [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Probe Exists(table="hbase:meta", key=[116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=0, region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key="")) failed org.hbase.async.NonRecoverableException: Too many attempts: Exists(table="hbase:meta", key=[116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=11, region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key="")) at org.hbase.async.HBaseClient.tooManyAttempts(HBaseClient.java:2056) [asynchbase-1.7.2.jar:na] at org.hbase.async.HBaseClient.sendRpcToRegion(HBaseClient.java:1920) [asynchbase-1.7.2.jar:na] at org.hbase.async.HBaseClient$1RetryRpc.call(HBaseClient.java:1944) [asynchbase-1.7.2.jar:na] at org.hbase.async.HBaseClient$1RetryRpc.call(HBaseClient.java:1927) [asynchbase-1.7.2.jar:na] at com.stumbleupon.async.Deferred.doCall(Deferred.java:1278) [async-1.4.0.jar:na] at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257) [async-1.4.0.jar:na] at com.stumbleupon.async.Deferred.callback(Deferred.java:1005) [async-1.4.0.jar:na] at org.hbase.async.HBaseClient$ZKClient$ZKCallback.processResult(HBaseClient.java:3645) [asynchbase-1.7.2.jar:na] at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:558) [zookeeper-3.4.5.jar:3.4.5-1392090] at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495) [zookeeper-3.4.5.jar:3.4.5-1392090] 2016-12-14 19:12:45,455 INFO [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Need to find the .META. region 2016-12-14 19:12:45,462 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x158fa6cce650288 closed 2016-12-14 19:12:45,462 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Initiating client connection, connectString=sandbox.hortonworks.com:2181 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@6e232ae3 2016-12-14 19:12:45,463 INFO [AsyncHBase Timer HBaseClient #1-SendThread(sandbox.hortonworks.com:2181)] ClientCnxn: Opening socket connection to server sandbox.hortonworks.com/10.0.2.15:2181. Will not attempt to authenticate using SASL (unknown error) 2016-12-14 19:12:45,465 INFO [AsyncHBase Timer HBaseClient #1-SendThread(sandbox.hortonworks.com:2181)] ClientCnxn: Socket connection established to sandbox.hortonworks.com/10.0.2.15:2181, initiating session 2016-12-14 19:12:45,466 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down 2016-12-14 19:12:45,481 INFO [AsyncHBase Timer HBaseClient #1-SendThread(sandbox.hortonworks.com:2181)] ClientCnxn: Session establishment complete on server sandbox.hortonworks.com/10.0.2.15:2181, sessionid = 0x158fa6cce650289, negotiated timeout = 5000 2016-12-14 19:12:45,483 INFO [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Connecting to .META. region @ 10.0.2.15:16020 2016-12-14 19:12:45,486 INFO [AsyncHBase I/O Worker #1] HBaseClient: Channel [id: 0x43b0c99f, /10.0.2.15:53700 :> /10.0.2.15:16020] is disconnecting: [id: 0x43b0c99f, /10.0.2.15:53700 :> /10.0.2.15:16020] DISCONNECTED 2016-12-14 19:12:45,

----------------------------------------------------------------------------------------------------------------------------------------------------------------

hbase-site.xmlopentsdbconf.txtopentsdblog-2.zip

3 REPLIES 3

Re: HBase .Meta Connection to OpenTSDB failing - OpenTSDB shutting down

It seems you HBase is not healthy. can you check zookeeper and regionserver logs to see if they are communicating well and running without any errors.

And , also try disabling authentication for zookeeper.

Re: HBase .Meta Connection to OpenTSDB failing - OpenTSDB shutting down

Expert Contributor

@Ankit Singhal - i'm able to start hbase & do a list or scan for the tables.

Howver, i'm getting the issue when i'm starting OpenTSDB (which tries to connect to Hbase)

------------------------------------------able to connect to hbase from terminal client -------------------

hbase(main):001:0> list

TABLE iemployee t1 tsdb tsdb-meta tsdb-tree tsdb-uid 6 row(s) in 0.3560 seconds

=> ["iemployee", "t1", "tsdb", "tsdb-meta", "tsdb-tree", "tsdb-uid"]

hbase(main):002:0> scan 'iemployee'

ROW COLUMN+CELL 1 column=insurance:dental, timestamp=1457967214359, value=metlife 1 column=insurance:health, timestamp=1457967214320, value=anthem 1 column=insurance:life, timestamp=1457967214426, value=metlife 1 column=insurance:vision, timestamp=1457967214394, value=visionOne 1 column=payroll:grade, timestamp=1457967214184, value=G16 1 column=payroll:salary, timestamp=1457967214215, value=250000.00 1 column=personal:city, timestamp=1457967214072, value=San Fransisco 1 column=personal:fname, timestamp=1457967213915, value=Mike 1 column=personal:lname, timestamp=1457967214021, value=Young 1 column=personal:zip, timestamp=1457967214149, value=12345 1 column=skills:interpersonal-rating, timestamp=1457967214292, value=medium 1 column=skills:management, timestamp=1457967214253, value=executive,creator,innovative

Re: HBase .Meta Connection to OpenTSDB failing - OpenTSDB shutting down

Expert Contributor

@Ted Yu - looping you as well, any ideas ?

Don't have an account?
Coming from Hortonworks? Activate your account here