Opentsdb Installation with Kerberos Enabled

[#1]

Hi All,

Does anybody has documentation for installing Opentsdb on Hortonworks platform with Kerberos enabled?

There is an example here to install opentsdb with Hortonworks Sandbox here: https://community.hortonworks.com/articles/4577/use-opentsdb-to-storevisualize-stock-data-on-hdp-s.h...

It does not seem to work with Kerberos, however I am able to install Opentsdb manually and made the required settings for Kerberos in the configuration file, but I am still having problems while starting the service. Somehow it is not able to read hbase-meta-region and throws following errors.

12:41:48.349 DEBUG [ClientCnxnSocketNIO.findSendablePacket] - deferring non-priming packet: clientPath:null serverPath:null finished:false header:: 0,-11  replyHeader:: 0,0,0  request:: null response:: nulluntil SASL authentication completes.
12:41:48.356 DEBUG [ClientCnxn.readResponse] - Reading reply sessionid:0x1559dca73430bca, packet:: clientPath:/hbase-secure/root-region-server serverPath:/hbase-secure/root-region-server finished:false header:: 3,4  replyHeader:: 3,240518300453,-101  request:: '/hbase-secure/root-region-server,T  response::
12:41:48.356 DEBUG [ClientCnxn.readResponse] - Reading reply sessionid:0x1559dca73430bca, packet:: clientPath:/hbase-secure/meta-region-server serverPath:/hbase-secure/meta-region-server finished:false header:: 4,4  replyHeader:: 4,240518300453,0  request:: '/hbase-secure/meta-region-server,T  response:: #ffffffff0001a726567696f6e7365727665723a3136303230ffffff87ffffffab61b48ffffff8827ffffff9a50425546a40a3464657370702d7368726b2d77322e6e6f64652e65752d776573742e6465762d657370702e696e7472616e65742e656f6e2e636f6d10ffffff947d18ffffffa8ffffffa9ffffffd9ffffffc8ffffffd72a100183,s{227633711798,227633714655,1466615355376,1466615687830,1,0,0,0,105,0,227633711798}
- Unexpected exception from downstream on [id: 0x0af73a5a, /10.66.48.100:55821 => /10.66.48.102:16020]
12:43:51.593 ERROR [RegionClient.decodeLast] - After decoding the last message on [id: 0x0af73a5a, /10.66.48.100:55821 😆 /10.66.48.102:16020], there was still some undecoded bytes in the channel's buffer (which are going to be lost):

If somebody else has faced the same error, please guide me how it should be resolved?

Thanks in Advance.

[#2]

@Hammad Ali - did you get this working ?

I'm facing similar issue.

[#3 · Accepted solution]

@Karan Alang

Yes I made it working, the first thing is that it will not work directly from Ambari atleast in version 2.1.1 because it does not provide you any support for securing it with Kerberos.

I fixed the issue by using the latest version of Opentsdb from githhub, then manually installing it, then interestingly it started working. In the older version of opentsdb the support with kerberos is not fully enabled.

You can download the opentsdb from here: https://github.com/OpenTSDB/opentsdb/releases/tag/v2.3.0RC2

[#4]

@Hammad Ali - i seem to be having issues in trying to install OpenTSDB on HDP 2.4,

can you pls. help confirm the steps ..

what i've done so far ->

-> Downloaded jar from https://github.com/OpenTSDB/opentsdb/releases/tag/v2.3.0RC2

- > run gunzip & tar -xvf opentsdb-2.3.0-RC2.tar

-> cd opentsdb-2.3.0-RC2.tar

-> ./build.sh

which gives me the following error -

make[1]: Entering directory `/root/opentsdb-2.3.0-RC2/build' /usr/lib/jvm/java/bin/java -cp third_party/javacc/javacc-6.1.2.jar javacc -OUTPUT_DIRECTORY:./src/net/opentsdb/query/expression/parser ../src/parser.jj; echo PWD: `pwd`; Error: Could not find or load main class javacc PWD: /root/opentsdb-2.3.0-RC2/build

Thanks for help in advance !

[#5]

@Hammad Ali

i used the following command to download the version 2.2.1

This went through fine, however - when i do 'Add Service' on Ambari, i don't see OpenTSDB as an option

git clone https://github.com/OpenTSDB/opentsdb.git /var/lib/ambari-server/resources/stacks/HDP/$VERSION/services/OPENTSDB

Alternately, when i use this (the earlier url specified in this link -https://community.hortonworks.com/articles/4577/use-opentsdb-to-storevisualize-stock-data-on-hdp-s.h...), it goes through fine..

however, as you mentioned, i need OpenTSDB v 2.2.0 and later.

Older command (using v2.0.1) :

sudo git clone https://github.com/hortonworks-gallery/ambari-opentsdb-service.git /var/lib/ambari-server/resources/stacks/HDP/$VERSION/services/OPENTSDB

Any ideas ?

Alternately, what are the steps you followed to download v2.3.0 & install OpenTSDB, and then add to Ambari ?

[#6]

@Hammad Ali - pls let me know how you installed OpenTSDB (which version) & also, how you added the OPENTSDB service to Ambari

Is there a document available for this ?

[#7]

@Hammad Ali

Seems like i was able to fix the installation issue, however on starting OpenTSDB, the following error was thrown.

Any ideas ?

------------------------

2016-12-14 00:51:16,030 INFO [AsyncHBase I/O Worker #3] HBaseClient: Channel [id: 0x0d57d67b, /10.0.2.15:37370 😆 /10.0.2.15:16020] is disconnecting: [id: 0x0d57d67b, /10.0.2.15:37370 😆 /10.0.2.15:16020] DISCONNECTED 2016-12-14 00:51:16,032 WARN [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Probe Exists(table="hbase:meta", key=[116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=0, region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key="")) failed org.hbase.async.NonRecoverableException: Too many attempts: Exists(table="hbase:meta", key=[116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], family=null, qualifiers=null, attempt=11, region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key="")) at org.hbase.async.HBaseClient.tooManyAttempts(HBaseClient.java:2056) [asynchbase-1.7.2.jar:na] at org.hbase.async.HBaseClient.sendRpcToRegion(HBaseClient.java:1920) [asynchbase-1.7.2.jar:na] at org.hbase.async.HBaseClient$1RetryRpc.call(HBaseClient.java:1944) [asynchbase-1.7.2.jar:na] at org.hbase.async.HBaseClient$1RetryRpc.call(HBaseClient.java:1927) [asynchbase-1.7.2.jar:na] at com.stumbleupon.async.Deferred.doCall(Deferred.java:1278) [async-1.4.0.jar:na] at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257) [async-1.4.0.jar:na] at com.stumbleupon.async.Deferred.callback(Deferred.java:1005) [async-1.4.0.jar:na] at org.hbase.async.HBaseClient$ZKClient$ZKCallback.processResult(HBaseClient.java:3645) [asynchbase-1.7.2.jar:na] at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:558) [zookeeper-3.4.5.jar:3.4.5-1392090] at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495) [zookeeper-3.4.5.jar:3.4.5-1392090] 2016-12-14 00:51:16,033 INFO [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Need to find the .META. region 2016-12-14 00:51:16,037 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x158fa6cce65008d closed 2016-12-14 00:51:16,037 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Initiating client connection, connectString=sandbox.hortonworks.com:2181 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@6e232ae3 2016-12-14 00:51:16,038 INFO [AsyncHBase Timer HBaseClient #1-SendThread(sandbox.hortonworks.com:2181)] ClientCnxn: Opening socket connection to server sandbox.hortonworks.com/10.0.2.15:2181. Will not attempt to authenticate using SASL (unknown error) 2016-12-14 00:51:16,040 INFO [AsyncHBase Timer HBaseClient #1-SendThread(sandbox.hortonworks.com:2181)] ClientCnxn: Socket connection established to sandbox.hortonworks.com/10.0.2.15:2181, initiating session 2016-12-14 00:51:16,042 INFO [AsyncHBase Timer HBaseClient #1-SendThread(sandbox.hortonworks.com:2181)] ClientCnxn: Session establishment complete on server sandbox.hortonworks.com/10.0.2.15:2181, sessionid = 0x158fa6cce65008e, negotiated timeout = 5000 2016-12-14 00:51:16,043 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down 2016-12-14 00:51:16,048 INFO [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Connecting to .META. region @ 10.0.2.15:16020 2016-12-14 00:51:16,051 INFO [AsyncHBase I/O Worker #4] HBaseClient: Channel [id: 0xf2cf1dfe, /10.0.2.15:37372 😆 /10.0.2.15:16020] is disconnecting: [id: 0xf2cf1dfe, /10.0.2.15:37372 😆 /10.0.2.15:16020] DISCONNECTED 2016-12-14 00:51:16,052 INFO [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Need to find the .META. region 2016-12-14 00:51:16,055 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x158fa6cce65008e closed 2016-12-14 00:51:16,055 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Initiating client connection, connectString=sandbox.hortonworks.com:2181 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@6e232ae3 2016-12-14 00:51:16,056 INFO [AsyncHBase Timer HBaseClient #1-SendThread(sandbox.hortonworks.com:2181)] ClientCnxn: Opening socket connection to server sandbox.hortonworks.com/10.0.2.15:2181. Will not attempt to authenticate using SASL (unknown error) 2016-12-14 00:51:16,058 INFO [AsyncHBase Timer HBaseClient #1-SendThread(sandbox.hortonworks.com:2181)] ClientCnxn: Socket connection established to sandbox.hortonworks.com/10.0.2.15:2181, initiating session 2016-12-14 00:51:16,061 INFO [AsyncHBase Timer HBaseClient #1-SendThread(sandbox.hortonworks.com:2181)] ClientCnxn: Session establishment complete on server sandbox.hortonworks.com/10.0.2.15:2181, sessionid = 0x158fa6cce65008f, negotiated timeout = 5000 2016-12-14 00:51:16,061 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down 2016-12-14 00:51:16,067 INFO [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Connecting to .META. region @ 10.0.2.15:16020 2016-12-14 00:51:16,069 INFO [AsyncHBase I/O Worker #1] HBaseClient: Channel [id: 0x24f85c35, /10.0.2.15:37374 😆 /10.0.2.15:16020] is disconnecting: [id: 0x24f85c35, /10.0.2.15:37374 😆 /10.0.2.15:16020] DISCONNECTED 2016-12-14 00:51:16,070 INFO [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Need to find the .META. region 2016-12-14 00:51:16,074 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x158fa6cce65008f closed 2016-12-14 00:51:16,074 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Initiating client connection, connectString=sandbox.hortonworks.com:2181 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@6e232ae3 2016-12-14 00:51:16,075 INFO [AsyncHBase Timer HBaseClient #1-SendThread(sandbox.hortonworks.com:2181)] ClientCnxn: Opening socket connection to server sandbox.hortonworks.com/10.0.2.15:2181. Will not attempt to authenticate using SASL (unknown error) 2016-12-14 00:51:16,078 INFO [AsyncHBase Timer HBaseClient #1-SendThread(sandbox.hortonworks.com:2181)] ClientCnxn: Socket connection established to sandbox.hortonworks.com/10.0.2.15:2181, initiating session 2016-12-14 00:51:16,080 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down 2016-12-14 00:51:16,084 INFO [AsyncHBase Timer HBaseClient #1-SendThread(sandbox.hortonworks.com:2181)] ClientCnxn: Session establishment complete on server sandbox.hortonworks.com/10.0.2.15:2181, sessionid = 0x158fa6cce650090, negotiated timeout = 5000 2016-12-14 00:51:16,087 INFO [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Connecting to .META. region @ 10.0.2.15:16020 2016-12-14 00:51:16,089 INFO [AsyncHBase I/O Worker #2] HBaseClient: Channel [id: 0xd63aca14, /10.0.2.15:37376 😆 /10.0.2.15:16020] is disconnecting: [id: 0xd63aca14, /10.0.2.15:37376 😆 /10.0.2.15:16020] DISCONNECTED 2016-12-14 00:51:16,090 INFO [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Need to find the .META. region 2016-12-14 00:51:16,092 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x158fa6cce650090 closed 2016-12-14 00:51:16,092 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Initiating client connection, connectString=sandbox.hortonworks.com:2181 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@6e232ae3 2016-12-14 00:51:16,094 INFO [AsyncHBase Timer HBaseClient #1-SendThread(sandbox.hortonworks.com:2181)] ClientCnxn: Opening socket connection to server sandbox.hortonworks.com/10.0.2.15:2181. Will not attempt to authenticate using SASL (unknown error) 2016-12-14 00:51:16,095 INFO [AsyncHBase Timer HBaseClient #1-SendThread(sandbox.hortonworks.com:2181)] ClientCnxn: Socket connection established to sandbox.hortonworks.com/10.0.2.15:2181, initiating session 2016-12-14 00:51:16,098 INFO [AsyncHBase Timer HBaseClient #1-SendThread(sandbox.hortonworks.com:2181)] ClientCnxn: Session establishment complete on server sandbox.hortonworks.com/10.0.2.15:2181, sessionid = 0x158fa6cce650091, negotiated timeout = 5000 2016-12-14 00:51:16,098 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down 2016-12-14 00:51:16,104 INFO [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Connecting to .META. region @ 10.0.2.15:16020 2016-12-14 00:51:16,106 INFO [AsyncHBase I/O Worker #3] HBaseClient: Channel [id: 0x24387fa0, /10.0.2.15:37378 😆 /10.0.2.15:16020] is disconnecting: [id: 0x24387fa0, /10.0.2.15:37378 😆 /10.0.2.15:16020] DISCONNECTED 2016-12-14 00:51:16,107 INFO [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Need to find the .META. region 2016-12-14 00:51:16,110 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x158fa6cce650091 closed 2016-12-14 00:51:16,110 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Initiating client connection, connectString=sandbox.hortonworks.com:2181 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@6e232ae3 2016-12-14 00:51:16,110 INFO [AsyncHBase Timer HBaseClient #1-SendThread(sandbox.hortonworks.com:2181)] ClientCnxn: Opening socket connection to server sandbox.hortonworks.com/10.0.2.15:2181. Will not attempt to authenticate using SASL (unknown error) 2016-12-14 00:51:16,112 INFO [AsyncHBase Timer HBaseClient #1-SendThread(sandbox.hortonworks.com:2181)] ClientCnxn: Socket connection established to sandbox.hortonworks.com/10.0.2.15:2181, initiating session 2016-12-14 00:51:16,115 INFO [AsyncHBase Timer HBaseClient #1-SendThread(sandbox.hortonworks.com:2181)] ClientCnxn: Session establishment complete on server sandbox.hortonworks.com/10.0.2.15:2181, sessionid = 0x158fa6cce650092, negotiated timeout = 5000 2016-12-14 00:51:16,116 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down 2016-12-14 00:51:16,122 INFO [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Connecting to .META. region @ 10.0.2.15:16020 2016-12-14 00:51:16,126 INFO [AsyncHBase I/O Worker #4] HBaseClient: Channel [id: 0x8d3a08ca, /10.0.2.15:37380 😆 /10.0.2.15:16020] is disconnecting: [id: 0x8d3a08ca, /10.0.2.15:37380 😆 /10.0.2.15:16020] DISCONNECTED 2016-12-14 00:51:16,127 INFO [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Need to find the .META. region 2016-12-14 00:51:16,130 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x158fa6cce650092 closed 2016-12-14 00:51:16,130 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Initiating client connection, connectString=sandbox.hortonworks.com:2181 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@6e232ae3 2016-12-14 00:51:16,131 INFO [AsyncHBase Timer HBaseClient #1-SendThread(sandbox.hortonworks.com:2181)] ClientCnxn: Opening socket connection to server sandbox.hortonworks.com/10.0.2.15:2181. Will not attempt to authenticate using SASL (unknown error) 2016-12-14 00:51:16,132 INFO [AsyncHBase Timer HBaseClient #1-SendThread(sandbox.hortonworks.com:2181)] ClientCnxn: Socket connection established to sandbox.hortonworks.com/10.0.2.15:2181, initiating session 2016-12-14 00:51:16,135 INFO [AsyncHBase Timer HBaseClient #1-SendThread(sandbox.hortonworks.com:2181)] ClientCnxn: Session establishment complete on server sandbox.hortonworks.com/10.0.2.15:2181, sessionid = 0x158fa6cce650093, negotiated timeout = 5000 2016-12-14 00:51:16,136 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down 2016-12-14 00:51:16,143 INFO [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Connecting to .META. region @ 10.0.2.15:16020 2016-12-14 00:51:16,146 INFO [AsyncHBase I/O Worker #1] HBaseClient: Channel [id: 0x43a1016f, /10.0.2.15:37382 😆 /10.0.2.15:16020] is disconnecting: [id: 0x43a1016f, /10.0.2.15:37382 😆 /10.0.2.15:16020] DISCONNECTED 2016-12-14 00:51:16,147 INFO [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Need to find the .META. region 2016-12-14 00:51:16,150 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x158fa6cce650093 closed 2016-12-14 00:51:16,150 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Initiating client connection, connectString=sandbox.hortonworks.com:2181 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@6e232ae3 2016-12-14 00:51:16,151 INFO [AsyncHBase Timer HBaseClient #1-SendThread(sandbox.hortonworks.com:2181)] ClientCnxn: Opening socket connection to server sandbox.hortonworks.com/10.0.2.15:2181. Will not attempt to authenticate using SASL (unknown error) 2016-12-14 00:51:16,153 INFO [AsyncHBase Timer HBaseClient #1-SendThread(sandbox.hortonworks.com:2181)] ClientCnxn: Socket connection established to sandbox.hortonworks.com/10.0.2.15:2181, initiating session 2016-12-14 00:51:16,155 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down 2016-12-14 00:51:16,160 INFO [AsyncHBase Timer HBaseClient #1-SendThread(sandbox.hortonworks.com:2181)] ClientCnxn: Session establishment complete on server sandbox.hortonworks.com/10.0.2.15:2181, sessionid = 0x158fa6cce650094, negotiated timeout = 5000 2016-12-14 00:51:16,163 INFO [AsyncHBase Timer HBaseClient #1-EventThread] HBaseClient: Connecting to .META. region @ 10.0.2.15:16020 2016-12-14 00:51:16,167 INFO [AsyncHBase I/O Worker #2] HBaseClient: Channel [id: 0x9b61659f, /10.0.2.15:37384 😆 /10.0.2.15:16020] is disconnecting: [id: 0x9b61659f, /10.0.2.15:37384 😆 /10.0.2.15:16020] DISCONNECTED 2016-12-14 00:51:16,174 INFO [main] TSDB: Flushing compaction queue 2016-12-14 00:51:16,185 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ZooKeeper: Session: 0x158fa6cce650094 closed 2016-12-14 00:51:16,185 INFO [AsyncHBase Timer HBaseClient #1-EventThread] ClientCnxn: EventThread shut down 2016-12-14 00:51:16,191 INFO [main] TSDB: Completed shutting down the TSDB Exception in thread "main" java.lang.RuntimeException: Initialization failed at net.opentsdb.tools.TSDMain.main(TSDMain.java:196) Caused by: com.stumbleupon.async.DeferredGroupException: At least one of the Deferreds failed, first exception: at com.stumbleupon.async.DeferredGroup.done(DeferredGroup.java:169) at com.stumbleupon.async.DeferredGroup.recordCompletion(DeferredGroup.java:142) at com.stumbleupon.async.DeferredGroup.access$000(DeferredGroup.java:36) at com.stumbleupon.async.DeferredGroup$1Notify.call(DeferredGroup.java:82) at com.stumbleupon.async.Deferred.doCall(Deferred.java:1278) at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257) at com.stumbleupon.async.Deferred.access$300(Deferred.java:430) at com.stumbleupon.async.Deferred$Continue.call(Deferred.java:1366) at com.stumbleupon.async.Deferred.doCall(Deferred.java:1278) at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257) at com.stumbleupon.async.Deferred.handleContinuation(Deferred.java:1313) at com.stumbleupon.async.Deferred.doCall(Deferred.java:1284) at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257) at com.stumbleupon.async.Deferred.callback(Deferred.java:1005) at org.hbase.async.HBaseRpc.callback(HBaseRpc.java:712) at org.hbase.async.HBaseClient.tooManyAttempts(HBaseClient.java:2058) at org.hbase.async.HBaseClient.sendRpcToRegion(HBaseClient.java:1920) at org.hbase.async.HBaseClient$1RetryRpc.call(HBaseClient.java:1944) at org.hbase.async.HBaseClient$1RetryRpc.call(HBaseClient.java:1927) at com.stumbleupon.async.Deferred.doCall(Deferred.java:1278) at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257) at com.stumbleupon.async.Deferred.callback(Deferred.java:1005) at org.hbase.async.HBaseClient$ZKClient$ZKCallback.processResult(HBaseClient.java:3645) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:558) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495) Caused by: org.hbase.async.NonRecoverableException: Too many attempts: HBaseRpc(method=getClosestRowBefore, table="hbase:meta", key=[116, 115, 100, 98, 45, 117, 105, 100, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58], region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", stop_key=""), attempt=11, timeout=0, hasTimedout=false) at org.hbase.async.HBaseClient.tooManyAttempts(HBaseClient.java:2056) ... 9 more