Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Kerberized HDP 2.4 - OpenTSDB startup error

Kerberized HDP 2.4 - OpenTSDB startup error

Expert Contributor

Hi All - I've Kerberized HDP 2.4, and i'm trying to start OpenTSDB (v2.0.1)

On starting it, i get the following error (in file - /var/logs/opentsdb.log)

Any ideas on what needs to be done ?

2016-12-12 21:38:40,162 ERROR [New I/O worker #7] RegionClient: Invalid rpcid: -1 found in ReplayingDecoderBuffer(ridx=99, widx=99)="\x00\x00\x00_^\x08\xFF\xFF\xFF\xFF\x0F\x12V\n6org.apache.hadoop.hbase.security.AccessDeniedException\x12\x1AAuthentication is required(\x01" 2016-12-12 21:38:40,163 ERROR [New I/O worker #7] RegionClient: Unexpected exception from downstream on [id: 0x1a3eb51a, /10.0.2.15:40409 => /10.0.2.15:16020] org.hbase.async.NonRecoverableException: Invalid rpcid: -1 found in ReplayingDecoderBuffer(ridx=99, widx=99)="\x00\x00\x00_^\x08\xFF\xFF\xFF\xFF\x0F\x12V\n6org.apache.hadoop.hbase.security.AccessDeniedException\x12\x1AAuthentication is required(\x01" at org.hbase.async.RegionClient.decode(RegionClient.java:1292) ~[asynchbase-1.6.0.jar:na] at org.hbase.async.RegionClient.decode(RegionClient.java:89) ~[asynchbase-1.6.0.jar:na] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500) ~[netty-3.9.4.Final.jar:na] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) ~[netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[netty-3.9.4.Final.jar:na] at org.hbase.async.RegionClient.handleUpstream(RegionClient.java:1082) ~[asynchbase-1.6.0.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.9.4.Final.jar:na]

7 REPLIES 7

Re: Kerberized HDP 2.4 - OpenTSDB startup error

It would appear that you need to use at least OpenTSDB 2.2.0 for Kerberos support

https://github.com/OpenTSDB/opentsdb/issues/491

Specifically, see this comment for information about configuring OpenTSDB.

Re: Kerberized HDP 2.4 - OpenTSDB startup error

Expert Contributor

@Josh Elser - i seem to be having issues with installing the Latest version of OpenTSDB

Here is what i've done ->

-> Downloaded jar from https://github.com/OpenTSDB/opentsdb/releases/tag/v2.3.0RC2

- > run gunzip & tar -xvf opentsdb-2.3.0-RC2.tar

-> cd opentsdb-2.3.0-RC2.tar

-> ./build.sh

which gives me the following error -

make[1]: Entering directory `/root/opentsdb-2.3.0-RC2/build' /usr/lib/jvm/java/bin/java -cp third_party/javacc/javacc-6.1.2.jar javacc -OUTPUT_DIRECTORY:./src/net/opentsdb/query/expression/parser ../src/parser.jj; echo PWD: `pwd`; Error: Could not find or load main class javacc PWD: /root/opentsdb-2.3.0-RC2/build

Any pointers on what needs to be done to install OpenTSDB ?

Re: Kerberized HDP 2.4 - OpenTSDB startup error

/usr/lib/jvm/java/bin/java -cp third_party/javacc/javacc-6.1.2.jar 
javacc -OUTPUT_DIRECTORY:./src/net/opentsdb/query/expression/parser 
../src/parser.jj;

It seems to be trying to run this command which, at a glance, doesn't seem to make sense to me. Sorry, I don't have much experience with opentsdb. Hopefully someone else has the expertise here. Please do share if you get to the bottom of it.

Re: Kerberized HDP 2.4 - OpenTSDB startup error

Expert Contributor

@Josh Elser - i used the following command to download the version 2.2.1

This went through fine, however - when i do 'Add Service' on Ambari, i dont see OpenTSDB as an option

git clone https://github.com/OpenTSDB/opentsdb.git /var/lib/ambari-server/resources/stacks/HDP/$VERSION/services/OPENTSDB

Alternately, when i use this (the earlier url specified in this link -

https://community.hortonworks.com/articles/4577/use-opentsdb-to-storevisualize-stock-data-on-hdp-s.h..., it goes through fine.. however, as you mentioned, i need OpenTSDB v 2.2.0 and later.

Older version : sudo git clone https://github.com/hortonworks-gallery/ambari-opentsdb-service.git /var/lib/ambari-server/resources/stacks/HDP/$VERSION/services/OPENTSDB

Any ideas ?

Re: Kerberized HDP 2.4 - OpenTSDB startup error

Expert Contributor

@Ali Bajwa - looping you in.. pls. let me know input on this.

Highlighted

Re: Kerberized HDP 2.4 - OpenTSDB startup error

@Karan Alang the Ambari service to deploy OpenTSDB was just for demo purposes so it was only tested with v2.0.1 and was not tested with kerberos. You can try to get Ambari to install newer version of OpenTSDB by updating the value of opentsdb.download_url in ambari (at the time of install). However, I would recommend to try and install it manually using the tips from @Josh Elser. If you're able to get it working with newer OpenTSDB version by manual install, let me know and I can try to update the Ambari service accordingly.

Re: Kerberized HDP 2.4 - OpenTSDB startup error

@Karan Alang I have updated the OpenTSDB Ambari service to deploy v2.2.1 (currently GA version). I tested the stock data tutorial and it worked fine on my RHEL7 cluster running HDP 2.5.3, but should work on both RHEL6/7. Have not tried on kerberized cluster though. Note that this Ambari service is not supported by Hortonworks and is for demo purposes only (do not deploy in production)