HDP 2.5 How to connect Zeppelin to Phoenix in secured cluster

[#1]

HDP 2.5 secured cluster with Zeppelin and HBase installed. I could connect to Phoenix using the sqlline.py utility using the following parameter.

/usr/hdp/current/phoenix-client/bin/sqlline.py <zk1>,<zk2>,<zk3>:2181:/hbase-secure:hbase@DOMAIN.COM:/etc/security/keytabs/hbase.headless.keytab

However when I try to use similar parameter for jdbc conn string in phoenix, I got

null
org.apache.phoenix.exception.PhoenixIOException: Failed after attempts=1, exceptions:
Wed Jan 18 15:13:11 UTC 2017, RpcRetryingCaller{globalStartTime=1484752390908, pause=100, retries=1}, org.apache.hadoop.hbase.MasterNotRunningException: com.google.protobuf.ServiceException: java.io.IOException: Broken pipe

class org.apache.zeppelin.interpreter.InterpreterException

The jdbc interpreter settings for Phoenix are

Do I need to create phoenixuser principal and use that keytab instead? and what is phoenix.user and phoenix.password?

[#2]

org.apache.hadoop.hbase.MasterNotRunningException

It would appear that HBase is not actually running. Can you verify that HBase is healthy? Can you interact with HBase via `hbase shell` (e.g. create a table, add data to it, delete the table)?

[#3]

@Josh Elser

HBase is running fine. I cold pull the table list and look into "atlas_titan" tables content

[root@qwang-hdp5 logs]# hbase shell
HBase Shell; enter 'help<RETURN>' for list of supported commands.
Type "exit<RETURN>" to leave the HBase Shell
Version 1.1.2.2.5.3.0-37, rcb8c969d1089f1a34e9df11b6eeb96e69bcf878d, Tue Nov 29 18:48:22 UTC 2016

hbase(main):001:0> list
TABLE
ATLAS_ENTITY_AUDIT_EVENTS
SYSTEM.CATALOG
SYSTEM.FUNCTION
SYSTEM.SEQUENCE
SYSTEM.STATS
atlas_titan
driver_dangerous_event
my_table
8 row(s) in 0.5710 seconds

hbase(main):003:0> scan 'atlas_titan'

[#4]

Can you please try to create a table? This specifically uses the Master whereas scanning a table can be done by talking to a RegionServer.

[#5]

@Josh Elser

HBase is up and running fine. I can create table in both hbase shell and sqlline.py

hbase(main):001:0> create 'my_table1', {NAME =>'cf1'}, {NAME =>'cf2'}
0 row(s) in 2.9560 seconds


=> Hbase::Table - my_table1
hbase(main):002:0> put 'my_table1', 'rowkey01', 'cf1:c1', 'test value'
0 row(s) in 0.6850 seconds


hbase(main):003:0> get 'my_table1', 'rowkey01'
COLUMN                          CELL
 cf1:c1                         timestamp=1484780190152, value=test value
1 row(s) in 0.0780 seconds

[#6]

Sorry for the tangent, Qi. Glad Sergey was here to steer you in a better direction 🙂

[#7]

I bet that HBase is healthy. Was able to reproduce it locally. The same exception running test from zeppelin with perfectly working sqlline (as well as hbase shell).

[#8]

@Sergey Soldatov

Are you running into the same problem on a secured cluster or non-secured one?

[#9 · Accepted solution]

@Qi Wang

Well, to fix it you will need to proceed few steps:

1. In the jdbc interpreters configuration you need to remove default artifacts for phoenix and hbase (phoenix-4.7... and hbase-1.1.3...).

2. add new artifact for phoenix-client.jar (just provide the path /usr/hdp/current/phoenix-client/phoenix-client.jar)

3. You need to set up interpreter to properly find hbase configuration dir. It could be done by adding to the end of zeppelin-env.sh

export CLASSPATH=$CLASSPATH:/etc/hbase/conf

using Ambari and after restart of Zeppelin service everything is supposed to work

[#10]

@Sergey Soldatov

It works after following your steps. It would be nice to figure out how to do step 3 in ambari if possible, always worry some later ambari update would wipe the change out. thanks a lot for your help!