phoenix connection fails on a secured hbase cluster with error code : 103

[#1]

Hi All, I have a secured hbase cluster and I am trying to connect to phoenix as user1

It fails with error code: 103

[user1@zk-host-2 bin]$ /usr/hdp/current/phoenix-client/bin/sqlline.py jdbc:phoenix:zk-host-1,zk-host-2,zk-host-3:/hbase-secure
Setting property: [incremental, false]
Setting property: [isolation, TRANSACTION_READ_COMMITTED]
issuing: !connect jdbc:phoenix:jdbc:phoenix:zk-host-1,zk-host-2,zk-host-3:/hbase-secure none none org.apache.phoenix.jdbc.PhoenixDriver
Connecting to jdbc:phoenix:jdbc:phoenix:zk-host-1,zk-host-2,zk-host-3:/hbase-secure
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/grid/0/hdp/2.6.0.2-55/phoenix/phoenix-4.7.0.2.6.0.2-55-client.j



ar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/grid/0/hdp/2.6.0.2-55/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
17/03/22 21:36:49 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Error: ERROR 103 (08004): Unable to establish connection. (state=08004,code=103)
java.sql.SQLException: ERROR 103 (08004): Unable to establish connection.
        at org.apache.phoenix.exception.SQLExceptionCode$Factory$1.newException(SQLExceptionCode.java:442)
        at org.apache.phoenix.exception.SQLExceptionInfo.buildException(SQLExceptionInfo.java:145)
        at org.apache.phoenix.jdbc.PhoenixEmbeddedDriver$ConnectionInfo.normalize(PhoenixEmbeddedDriver.java:395)
        at org.apache.phoenix.jdbc.PhoenixDriver.getConnectionQueryServices(PhoenixDriver.java:217)
        at org.apache.phoenix.jdbc.PhoenixEmbeddedDriver.createConnection(PhoenixEmbeddedDriver.java:150)
        at org.apache.phoenix.jdbc.PhoenixDriver.connect(PhoenixDriver.java:203)
        at sqlline.DatabaseConnection.connect(DatabaseConnection.java:157)
        at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:203)
        at sqlline.Commands.connect(Commands.java:1064)
        at sqlline.Commands.connect(Commands.java:996)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:36)
        at sqlline.SqlLine.dispatch(SqlLine.java:804)
        at sqlline.SqlLine.initArgs(SqlLine.java:588)
        at sqlline.SqlLine.begin(SqlLine.java:656)
        at sqlline.SqlLine.start(SqlLine.java:398)
        at sqlline.SqlLine.main(SqlLine.java:292)
Caused by: java.io.IOException: Login failure for phoenix from keytab zk-host-1,zk-host-2,zk-host-3: javax.security.auth.login.LoginException: Unable to obtain password from user


        at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:1098)
        at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:307)
        at org.apache.hadoop.hbase.security.User$SecureHadoopUser.login(User.java:386)
        at org.apache.hadoop.hbase.security.User.login(User.java:253)
        at org.apache.phoenix.jdbc.PhoenixEmbeddedDriver$ConnectionInfo.normalize(PhoenixEmbeddedDriver.java:386)
        ... 17 more
Caused by: javax.security.auth.login.LoginException: Unable to obtain password from user


        at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:860)
        at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:723)

        at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:588)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at javax.security.auth.login.LoginContext.invoke(LoginContext.java:762)
        at javax.security.auth.login.LoginContext.access$000(LoginContext.java:203)
        at javax.security.auth.login.LoginContext$4.run(LoginContext.java:690)
        at javax.security.auth.login.LoginContext$4.run(LoginContext.java:688)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:687)
        at javax.security.auth.login.LoginContext.login(LoginContext.java:595)
        at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:1089)
        ... 21 more
sqlline version 1.1.8

However, if I try to access hbase shell using the same user - it succeeds

hbase(main):001:0> whoami
user1@EXAMPLE.COM (auth:KERBEROS)
    groups: user1, users, hivetest


hbase(main):002:0> create 't2', 'f1', 'f2', 'f3'
0 row(s) in 1.6400 seconds


=> Hbase::Table - t2
hbase(main):003:0> !quit

Am I missing any configs?

[#2 · Accepted solution]

re-read that exception message again: "Login failure for phoenix from keytab zk-host-1,zk-host-2,zk-host-3"

JAAS is trying to perform a login using your list of ZooKeeper hosts as the path to a keytab. Pretty sure that isn't right 🙂

If we look at your argument to sqlline.py, we can see the error:

jdbc:phoenix:zk-host-1,zk-host-2,zk-host-3:/hbase-secure

We can see from the usage statement on sqlline:

Usage: sqlline.py [zookeeper] [optional_sql_file]

Example:
1. sqlline.py
2. sqlline.py localhost:2181:/hbase
3. sqlline.py localhost:2181:/hbase ../examples/stock_symbol.sql
4. sqlline.py ../examples/stock_symbol.sql

In short: the argument to sqlline is not the full JDBC url, just the list of ZooKeeper hosts and the root znode:

zk-host-1,zk-host-2,zk-host-3:/hbase-secure

[#3]

@Josh Elser Thanks for explanation !! This works