Phoenix connection in kerberos cluster failing

[#1]

i do have a Kerberos ticket but i get error connecting to Phoenix.

We are using Ranger, does the admin have to give privileges via Ranger?

[kn857sa@dotstohdaut202 conf]$ sqlline.py dotstohdamn201:2181:/hbase-secure
Setting property: [incremental, false]
Setting property: [isolation, TRANSACTION_READ_COMMITTED]
issuing: !connect jdbc:phoenix:dotstohdamn201:2181:/hbase-secure none none org.apache.phoenix.jdbc.PhoenixDriver
Connecting to jdbc:phoenix:dotstohdamn201:2181:/hbase-secure
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hdp/2.6.2.0-205/phoenix/phoenix-4.7.0.2.6.2.0-205-client.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/2.6.2.0-205/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
18/06/18 11:34:46 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
18/06/18 11:34:47 WARN shortcircuit.DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
18/06/18 11:34:47 WARN ipc.AbstractRpcClient: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
18/06/18 11:34:47 FATAL ipc.AbstractRpcClient: SASL authentication failed. The most likely cause is missing or invalid credentials. Consider 'kinit'.
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
        at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
        at org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:179)
        at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupSaslConnection(RpcClientImpl.java:611)
        at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.access$600(RpcClientImpl.java:156)
        at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:737)
        at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:734)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        

ticket

[kn857sa@dotstohdaut202 conf]$ klist
Ticket cache: FILE:/tmp/krb5cc_1696202362
Default principal: [email protected]
Valid starting       Expires              Service principal
06/18/2018 10:53:54  06/18/2018 20:53:54  krbtgt/[email protected]
        renew until 06/25/2018 10:53:54
06/18/2018 10:53:54  06/18/2018 20:53:54  krbtgt/[email protected]
        renew until 06/25/2018 10:53:54
[kn857sa@dotstohdaut202 conf]$

[#2]

Hey @Sami Ahmad!
Did you tried to add the principal + keytab path into your jdbc string connection?

jdbc:phoenix:<ZK-QUORUM>:<ZK-PORT>:<ZK-HBASE-NODE>:principal_name@REALM:/path/to/keytab

Hope this helps!

[#3]

I did that but then i get the error as shown below

sqlline.py dotstohdamn201.tolls.dot.state.fl.us:2181:/hbase-secure/hbase:[email protected]:/etc/krb5.keytab

error

rror: ERROR 103 (08004): Unable to establish connection. (state=08004,code=103)
java.sql.SQLException: ERROR 103 (08004): Unable to establish connection.
        at org.apache.phoenix.exception.SQLExceptionCode$Factory$1.newException(SQLExceptionCode.java:443)
        at org.apache.phoenix.exception.SQLExceptionInfo.buildException(SQLExceptionInfo.java:145)
        at org.apache.phoenix.jdbc.PhoenixEmbeddedDriver$ConnectionInfo.normalize(PhoenixEmbeddedDriver.java:396)
        at org.apache.phoenix.jdbc.PhoenixDriver.getConnectionQueryServices(PhoenixDriver.java:219)
        at org.apache.phoenix.jdbc.PhoenixEmbeddedDriver.createConnection(PhoenixEmbeddedDriver.java:150)
        at org.apache.phoenix.jdbc.PhoenixDriver.connect(PhoenixDriver.java:205)
        at sqlline.DatabaseConnection.connect(DatabaseConnection.java:157)
        at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:203)
        at sqlline.Commands.connect(Commands.java:1064)
        at sqlline.Commands.connect(Commands.java:996)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:36)
        at sqlline.SqlLine.dispatch(SqlLine.java:804)
        at sqlline.SqlLine.initArgs(SqlLine.java:588)
        at sqlline.SqlLine.begin(SqlLine.java:656)
        at sqlline.SqlLine.start(SqlLine.java:398)
        at sqlline.SqlLine.main(SqlLine.java:292)
Caused by: java.io.IOException: Login failure for [email protected] from keytab /etc/krb5.keytab: javax.security.auth.login.LoginException: Unable to obtain password from user

[#4]

@Sami Ahmad

Looks like there is problem with your keytab/principal.

Caused by: java.io.IOException: Login failure for [email protected] from keytab /etc/krb5.keytab: javax.security.auth.login.LoginException: Unable to obtain password from user

Can you try doing kinit as below and see if it executes successfully:

kinit -kt /etc/krb5.keytab [email protected]

[#5]

i get permission denied but then if i do a simple kinit it works ..whats the difference ?

[kn857sa@dotstohdaut202 log]$ kinit -kt /etc/krb5.keytab [email protected]
kinit: Pre-authentication failed: Permission denied while getting initial credentials
[kn857sa@dotstohdaut202 log]$
[kn857sa@dotstohdaut202 log]$ kinit
Password for [email protected]:
[kn857sa@dotstohdaut202 log]$ klist
Ticket cache: FILE:/tmp/krb5cc_1696202362
Default principal: [email protected]
Valid starting       Expires              Service principal
06/18/2018 13:04:45  06/18/2018 23:04:45  krbtgt/[email protected]
        renew until 06/25/2018 13:04:36
[kn857sa@dotstohdaut202 log]$

[#6]

Then there is some issue with the keytab. In the second attempt a password was provided instead of keytab.

Please check if its a valid keytab and make the kinit command. Then you can use the same in phoenix connection.

[#7]

if i do a klist -k i do see its going to the /etc/krb5.keytab but i don't have privilege to read it .

how else can i check ?

[#8]

You'd need to get the read access to the keytab file.

[#9]

keyatab should be owned by the owner, If that keytab belongs to your user. you should have read permissions. Pls check more on this with your system admins.

[#10]

when i do a "kinit" am i not using a keytab? if yes which keytab i am using?