question Re: Phoenix connection in kerberos cluster failing in Archives of Support Questions (Read Only)

Phoenix connection in kerberos cluster failing

aliyesami — Mon, 18 Jun 2018 22:43:30 GMT

i do have a Kerberos ticket but i get error connecting to Phoenix.

We are using Ranger, does the admin have to give privileges via Ranger?

[kn857sa@dotstohdaut202 conf]$ sqlline.py dotstohdamn201:2181:/hbase-secure
Setting property: [incremental, false]
Setting property: [isolation, TRANSACTION_READ_COMMITTED]
issuing: !connect jdbc:phoenix:dotstohdamn201:2181:/hbase-secure none none org.apache.phoenix.jdbc.PhoenixDriver
Connecting to jdbc:phoenix:dotstohdamn201:2181:/hbase-secure
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hdp/2.6.2.0-205/phoenix/phoenix-4.7.0.2.6.2.0-205-client.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/2.6.2.0-205/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
18/06/18 11:34:46 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
18/06/18 11:34:47 WARN shortcircuit.DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
18/06/18 11:34:47 WARN ipc.AbstractRpcClient: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
18/06/18 11:34:47 FATAL ipc.AbstractRpcClient: SASL authentication failed. The most likely cause is missing or invalid credentials. Consider 'kinit'.
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
        at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
        at org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:179)
        at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupSaslConnection(RpcClientImpl.java:611)
        at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.access$600(RpcClientImpl.java:156)
        at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:737)
        at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:734)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)

ticket

[kn857sa@dotstohdaut202 conf]$ klist
Ticket cache: FILE:/tmp/krb5cc_1696202362
Default principal: kn857sa@xxxxxxxx.US
Valid starting       Expires              Service principal
06/18/2018 10:53:54  06/18/2018 20:53:54  krbtgt/xxxxxxx@xxxxxx.US
        renew until 06/25/2018 10:53:54
06/18/2018 10:53:54  06/18/2018 20:53:54  krbtgt/xxxxxxx@xxxxxx.US
        renew until 06/25/2018 10:53:54
[kn857sa@dotstohdaut202 conf]$

Re: Phoenix connection in kerberos cluster failing

vmurakami — Mon, 18 Jun 2018 23:26:38 GMT

Hey @Sami Ahmad!
Did you tried to add the principal + keytab path into your jdbc string connection?

jdbc:phoenix:<ZK-QUORUM>:<ZK-PORT>:<ZK-HBASE-NODE>:principal_name@REALM:/path/to/keytab

Hope this helps!

Re: Phoenix connection in kerberos cluster failing

aliyesami — Mon, 18 Jun 2018 23:48:15 GMT

I did that but then i get the error as shown below

sqlline.py dotstohdamn201.tolls.dot.state.fl.us:2181:/hbase-secure/hbase:kn857sa@XXXX.US:/etc/krb5.keytab

error

rror: ERROR 103 (08004): Unable to establish connection. (state=08004,code=103)
java.sql.SQLException: ERROR 103 (08004): Unable to establish connection.
        at org.apache.phoenix.exception.SQLExceptionCode$Factory$1.newException(SQLExceptionCode.java:443)
        at org.apache.phoenix.exception.SQLExceptionInfo.buildException(SQLExceptionInfo.java:145)
        at org.apache.phoenix.jdbc.PhoenixEmbeddedDriver$ConnectionInfo.normalize(PhoenixEmbeddedDriver.java:396)
        at org.apache.phoenix.jdbc.PhoenixDriver.getConnectionQueryServices(PhoenixDriver.java:219)
        at org.apache.phoenix.jdbc.PhoenixEmbeddedDriver.createConnection(PhoenixEmbeddedDriver.java:150)
        at org.apache.phoenix.jdbc.PhoenixDriver.connect(PhoenixDriver.java:205)
        at sqlline.DatabaseConnection.connect(DatabaseConnection.java:157)
        at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:203)
        at sqlline.Commands.connect(Commands.java:1064)
        at sqlline.Commands.connect(Commands.java:996)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:36)
        at sqlline.SqlLine.dispatch(SqlLine.java:804)
        at sqlline.SqlLine.initArgs(SqlLine.java:588)
        at sqlline.SqlLine.begin(SqlLine.java:656)
        at sqlline.SqlLine.start(SqlLine.java:398)
        at sqlline.SqlLine.main(SqlLine.java:292)
Caused by: java.io.IOException: Login failure for kn857sa@XXXXXX.US from keytab /etc/krb5.keytab: javax.security.auth.login.LoginException: Unable to obtain password from user

Re: Phoenix connection in kerberos cluster failing

sandyy006 — Tue, 19 Jun 2018 00:04:30 GMT

@Sami Ahmad

Looks like there is problem with your keytab/principal.

Caused by: java.io.IOException: Login failure for kn857sa@XXXXXX.US from keytab /etc/krb5.keytab: javax.security.auth.login.LoginException: Unable to obtain password from user

Can you try doing kinit as below and see if it executes successfully:

kinit -kt /etc/krb5.keytab kn857sa@XXXXXX.US

Re: Phoenix connection in kerberos cluster failing

aliyesami — Tue, 19 Jun 2018 00:10:50 GMT

i get permission denied but then if i do a simple kinit it works ..whats the difference ?

[kn857sa@dotstohdaut202 log]$ kinit -kt /etc/krb5.keytab kn857sa@XXXXXXXXX.US
kinit: Pre-authentication failed: Permission denied while getting initial credentials
[kn857sa@dotstohdaut202 log]$
[kn857sa@dotstohdaut202 log]$ kinit
Password for kn857sa@XXXXXXXXX.US:
[kn857sa@dotstohdaut202 log]$ klist
Ticket cache: FILE:/tmp/krb5cc_1696202362
Default principal: kn857sa@XXXXXXXXXXX.US
Valid starting       Expires              Service principal
06/18/2018 13:04:45  06/18/2018 23:04:45  krbtgt/TOLLS.DOT.STATE.FL.US@XXXXXXXXXXXX.US
        renew until 06/25/2018 13:04:36
[kn857sa@dotstohdaut202 log]$

Re: Phoenix connection in kerberos cluster failing

sandyy006 — Tue, 19 Jun 2018 00:29:19 GMT

Then there is some issue with the keytab. In the second attempt a password was provided instead of keytab.

Please check if its a valid keytab and make the kinit command. Then you can use the same in phoenix connection.

Re: Phoenix connection in kerberos cluster failing

aliyesami — Tue, 19 Jun 2018 00:38:19 GMT

if i do a klist -k i do see its going to the /etc/krb5.keytab but i don't have privilege to read it .

how else can i check ?

Re: Phoenix connection in kerberos cluster failing

sandyy006 — Tue, 19 Jun 2018 00:44:00 GMT

You'd need to get the read access to the keytab file.

Re: Phoenix connection in kerberos cluster failing

aliyesami — Tue, 19 Jun 2018 00:56:37 GMT

if i do "kinit " does it use a keytab file . .if yes which one ?

Re: Phoenix connection in kerberos cluster failing

sandyy006 — Tue, 19 Jun 2018 01:05:40 GMT

keyatab should be owned by the owner, If that keytab belongs to your user. you should have read permissions. Pls check more on this with your system admins.

Re: Phoenix connection in kerberos cluster failing

aliyesami — Tue, 19 Jun 2018 01:08:33 GMT

when i do a "kinit" am i not using a keytab? if yes which keytab i am using?

Re: Phoenix connection in kerberos cluster failing

vmurakami — Tue, 19 Jun 2018 01:25:06 GMT

Hey @Sami Ahmad!
So answering your questions, usually when you use kinit -kt command you're passing a keytab file, and when you don't use it -kt (only kinit), you will use a password authentication.
In all of my jobs when I had a kerberized environment, usually the sysadmin gave a keytab file to my user or service. It's a common practice, since most of the components in Hadoop works pretty well with keytabs instead of passphrases for principals in KRB.
In your case, if you don't have a keytab, the best approach would be to ask for the sysadmin to generate a keytab for you.