Support Questions
Find answers, ask questions, and share your expertise

beeline and kerberos

Solved Go to solution
Highlighted

beeline and kerberos

Rising Star

I am trying to use beeline with hive + kerberos (Hortonworks sandbox 2.3)

The problem is that I can use hdfs but not beeline and I do not know what is wrong.

Console output:

[margusja@sandbox ~]$ kdestroy

[margusja@sandbox ~]$ hdfs dfs -ls /user/

16/01/09 15:45:32 WARN ipc.Client: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]

ls: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "sandbox.hortonworks.com/10.0.2.15"; destination host is: "sandbox.hortonworks.com":8020;

[margusja@sandbox ~]$ kinit margusja

Password for margusja@EXAMPLE.COM:

[margusja@sandbox ~]$ hdfs dfs -ls /user/

Found 11 items

drwxrwx--- - ambari-qa hdfs 0 2015-10-27 12:39 /user/ambari-qa

drwxr-xr-x - guest guest 0 2015-10-27 12:55 /user/guest

drwxr-xr-x - hcat hdfs 0 2015-10-27 12:43 /user/hcat

drwx------ - hdfs hdfs 0 2015-10-27 13:22 /user/hdfs

drwx------ - hive hdfs 0 2016-01-08 19:44 /user/hive

drwxrwxrwx - hue hdfs 0 2015-10-27 12:55 /user/hue

drwxrwxr-x - oozie hdfs 0 2015-10-27 12:44 /user/oozie

drwxr-xr-x - solr hdfs 0 2015-10-27 12:48 /user/solr

drwxrwxr-x - spark hdfs 0 2015-10-27 12:41 /user/spark

drwxr-xr-x - unit hdfs 0 2015-10-27 12:46 /user/unit

So I think margusja's credential is ok

[margusja@sandbox ~]$ klist -f Ticket cache: FILE:/tmp/krb5cc_1024 Default principal: margusja@EXAMPLE.COM Valid starting Expires Service principal 01/10/16 07:54:34 01/11/16 07:54:34 krbtgt/EXAMPLE.COM@EXAMPLE.COM renew until 01/17/16 07:54:34, Flags: FRI

Now I try to use beeline:

[margusja@sandbox ~]$ beeline -u "jdbc:hive2://127.0.0.1:10000/default;principal=hive/sandbox.hortonworks.com@EXAMPLE.COM"

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/usr/hdp/2.3.2.0-2950/spark/lib/spark-assembly-1.4.1.2.3.2.0-2950-hadoop2.7.1.2.3.2.0-2950.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/usr/hdp/2.3.2.0-2950/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

WARNING: Use "yarn jar" to launch YARN applications.

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/usr/hdp/2.3.2.0-2950/spark/lib/spark-assembly-1.4.1.2.3.2.0-2950-hadoop2.7.1.2.3.2.0-2950.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/usr/hdp/2.3.2.0-2950/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

Connecting to jdbc:hive2://127.0.0.1:10000/default;principal=hive/sandbox.hortonworks.com@EXAMPLE.COM

16/01/09 15:46:59 [main]: ERROR transport.TSaslTransport: SASL negotiation failure

javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]

at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)

at org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94)

at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)

at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)

at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)

at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:415)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)

at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49)

at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:210)

at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:180)

at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)

at java.sql.DriverManager.getConnection(DriverManager.java:571)

at java.sql.DriverManager.getConnection(DriverManager.java:187)

at org.apache.hive.beeline.DatabaseConnection.connect(DatabaseConnection.java:142)

at org.apache.hive.beeline.DatabaseConnection.getConnection(DatabaseConnection.java:207)

at org.apache.hive.beeline.Commands.connect(Commands.java:1149)

at org.apache.hive.beeline.Commands.connect(Commands.java:1070)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at org.apache.hive.beeline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:52)

at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:970)

at org.apache.hive.beeline.BeeLine.initArgs(BeeLine.java:707)

at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:757)

at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:484)

at org.apache.hive.beeline.BeeLine.main(BeeLine.java:467)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at org.apache.hadoop.util.RunJar.run(RunJar.java:221)

at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)

at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)

at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)

at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)

at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)

at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)

at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)

at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)

... 34 more

Error: Could not open client transport with JDBC Uri: jdbc:hive2://127.0.0.1:10000/default;principal=hive/sandbox.hortonworks.com@EXAMPLE.COM: GSS initiate failed (state=08S01,code=0)

Beeline version 1.2.1.2.3.2.0-2950 by Apache Hive

0: jdbc:hive2://127.0.0.1:10000/default (closed)>

Hive is configured as documentation requires:

<property>

<name>hive.server2.authentication</name>

<value>KERBEROS</value>

</property>

<property>

<name>hive.server2.authentication.kerberos.keytab</name>

<value>/etc/security/keytabs/hive.service.keytab</value>

</property>

<property>

<name>hive.server2.authentication.kerberos.principal</name>

<value>hive/_HOST@EXAMPLE.COM</value>

</property>

One more notice

When I do:

[margusja@sandbox ~]$ hdfs dfs -ls /

I see in krb5kdc log:

Jan 09 21:36:53 sandbox.hortonworks.com krb5kdc[8565](info): TGS_REQ (6 etypes {18 17 16 23 1 3}) 10.0.2.15: ISSUE: authtime 1452375310, etypes {rep=18 tkt=18 ses=18}, margusja@EXAMPLE.COM for nn/sandbox.hortonworks.com@EXAMPLE.COM

but when I use beeline I see there no lines in krb5kdc log.

When I do

[margusja@sandbox ~]$ kdestroy

and hdfs dfs -ls / - I see there no lines also in krb5kdc log.

I am so confused - What beeline expecting? I do kinit and I am getting ticket before using beeline.

Any hints, because I am out of ideas.

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: beeline and kerberos

Rising Star

I do not know is it solution here but one helpful think is to enable kerberos debug mode to see what kerberos wants:

export HADOOP_OPTS="-Dsun.security.krb5.debug=true"

It helped me

View solution in original post

19 REPLIES 19
Highlighted

Re: beeline and kerberos

@Margus Roo

ERROR transport.TSaslTransport: SASL negotiation failure

javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]

kinit using hive keytab and see if you can login.

Highlighted

Re: beeline and kerberos

@Margus Roo Thanks for trying that.

Try this

beeline then press enter

!connect jdbc:hive2://localhost:10000/;principal=hive/sandbox.hortonworks.com@EXAMPLE.COM

Highlighted

Re: beeline and kerberos

@Margus Roo Also, are you able to login using hive cli?

Highlighted

Re: beeline and kerberos

Rising Star

Hi

[root@sandbox ~]# kinit -kt /etc/security/keytabs/hive.service.keytab hive/sandbox.hortonworks.com@EXAMPLE.COM
[root@sandbox ~]# klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: hive/sandbox.hortonworks.com@EXAMPLE.COM
Valid starting     Expires            Service principal
01/10/16 12:21:27  01/11/16 12:21:27  krbtgt/EXAMPLE.COM@EXAMPLE.COM
renew until 01/17/16 12:21:27
Is it ok until now? Do I have valid ticket?
[root@sandbox ~]# beeline -u "jdbc:hive2://localhost:10000/;principal=hive/sandbox.hortonworks.com@EXAMPLE.COM" 
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hdp/2.3.2.0-2950/spark/lib/spark-assembly-1.4.1.2.3.2.0-2950-hadoop2.7.1.2.3.2.0-2950.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/2.3.2.0-2950/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
WARNING: Use "yarn jar" to launch YARN applications.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hdp/2.3.2.0-2950/spark/lib/spark-assembly-1.4.1.2.3.2.0-2950-hadoop2.7.1.2.3.2.0-2950.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/2.3.2.0-2950/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Connecting to jdbc:hive2://localhost:10000/;principal=hive/sandbox.hortonworks.com@EXAMPLE.COM
16/01/10 12:23:42 [main]: ERROR transport.TSaslTransport: SASL negotiation failure
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
at org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94)
at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)
at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)
at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49)
at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:210)
at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:180)
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)
at java.sql.DriverManager.getConnection(DriverManager.java:571)
at java.sql.DriverManager.getConnection(DriverManager.java:187)
at org.apache.hive.beeline.DatabaseConnection.connect(DatabaseConnection.java:142)
at org.apache.hive.beeline.DatabaseConnection.getConnection(DatabaseConnection.java:207)
at org.apache.hive.beeline.Commands.connect(Commands.java:1149)
at org.apache.hive.beeline.Commands.connect(Commands.java:1070)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hive.beeline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:52)
at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:970)
at org.apache.hive.beeline.BeeLine.initArgs(BeeLine.java:707)
at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:757)
at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:484)
at org.apache.hive.beeline.BeeLine.main(BeeLine.java:467)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)
at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)
... 34 more
Error: Could not open client transport with JDBC Uri: jdbc:hive2://localhost:10000/;principal=hive/sandbox.hortonworks.com@EXAMPLE.COM: GSS initiate failed (state=08S01,code=0)
Beeline version 1.2.1.2.3.2.0-2950 by Apache Hive
0: jdbc:hive2://localhost:10000/ (closed)>
:(
Highlighted

Re: beeline and kerberos

Rising Star

Hi and thanks for dialog.

I can log in using hive command.

And I see from /var/log/krb5kdc.log that there is communication. Using beeline there is silence.

beeline still gives: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]

I can not understand that message quite well. It is saying that there is no tgt.

I even made a new user for testing - margusja, because in some documentation recommended to use separated user than hive.

[margusja@sandbox ~]$ klist -f -e
Ticket cache: FILE:/tmp/krb5cc_1024
Default principal: margusja@EXAMPLE.COM
Valid starting     Expires            Service principal
01/10/16 16:05:29  01/11/16 16:05:29  krbtgt/EXAMPLE.COM@EXAMPLE.COM
renew until 01/17/16 16:05:29, Flags: FRI
Etype (skey, tkt): arcfour-hmac, aes256-cts-hmac-sha1-96

Above means that I have tgt?

How beeline checks tgt?

Any help is welcome.

Br, Margusja

Highlighted

Re: beeline and kerberos

Guru

Hi @Margus Roo ,

does the hive user on the Hiveserver node have a valid Kerberos ticket as well ?

Try to re-init one for user 'hive'.

I had similar issue in certain versions, where the ticket for user 'hive' hasn't been updated automatically....

Highlighted

Re: beeline and kerberos

Rising Star

Tried to re-init:

[margusja@sandbox ~]$ klist -e -f
Ticket cache: FILE:/tmp/krb5cc_1024
Default principal: margusja@EXAMPLE.COM
Valid starting     Expires            Service principal
01/10/16 16:21:10  01/11/16 16:21:10  krbtgt/EXAMPLE.COM@EXAMPLE.COM
renew until 01/17/16 16:21:10, Flags: FRI
Etype (skey, tkt): arcfour-hmac, aes256-cts-hmac-sha1-96
[margusja@sandbox ~]$

And I can re-init:

[margusja@sandbox ~]$ klist -e -f
Ticket cache: FILE:/tmp/krb5cc_1024
Default principal: margusja@EXAMPLE.COM
Valid starting     Expires            Service principal
01/10/16 16:34:54  01/11/16 16:34:54  krbtgt/EXAMPLE.COM@EXAMPLE.COM
renew until 01/17/16 16:21:10, Flags: FRIT
Etype (skey, tkt): arcfour-hmac, aes256-cts-hmac-sha1-96
[margusja@sandbox ~]$

unfortunately I have no success

beeline> !connect jdbc:hive2://127.0.0.1:10000/default;principal=hive/sandbox.hortonworks.com@EXAMPLE.COM
Connecting to jdbc:hive2://127.0.0.1:10000/default;principal=hive/sandbox.hortonworks.com@EXAMPLE.COM
Enter username for jdbc:hive2://127.0.0.1:10000/default;principal=hive/sandbox.hortonworks.com@EXAMPLE.COM:
Enter password for jdbc:hive2://127.0.0.1:10000/default;principal=hive/sandbox.hortonworks.com@EXAMPLE.COM:
16/01/10 16:35:36 [main]: ERROR transport.TSaslTransport: SASL negotiation failure
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]

I can not understand what is missing for beeline - "Failed to find any Kerberos tgt"

What beeline searching for? I have tgt in cache as you can see above.

Br, Margusja

Highlighted

Re: beeline and kerberos

Guru

Hi @Margus Roo , in my previous answer I meant to check the kerberos ticket for user 'hive', not for your personal user.

sudo su - hive
kdestroy
kinit -kt <path-to-keytab> hive/sandbox.hortonworks.com
klist

and then again the beeline command...

Highlighted

Re: beeline and kerberos

Can you try HQDN or hostname/IP instead of localhost or 127.0.0.1 in the beeline connect string?