Beeline connection fails with: Peer indicated failure: Unsupported mechanism type PLAIN

[#1]

Hi,

I am running a fresh installed HDP cluster with Kerberos enabled.

I try to connect to Hive using beeline command:

beeline -u jdbc:hive2://deala01876.corp:10000/default;principal=hive/deala01876.corp@HDP.REALM

I receive the following error:

scan complete in 5ms
Connecting to jdbc:hive2://deala01876.corp:10000/default
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hdp/2.2.4.2-2/hadoop/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/2.2.4.2-2/hive/lib/hive-jdbc-0.14.0.2.2.4.2-2-standalone.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings... for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Error: Could not open client transport with JDBC Uri: jdbc:hive2://deala01876.corp:10000/default: Peer indicated failure: Unsupported mechanism type PLAIN (state=08S01,code=0)
Beeline version 0.14.0.2.2.4.2-2 by Apache Hive
0: jdbc:hive2://deala01876.corp:1 (closed)> 

The local user has a valid Kerberos ticket, as well as the hive user on the Hiveserver node.

HiveServer2 authentication is set to "Kerberos", and property "hive.server2.authentication.kerberos.principal" is set to value " hive/_HOST@HDP.REALM"

What is going wrong here, what to check further ?

[#2]

@Gerd Koenig please paste output of klist

make sure that you have correct/valid ticket.

[#3]

Hi @Neeraj Sabharwal ,

here the details from the client node, from which I want to execute beeline:

W999711@DEALA01885:~$ kinit -kt /etc/security/keytabs/w999711.user.keytab w999711
W999711@DEALA01885:~$ klist
Ticket cache: FILE:/tmp/krb5cc_2001012
Default principal: w999711@HDP.REALM
Valid starting     Expires            Service principal
12/20/15 11:48:17  12/21/15 11:48:17  krbtgt/HDP.REALM@HDP.REALM
        renew until 12/20/15 11:48:17
W999711@DEALA01885:~$ beeline -u jdbc:hive2://deala01876.corp:10000/default;principal=hive/deala01876.corp@HDP.REALM
scan complete in 5ms
Connecting to jdbc:hive2://deala01876.corp:10000/default
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hdp/2.2.4.2-2/hadoop/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/2.2.4.2-2/hive/lib/hive-jdbc-0.14.0.2.2.4.2-2-standalone.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Error: Could not open client transport with JDBC Uri: jdbc:hive2://deala01876.corp:10000/default: Peer indicated failure: Unsupported mechanism type PLAIN (state=08S01,code=0)
Beeline version 0.14.0.2.2.4.2-2 by Apache Hive
0: jdbc:hive2://deala01876.corp:1 (closed)>

and here the details from the node running Hiveserver2:

$ klist
Ticket cache: FILE:/tmp/krb5cc_16940
Default principal: hive/deala01876.corp@HDP.REALM
Valid starting     Expires            Service principal
12/19/15 20:46:28  12/20/15 20:46:28  krbtgt/HDP.REALM@HDP.REALM
        renew until 12/19/15 20:46:28
hive@DEALA01876:/home/hive 0
$ tail -f /var/log/hadooplogs/hive/hiveserver2.log
...
2015-12-20 11:53:03,660 ERROR [HiveServer2-Handler-Pool: Thread-56]: server.TThreadPoolServer (TThreadPoolServer.java:run(215)) - Error occurred during processing of message.
java.lang.RuntimeException: org.apache.thrift.transport.TTransportException: Unsupported mechanism type PLAIN
        at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$HiveSaslServerTransportFactory.getTransport(HadoopThriftAuthBridge.java:180)
        at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge20S.java:726)
        at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge20S.java:723)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:356)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1608)
        at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge20S.java:723)
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:189)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.thrift.transport.TTransportException: Unsupported mechanism type PLAIN
        at org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:221)
        at org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:138)
        at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:253)
        at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
        at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$HiveSaslServerTransportFactory.getTransport(HadoopThriftAuthBridge.java:177)
        ... 10 more

[#4]

@Gerd Koenig

In the hiveserver2 server, Get the ticket using hive keytab and then try to login to beeline. Paste the output of klist once you get the ticket

[#5]

Hi @Neeraj Sabharwal ,

on the Hiveserver2 server I have no keytab available for my personal user-id (w999711) with which I want to create a beeline connection. Therefore I tried to connect as (OS-)user 'hive' via beeline, but receive the same error message:

$ kdestroy
hive@DEALA01876:/home/hive 0
$ klist
klist: No credentials cache found (ticket cache FILE:/tmp/krb5cc_16940)
hive@DEALA01876:/home/hive 1
$ kinit -kt /etc/security/keytabs/hive.service.keytab hive/deala01876.corp
hive@DEALA01876:/home/hive 0
$ klist
Ticket cache: FILE:/tmp/krb5cc_16940
Default principal: hive/deala01876.corp@HDP.REALM
Valid starting     Expires            Service principal
12/20/15 15:29:05  12/21/15 15:29:05  krbtgt/HDP.REALM@HDP.REALM
        renew until 12/20/15 15:29:05
hive@DEALA01876:/home/hive 0
$ beeline -u jdbc:hive2://deala01876.corp:10000/default;principal=hive/deala01876.corp@HDP.REALM
scan complete in 5ms
Connecting to jdbc:hive2://deala01876.corp:10000/default
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hdp/2.2.4.2-2/hadoop/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/2.2.4.2-2/hive/lib/hive-jdbc-0.14.0.2.2.4.2-2-standalone.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings. for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Error: Could not open client transport with JDBC Uri: jdbc:hive2://deala01876.corp:10000/default: Peer indicated failure: Unsupported mechanism type PLAIN (state=08S01,code=0)
Beeline version 0.14.0.2.2.4.2-2 by Apache Hive
0: jdbc:hive2://deala01876.corp:1 (closed)>

..and below the corresponding config values from /etc/hive/conf.server/

/etc/hive/conf.server/hive-site.xml-    <property>
/etc/hive/conf.server/hive-site.xml:      <name>hive.server2.authentication</name>
/etc/hive/conf.server/hive-site.xml-      <value>KERBEROS</value>
/etc/hive/conf.server/hive-site.xml-    </property>
/etc/hive/conf.server/hive-site.xml-
/etc/hive/conf.server/hive-site.xml-    <property>
/etc/hive/conf.server/hive-site.xml:      <name>hive.server2.authentication.kerberos.keytab</name>
/etc/hive/conf.server/hive-site.xml-      <value>/etc/security/keytabs/hive.service.keytab</value>
/etc/hive/conf.server/hive-site.xml-    </property>
/etc/hive/conf.server/hive-site.xml-
/etc/hive/conf.server/hive-site.xml-    <property>
/etc/hive/conf.server/hive-site.xml:      <name>hive.server2.authentication.kerberos.principal</name>
/etc/hive/conf.server/hive-site.xml-      <value>hive/_HOST@HDP.REALM</value>
/etc/hive/conf.server/hive-site.xml-    </property>
/etc/hive/conf.server/hive-site.xml-
/etc/hive/conf.server/hive-site.xml-    <property>
/etc/hive/conf.server/hive-site.xml:      <name>hive.server2.authentication.spnego.keytab</name>
/etc/hive/conf.server/hive-site.xml-      <value>/etc/security/keytabs/spnego.service.keytab</value>
/etc/hive/conf.server/hive-site.xml-    </property>
/etc/hive/conf.server/hive-site.xml-
/etc/hive/conf.server/hive-site.xml-    <property>
/etc/hive/conf.server/hive-site.xml:      <name>hive.server2.authentication.spnego.principal</name>
/etc/hive/conf.server/hive-site.xml-      <value>HTTP/_HOST@HDP.REALM</value>
/etc/hive/conf.server/hive-site.xml-    </property>
/etc/hive/conf.server/hive-site.xml-
/etc/hive/conf.server/hive-site.xml-    <property>
/etc/hive/conf.server/hive-site.xml:      <name>hive.server2.enable.doAs</name>
/etc/hive/conf.server/hive-site.xml-      <value>true</value>
/etc/hive/conf.server/hive-site.xml-    </property>

[#6]

Hi @Gerd Koenig ,

From the beeline log, it is clear that the beeline is still not able to understand that you want to communication over Kerberos. Can you please post the output of same the beeline command with '-v' flag?

[#7]

Hi @Vipin Rathor,

thanks for jumping in 😄 . Please find the output below (unfortunately not that meaningful.....):

W999711@DEALA01885:~$ beeline --verbose=true -u jdbc:hive2://deala01876.corp:10000/default;principal=hive/_HOST@HDP.REALM                
issuing: !connect jdbc:hive2://deala01876.corp:10000/default '' ''
scan complete in 5ms
Connecting to jdbc:hive2://deala01876.corp:10000/default
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hdp/2.2.4.2-2/hadoop/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/2.2.4.2-2/hive/lib/hive-jdbc-0.14.0.2.2.4.2-2-standalone.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings... for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Error: Could not open client transport with JDBC Uri: jdbc:hive2://deala01876.corp:10000/default: Peer indicated failure: Unsupported mechanism type PLAIN (state=08S01,code=0)
java.sql.SQLException: Could not open client transport with JDBC Uri: jdbc:hive2://deala01876.corp:10000/default: Peer indicated failure: Unsupported mechanism type PLAIN
        at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:215)
        at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:163)
        at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)
        at java.sql.DriverManager.getConnection(DriverManager.java:571)
        at java.sql.DriverManager.getConnection(DriverManager.java:187)
        at org.apache.hive.beeline.DatabaseConnection.connect(DatabaseConnection.java:138)
        at org.apache.hive.beeline.DatabaseConnection.getConnection(DatabaseConnection.java:179)
        at org.apache.hive.beeline.Commands.connect(Commands.java:1078)
        at org.apache.hive.beeline.Commands.connect(Commands.java:999)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hive.beeline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:45)
        at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:936)
        at org.apache.hive.beeline.BeeLine.initArgs(BeeLine.java:698)
        at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:748)
        at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:476)
        at org.apache.hive.beeline.BeeLine.main(BeeLine.java:459)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: org.apache.thrift.transport.TTransportException: Peer indicated failure: Unsupported mechanism type PLAIN
        at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:190)
        at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:288)
        at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
        at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:190)
        ... 24 more
Beeline version 0.14.0.2.2.4.2-2 by Apache Hive
0: jdbc:hive2://deala01876.corp:1 (closed)>

...and the error message is the same, either using "_HOST" or the real hostname "deala01876.corp" in the principal.

[#8 · Accepted solution]

@Vipin Rathor , @Neeraj Sabharwal,

it is solved.....and what a stupid cause 😉

The connection can be established if I put quotes around the JDBC URL =>

W999711@DEALA01885:~$ beeline -u "jdbc:hive2://deala01876.corp:10000/default;principal=hive/deala01876.corp@HDP.REALM"
scan complete in 5ms
Connecting to jdbc:hive2://deala01876.corp:10000/default;principal=hive/deala01876.corp@HDP.REALM
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hdp/2.2.4.2-2/hadoop/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/2.2.4.2-2/hive/lib/hive-jdbc-0.14.0.2.2.4.2-2-standalone.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings. for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Connected to: Apache Hive (version 0.14.0.2.2.4.2-2)
Driver: Hive JDBC (version 0.14.0.2.2.4.2-2)
Transaction isolation: TRANSACTION_REPEATABLE_READ
Beeline version 0.14.0.2.2.4.2-2 by Apache Hive
0: jdbc:hive2://deala01876.corp:1>

sorry for causing any confusion 😄

[#9]

@Gerd Koenig That's :))

[#10]

Did you figure out why this happens though? Is @ a special character?