Hi,
I have a good old GSS initiate failed Keberos error message when my application (JAVA) tries to connect to Impala via JDBC. I tried to eliminate all the usuall root causes but this time I think I missed something, because it still not connects:
Cannot connect: connection refused: Java::JavaSql::SQLException: [Cloudera][ImpalaJDBCDriver](500164) Error initialized or created transport for authentication: [Cloudera][ImpalaJDBCDriver](500169) Unable to connect to server: GSS initiate failed.
The settings are quite standard, MIT Kerberos, kerberized Impala with SSL, here are the params for the JDBC:
-> KDC host: ip-10-85-150-11.eu-west-1.compute.internal
-> ImpaladD host: ip-10-85-150-6.eu-west-1.compute.internal
JDBC params:
-> Host: ip-10-85-150-6.eu-west-1.compute.internal
-> Port: 21050
-> Additional params in URL: KrbHostFQDN=ip-10-85-150-11.eu-west-1.compute.internal;KrbRealm=HADOOP.DEV.REALM.LOCALL;KrbServiceName=impala;SSL=1;CAIssuedCertNamesMismatch=1;AuthMech=1;LogLevel=6;AllowSelfSignedCerts=1;SSLTrustStore=/var/lib/cloudera-scm-agent/agent-cert/cm-auto-global_truststore.jks;SSLTrustStorePwd=xxx;LogPath=/tmp/jdbc.log
The user running the app has a valid ticket:
-> Klist
[myapp@ip-10-85-150-42 ~]$ klist
Ticket cache: FILE:/tmp/krb5cc_1004
Default principal: myapp/ip-10-85-150-42.eu-west-1.compute.internal@HADOOP.DEV.REALM.LOCALL
Valid starting Expires Service principal
01/18/2019 09:31:21 01/19/2019 09:31:21 krbtgt/HADOOP.DEV.REALM.LOCALL@HADOOP.DEV.REALM.LOCALL
renew until 01/25/2019 09:31:21
I verified the params are correctly passed to the JDBC, as it can be checked in the JDBC debug log:
Jan 18 13:33:35.428 TRACE 171 com.cloudera.impala.jdbc.common.CommonCoreUtils.logConnectionFunctionEntrance({
AllowSelfSignedCerts=Variant[type: TYPE_WSTRING, value: 1],
AuthMech=Variant[type: TYPE_WSTRING, value: 1],
CAIssuedCertNamesMismatch=Variant[type: TYPE_WSTRING, value: 1],
ConnSchema=Variant[type: TYPE_WSTRING, value: analytics],
DatabaseType=Variant[type: TYPE_WSTRING, value: Impala],
HiveServerType=Variant[type: TYPE_WSTRING, value: 2],
Host=Variant[type: TYPE_WSTRING, value: ip-10-85-150-6.eu-west-1.compute.internal],
KrbHostFQDN=Variant[type: TYPE_WSTRING, value: ip-10-85-150-11.eu-west-1.compute.internal],
KrbRealm=Variant[type: TYPE_WSTRING, value: HADOOP.DEV.REALM.LOCALL],
KrbServiceName=Variant[type: TYPE_WSTRING, value: impala],
LogLevel=Variant[type: TYPE_WSTRING, value: 6],
LogPath=Variant[type: TYPE_WSTRING, value: /tmp/jdbc.log],
Port=Variant[type: TYPE_WSTRING, value: 21050],
SSL=Variant[type: TYPE_WSTRING, value: 1],
SSLTrustStore=Variant[type: TYPE_WSTRING, value: /var/lib/cloudera-scm-agent/agent-cert/cm-auto-global_truststore.jks],
SSLTrustStorePwd=Variant[type: TYPE_WSTRING, value: xxxxxxxxxx],
UseNativeQuery=Variant[type: TYPE_WSTRING, value: 1]},
"Major Version: 2", "Minor Version: 6", "Hot Fix Version: 4", "Build Number: 1005", "java.vendor:Oracle Corporation", "java.version:1.8.0_191", "os.arch:amd64",
"os.name:Linux", "os.version:3.10.0-862.14.4.el7.x86_64", "Runtime.totalMemory:2097152000", "Runtime.maxMemory:2097152000", "Runtime.avaialableProcessors:4",
URLClassLoader.getURLs(): /home/myapp/myapp/myapp.jar): +++++ enter +++++
Jan 18 13:33:35.429 TRACE 171 com.cloudera.impala.dsi.core.impl.DSIConnection.getProperty(170): +++++ enter +++++
Jan 18 13:33:35.431 DEBUG 171 com.cloudera.impala.hivecommon.core.HiveJDBCCommonConnection.establishConnection: socketTimeout = 0, loginTimeout = 0
Jan 18 13:33:35.432 DEBUG 171 com.cloudera.impala.hivecommon.core.HiveJDBCCommonConnection.establishConnection: SocketTimeout is: 0 seconds for test
Jan 18 13:33:35.434 TRACE 171 com.cloudera.impala.jdbc.kerberos.Kerberos.getSubjectViaAccessControlContext(): +++++ enter +++++
Jan 18 13:33:35.436 TRACE 171 com.cloudera.impala.jdbc.kerberos.Kerberos.getSubjectViaJAASConfig(): +++++ enter +++++
Jan 18 13:33:35.437 DEBUG 171 com.cloudera.impala.jdbc.kerberos.Kerberos.getSubjectViaJAASConfig: System.getProperty(java.security.auth.login.config): /home/myapp/myapp/gss-jaas.conf
Jan 18 13:33:35.440 DEBUG 171 com.cloudera.impala.hivecommon.api.HiveServer2ClientFactory.createTransport: Kerberos subject retrieved via JAAS config
Jan 18 13:33:35.572 ERROR 171 com.cloudera.impala.exceptions.ExceptionConverter.toSQLException: [Cloudera][ImpalaJDBCDriver](500164) Error initialized or created transport for authentication: [Cloudera][ImpalaJDBCDriver](500169) Unable to connect to server: GSS initiate failed.
java.sql.SQLException: [Cloudera][ImpalaJDBCDriver](500164) Error initialized or created transport for authentication: [Cloudera][ImpalaJDBCDriver](500169) Unable to connect to server: GSS initiate failed.
The params for the Java app:
[myapp@ip-10-85-150-42 ~]$ cat myapp/gss-jaas.conf
Client {
com.sun.security.auth.module.Krb5LoginModule required
useTicketCache=true
doNotPrompt=true
debug=true;
};
The JAVA args:
-Djava.security.auth.login.config=/home/myapp/myapp/gss-jaas.conf -Djavax.security.auth.useSubjectCredsOnly=false -Djava.security.krb5.conf=/etc/krb5.conf -Dsun.security.jgss.debug=true -Dsun.security.krb5.debug=true
And here is the issue - KDC log:
-> KDC log:
Jan 18 13:33:34 ip-10-85-150-11.eu-west-1.compute.internal krb5kdc[13372](info): TGS_REQ (2 etypes {18 17}) 10.85.150.42: LOOKING_UP_SERVER: authtime 0, myapp/ip-10-85-150-42.eu-west-1.compute.internal@HADOOP.DEV.REALM.LOCALL for impala/ip-10-85-150-11.eu-west-1.compute.internal@HADOOP.DEV.REALM.LOCALL, Server not found in Kerberos database
Jan 18 13:33:34 ip-10-85-150-11.eu-west-1.compute.internal krb5kdc[13372](info): TGS_REQ (2 etypes {18 17}) 10.85.150.42: LOOKING_UP_SERVER: authtime 0, myapp/ip-10-85-150-42.eu-west-1.compute.internal@HADOOP.DEV.REALM.LOCALL for impala/ip-10-85-150-11.eu-west-1.compute.internal@HADOOP.DEV.REALM.LOCALL, Server not found in Kerberos database
Jan 18 13:33:35 ip-10-85-150-11.eu-west-1.compute.internal krb5kdc[13372](info): TGS_REQ (2 etypes {18 17}) 10.85.150.42: LOOKING_UP_SERVER: authtime 0, myapp/ip-10-85-150-42.eu-west-1.compute.internal@HADOOP.DEV.REALM.LOCALL for impala/ip-10-85-150-11.eu-west-1.compute.internal@HADOOP.DEV.REALM.LOCALL, Server not found in Kerberos database
During the TGS_REQ the KDC is trying to look up for impala/<KDCHOST> instead of impala/<IMPALADHOST> principal.
I double checked everything, tried variious versions of JDBC driver, but the result is the same.
Any hints would be welcome,
Thanks
