Impala JDBC fails to connect with wrong service principal during TGS

[#1]

Hi,

 I have a good old GSS initiate failed Keberos error message when my application (JAVA) tries to connect to Impala via JDBC. I tried to eliminate all the usuall root causes but this time I think I missed something, because it still not connects: 

Cannot connect: connection refused: Java::JavaSql::SQLException: [Cloudera][ImpalaJDBCDriver](500164) Error initialized or created transport for authentication: [Cloudera][ImpalaJDBCDriver](500169) Unable to connect to server: GSS initiate failed.

The settings are quite standard, MIT Kerberos, kerberized Impala with SSL, here are the params for the JDBC:

-> KDC host: ip-10-85-150-11.eu-west-1.compute.internal
-> ImpaladD host: ip-10-85-150-6.eu-west-1.compute.internal

JDBC params:
-> Host: ip-10-85-150-6.eu-west-1.compute.internal 
-> Port: 21050
-> Additional params in URL: KrbHostFQDN=ip-10-85-150-11.eu-west-1.compute.internal;KrbRealm=HADOOP.DEV.REALM.LOCALL;KrbServiceName=impala;SSL=1;CAIssuedCertNamesMismatch=1;AuthMech=1;LogLevel=6;AllowSelfSignedCerts=1;SSLTrustStore=/var/lib/cloudera-scm-agent/agent-cert/cm-auto-global_truststore.jks;SSLTrustStorePwd=xxx;LogPath=/tmp/jdbc.log

The user running the app has a valid ticket:

-> Klist
[myapp@ip-10-85-150-42 ~]$ klist
Ticket cache: FILE:/tmp/krb5cc_1004
Default principal: myapp/ip-10-85-150-42.eu-west-1.compute.internal@HADOOP.DEV.REALM.LOCALL

Valid starting       Expires              Service principal
01/18/2019 09:31:21  01/19/2019 09:31:21  krbtgt/[email protected]
        renew until 01/25/2019 09:31:21

I verified the params are correctly passed to the JDBC, as it can be checked in the JDBC debug log:

Jan 18 13:33:35.428 TRACE 171 com.cloudera.impala.jdbc.common.CommonCoreUtils.logConnectionFunctionEntrance({
    AllowSelfSignedCerts=Variant[type: TYPE_WSTRING, value: 1], 
    AuthMech=Variant[type: TYPE_WSTRING, value: 1], 
    CAIssuedCertNamesMismatch=Variant[type: TYPE_WSTRING, value: 1], 
    ConnSchema=Variant[type: TYPE_WSTRING, value: analytics], 
    DatabaseType=Variant[type: TYPE_WSTRING, value: Impala], 
    HiveServerType=Variant[type: TYPE_WSTRING, value: 2], 
    Host=Variant[type: TYPE_WSTRING, value: ip-10-85-150-6.eu-west-1.compute.internal], 
    KrbHostFQDN=Variant[type: TYPE_WSTRING, value: ip-10-85-150-11.eu-west-1.compute.internal], 
    KrbRealm=Variant[type: TYPE_WSTRING, value: HADOOP.DEV.REALM.LOCALL], 
    KrbServiceName=Variant[type: TYPE_WSTRING, value: impala], 
    LogLevel=Variant[type: TYPE_WSTRING, value: 6], 
    LogPath=Variant[type: TYPE_WSTRING, value: /tmp/jdbc.log], 
    Port=Variant[type: TYPE_WSTRING, value: 21050], 
    SSL=Variant[type: TYPE_WSTRING, value: 1], 
    SSLTrustStore=Variant[type: TYPE_WSTRING, value: /var/lib/cloudera-scm-agent/agent-cert/cm-auto-global_truststore.jks], 
    SSLTrustStorePwd=Variant[type: TYPE_WSTRING, value: xxxxxxxxxx], 
    UseNativeQuery=Variant[type: TYPE_WSTRING, value: 1]}, 
    "Major Version: 2", "Minor Version: 6", "Hot Fix Version: 4", "Build Number: 1005", "java.vendor:Oracle Corporation", "java.version:1.8.0_191", "os.arch:amd64", 
    "os.name:Linux", "os.version:3.10.0-862.14.4.el7.x86_64", "Runtime.totalMemory:2097152000", "Runtime.maxMemory:2097152000", "Runtime.avaialableProcessors:4", 
    URLClassLoader.getURLs(): /home/myapp/myapp/myapp.jar): +++++ enter +++++
Jan 18 13:33:35.429 TRACE 171 com.cloudera.impala.dsi.core.impl.DSIConnection.getProperty(170): +++++ enter +++++
Jan 18 13:33:35.431 DEBUG 171 com.cloudera.impala.hivecommon.core.HiveJDBCCommonConnection.establishConnection: socketTimeout = 0, loginTimeout = 0
Jan 18 13:33:35.432 DEBUG 171 com.cloudera.impala.hivecommon.core.HiveJDBCCommonConnection.establishConnection: SocketTimeout is: 0 seconds for test
Jan 18 13:33:35.434 TRACE 171 com.cloudera.impala.jdbc.kerberos.Kerberos.getSubjectViaAccessControlContext(): +++++ enter +++++
Jan 18 13:33:35.436 TRACE 171 com.cloudera.impala.jdbc.kerberos.Kerberos.getSubjectViaJAASConfig(): +++++ enter +++++
Jan 18 13:33:35.437 DEBUG 171 com.cloudera.impala.jdbc.kerberos.Kerberos.getSubjectViaJAASConfig: System.getProperty(java.security.auth.login.config): /home/myapp/myapp/gss-jaas.conf
Jan 18 13:33:35.440 DEBUG 171 com.cloudera.impala.hivecommon.api.HiveServer2ClientFactory.createTransport: Kerberos subject retrieved via JAAS config
Jan 18 13:33:35.572 ERROR 171 com.cloudera.impala.exceptions.ExceptionConverter.toSQLException: [Cloudera][ImpalaJDBCDriver](500164) Error initialized or created transport for authentication: [Cloudera][ImpalaJDBCDriver](500169) Unable to connect to server: GSS initiate failed.
java.sql.SQLException: [Cloudera][ImpalaJDBCDriver](500164) Error initialized or created transport for authentication: [Cloudera][ImpalaJDBCDriver](500169) Unable to connect to server: GSS initiate failed.

The params for the Java app:

[myapp@ip-10-85-150-42 ~]$ cat myapp/gss-jaas.conf
Client {
    com.sun.security.auth.module.Krb5LoginModule required
    useTicketCache=true
    doNotPrompt=true
    debug=true;
};

The JAVA args:
-Djava.security.auth.login.config=/home/myapp/myapp/gss-jaas.conf -Djavax.security.auth.useSubjectCredsOnly=false -Djava.security.krb5.conf=/etc/krb5.conf -Dsun.security.jgss.debug=true -Dsun.security.krb5.debug=true

And here is the issue - KDC log:

-> KDC log:
Jan 18 13:33:34 ip-10-85-150-11.eu-west-1.compute.internal krb5kdc[13372](info): TGS_REQ (2 etypes {18 17}) 10.85.150.42: LOOKING_UP_SERVER: authtime 0,  myapp/ip-10-85-150-42.eu-west-1.compute.internal@HADOOP.DEV.REALM.LOCALL for impala/ip-10-85-150-11.eu-west-1.compute.internal@HADOOP.DEV.REALM.LOCALL, Server not found in Kerberos database
Jan 18 13:33:34 ip-10-85-150-11.eu-west-1.compute.internal krb5kdc[13372](info): TGS_REQ (2 etypes {18 17}) 10.85.150.42: LOOKING_UP_SERVER: authtime 0,  myapp/ip-10-85-150-42.eu-west-1.compute.internal@HADOOP.DEV.REALM.LOCALL for impala/ip-10-85-150-11.eu-west-1.compute.internal@HADOOP.DEV.REALM.LOCALL, Server not found in Kerberos database
Jan 18 13:33:35 ip-10-85-150-11.eu-west-1.compute.internal krb5kdc[13372](info): TGS_REQ (2 etypes {18 17}) 10.85.150.42: LOOKING_UP_SERVER: authtime 0,  myapp/ip-10-85-150-42.eu-west-1.compute.internal@HADOOP.DEV.REALM.LOCALL for impala/ip-10-85-150-11.eu-west-1.compute.internal@HADOOP.DEV.REALM.LOCALL, Server not found in Kerberos database

During the TGS_REQ the KDC is trying to look up for impala/<KDCHOST> instead of impala/<IMPALADHOST> principal. 

I double checked everything, tried variious versions of JDBC driver, but the result is the same. 

Any hints would be welcome,

Thanks

[#2 · Accepted solution]

Annoying bug, but very simple solution, made a mistake in KrbHostFQDN. That should be the impalad fqdn.