Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Impala JDBC fails to connect with wrong service principal during TGS

avatar

Hi,

 I have a good old GSS initiate failed Keberos error message when my application (JAVA) tries to connect to Impala via JDBC. I tried to eliminate all the usuall root causes but this time I think I missed something, because it still not connects: 

 

 

Cannot connect: connection refused: Java::JavaSql::SQLException: [Cloudera][ImpalaJDBCDriver](500164) Error initialized or created transport for authentication: [Cloudera][ImpalaJDBCDriver](500169) Unable to connect to server: GSS initiate failed.

The settings are quite standard, MIT Kerberos, kerberized Impala with SSL, here are the params for the JDBC:

 

-> KDC host: ip-10-85-150-11.eu-west-1.compute.internal
-> ImpaladD host: ip-10-85-150-6.eu-west-1.compute.internal

JDBC params:
-> Host: ip-10-85-150-6.eu-west-1.compute.internal 
-> Port: 21050
-> Additional params in URL: KrbHostFQDN=ip-10-85-150-11.eu-west-1.compute.internal;KrbRealm=HADOOP.DEV.REALM.LOCALL;KrbServiceName=impala;SSL=1;CAIssuedCertNamesMismatch=1;AuthMech=1;LogLevel=6;AllowSelfSignedCerts=1;SSLTrustStore=/var/lib/cloudera-scm-agent/agent-cert/cm-auto-global_truststore.jks;SSLTrustStorePwd=xxx;LogPath=/tmp/jdbc.log

The user running the app has a valid ticket:

 

-> Klist
[myapp@ip-10-85-150-42 ~]$ klist
Ticket cache: FILE:/tmp/krb5cc_1004
Default principal: myapp/ip-10-85-150-42.eu-west-1.compute.internal@HADOOP.DEV.REALM.LOCALL

Valid starting       Expires              Service principal
01/18/2019 09:31:21  01/19/2019 09:31:21  krbtgt/HADOOP.DEV.REALM.LOCALL@HADOOP.DEV.REALM.LOCALL
        renew until 01/25/2019 09:31:21

I verified the params are correctly passed to the JDBC, as it can be checked in the JDBC debug log:

 

Jan 18 13:33:35.428 TRACE 171 com.cloudera.impala.jdbc.common.CommonCoreUtils.logConnectionFunctionEntrance({
    AllowSelfSignedCerts=Variant[type: TYPE_WSTRING, value: 1], 
    AuthMech=Variant[type: TYPE_WSTRING, value: 1], 
    CAIssuedCertNamesMismatch=Variant[type: TYPE_WSTRING, value: 1], 
    ConnSchema=Variant[type: TYPE_WSTRING, value: analytics], 
    DatabaseType=Variant[type: TYPE_WSTRING, value: Impala], 
    HiveServerType=Variant[type: TYPE_WSTRING, value: 2], 
    Host=Variant[type: TYPE_WSTRING, value: ip-10-85-150-6.eu-west-1.compute.internal], 
    KrbHostFQDN=Variant[type: TYPE_WSTRING, value: ip-10-85-150-11.eu-west-1.compute.internal], 
    KrbRealm=Variant[type: TYPE_WSTRING, value: HADOOP.DEV.REALM.LOCALL], 
    KrbServiceName=Variant[type: TYPE_WSTRING, value: impala], 
    LogLevel=Variant[type: TYPE_WSTRING, value: 6], 
    LogPath=Variant[type: TYPE_WSTRING, value: /tmp/jdbc.log], 
    Port=Variant[type: TYPE_WSTRING, value: 21050], 
    SSL=Variant[type: TYPE_WSTRING, value: 1], 
    SSLTrustStore=Variant[type: TYPE_WSTRING, value: /var/lib/cloudera-scm-agent/agent-cert/cm-auto-global_truststore.jks], 
    SSLTrustStorePwd=Variant[type: TYPE_WSTRING, value: xxxxxxxxxx], 
    UseNativeQuery=Variant[type: TYPE_WSTRING, value: 1]}, 
    "Major Version: 2", "Minor Version: 6", "Hot Fix Version: 4", "Build Number: 1005", "java.vendor:Oracle Corporation", "java.version:1.8.0_191", "os.arch:amd64", 
    "os.name:Linux", "os.version:3.10.0-862.14.4.el7.x86_64", "Runtime.totalMemory:2097152000", "Runtime.maxMemory:2097152000", "Runtime.avaialableProcessors:4", 
    URLClassLoader.getURLs(): /home/myapp/myapp/myapp.jar): +++++ enter +++++
Jan 18 13:33:35.429 TRACE 171 com.cloudera.impala.dsi.core.impl.DSIConnection.getProperty(170): +++++ enter +++++
Jan 18 13:33:35.431 DEBUG 171 com.cloudera.impala.hivecommon.core.HiveJDBCCommonConnection.establishConnection: socketTimeout = 0, loginTimeout = 0
Jan 18 13:33:35.432 DEBUG 171 com.cloudera.impala.hivecommon.core.HiveJDBCCommonConnection.establishConnection: SocketTimeout is: 0 seconds for test
Jan 18 13:33:35.434 TRACE 171 com.cloudera.impala.jdbc.kerberos.Kerberos.getSubjectViaAccessControlContext(): +++++ enter +++++
Jan 18 13:33:35.436 TRACE 171 com.cloudera.impala.jdbc.kerberos.Kerberos.getSubjectViaJAASConfig(): +++++ enter +++++
Jan 18 13:33:35.437 DEBUG 171 com.cloudera.impala.jdbc.kerberos.Kerberos.getSubjectViaJAASConfig: System.getProperty(java.security.auth.login.config): /home/myapp/myapp/gss-jaas.conf
Jan 18 13:33:35.440 DEBUG 171 com.cloudera.impala.hivecommon.api.HiveServer2ClientFactory.createTransport: Kerberos subject retrieved via JAAS config
Jan 18 13:33:35.572 ERROR 171 com.cloudera.impala.exceptions.ExceptionConverter.toSQLException: [Cloudera][ImpalaJDBCDriver](500164) Error initialized or created transport for authentication: [Cloudera][ImpalaJDBCDriver](500169) Unable to connect to server: GSS initiate failed.
java.sql.SQLException: [Cloudera][ImpalaJDBCDriver](500164) Error initialized or created transport for authentication: [Cloudera][ImpalaJDBCDriver](500169) Unable to connect to server: GSS initiate failed.

The params for the Java app:

[myapp@ip-10-85-150-42 ~]$ cat myapp/gss-jaas.conf
Client {
    com.sun.security.auth.module.Krb5LoginModule required
    useTicketCache=true
    doNotPrompt=true
    debug=true;
};

The JAVA args:
-Djava.security.auth.login.config=/home/myapp/myapp/gss-jaas.conf -Djavax.security.auth.useSubjectCredsOnly=false -Djava.security.krb5.conf=/etc/krb5.conf -Dsun.security.jgss.debug=true -Dsun.security.krb5.debug=true

And here is the issue - KDC log:

-> KDC log:
Jan 18 13:33:34 ip-10-85-150-11.eu-west-1.compute.internal krb5kdc[13372](info): TGS_REQ (2 etypes {18 17}) 10.85.150.42: LOOKING_UP_SERVER: authtime 0,  myapp/ip-10-85-150-42.eu-west-1.compute.internal@HADOOP.DEV.REALM.LOCALL for impala/ip-10-85-150-11.eu-west-1.compute.internal@HADOOP.DEV.REALM.LOCALL, Server not found in Kerberos database
Jan 18 13:33:34 ip-10-85-150-11.eu-west-1.compute.internal krb5kdc[13372](info): TGS_REQ (2 etypes {18 17}) 10.85.150.42: LOOKING_UP_SERVER: authtime 0,  myapp/ip-10-85-150-42.eu-west-1.compute.internal@HADOOP.DEV.REALM.LOCALL for impala/ip-10-85-150-11.eu-west-1.compute.internal@HADOOP.DEV.REALM.LOCALL, Server not found in Kerberos database
Jan 18 13:33:35 ip-10-85-150-11.eu-west-1.compute.internal krb5kdc[13372](info): TGS_REQ (2 etypes {18 17}) 10.85.150.42: LOOKING_UP_SERVER: authtime 0,  myapp/ip-10-85-150-42.eu-west-1.compute.internal@HADOOP.DEV.REALM.LOCALL for impala/ip-10-85-150-11.eu-west-1.compute.internal@HADOOP.DEV.REALM.LOCALL, Server not found in Kerberos database

During the TGS_REQ the KDC is trying to look up for impala/<KDCHOST> instead of impala/<IMPALADHOST> principal. 

 

I double checked everything, tried variious versions of JDBC driver, but the result is the same. 

Any hints would be welcome,

Thanks

 

 

 

1 ACCEPTED SOLUTION

avatar
Annoying bug, but very simple solution, made a mistake in KrbHostFQDN. That should be the impalad fqdn.

View solution in original post

1 REPLY 1

avatar
Annoying bug, but very simple solution, made a mistake in KrbHostFQDN. That should be the impalad fqdn.