Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Help connecting to Impala through impala-shell and jdbc in Kerberos/LDAP cluster

avatar
Expert Contributor

Hello everybody

 

I am working on a CDH 5.13.2 cluster configured with Kerberos and LDAP authentication.

 

I need to connect to Impala thorugh jdbc and impala-shell, but I am having problems on both (Impala queries on HUE work fine).

 

For impala-shell I've tried:

 

 

impala-shell -k -i trwor-b9a4f2a7.azcloud.local

--->

Starting Impala Shell using Kerberos authentication
Using service name 'impala'
Error connecting: TTransportException, TSocket read 0 bytes
***********************************************************************************
Welcome to the Impala shell.
(Impala Shell v2.10.0-cdh5.13.2 (dc867db) built on Fri Feb 2 10:46:38 PST 2018)

 

I've also tried without Kerberos:

 

impala-shell -i trwor-b9a4f2a7.azcloud.local

--->


Starting Impala Shell without Kerberos authentication
Error connecting: TTransportException, TSocket read 0 bytes
Kerberos ticket found in the credentials cache, retrying the connection with a secure transport.
Error connecting: TTransportException, TSocket read 0 bytes
***********************************************************************************
Welcome to the Impala shell.
(Impala Shell v2.10.0-cdh5.13.2 (dc867db) built on Fri Feb 2 10:46:38 PST 2018)

 

In both cases I got a TTransportException.

 

 

I am having trouble also for connecting to Impala through jdbc (using Cloudera_ImpalaJDBC4_2.5.5.1007 driver):

 

 

String impalaConnectionUrl = "jdbc:impala://trwor-dafb587f.azcloud.local:21050;AuthMech=1;KrbRealm=AZCLOUD.LOCAL;KrbHostFQDN=trwor-dafb587f.azcloud.local;KrbServiceName=impala";

        try {
            Connection impalaConn = DriverManager.getConnection(impalaConnectionUrl);
            [...]
        }
        catch (SQLEception ex) {
            [...]
        }



---->


java.sql.SQLException: [Simba][ImpalaJDBCDriver](500310) Invalid operation: Unable to connect to server:;
at com.cloudera.impala.hivecommon.api.HiveServer2ClientFactory.createTransport(HiveServer2ClientFactory.java:224)
at com.cloudera.impala.hivecommon.api.HiveServer2ClientFactory.createClient(HiveServer2ClientFactory.java:52)
at com.cloudera.impala.hivecommon.core.HiveJDBCConnection.connect(HiveJDBCConnection.java:597)
at com.cloudera.impala.jdbc.common.BaseConnectionFactory.doConnect(BaseConnectionFactory.java:219)
at com.cloudera.impala.jdbc.common.AbstractDriver.connect(AbstractDriver.java:216)
at java.sql.DriverManager.getConnection(DriverManager.java:571)
at java.sql.DriverManager.getConnection(DriverManager.java:233)
at ico.az.deploy.TestSuite.testTeradata(TestSuite.java:98)
at ico.az.deploy.TestSuite.run(TestSuite.java:311)
Caused by: com.cloudera.impala.support.exceptions.GeneralException: [Simba][ImpalaJDBCDriver](500310) Invalid operation: Unable to connect to server:;
... 9 more
Caused by: java.lang.RuntimeException: Unable to connect to server:
at com.cloudera.impala.hivecommon.api.HiveServer2ClientFactory$1.run(HiveServer2ClientFactory.java:150)
at com.cloudera.impala.hivecommon.api.HiveServer2ClientFactory$1.run(HiveServer2ClientFactory.java:141)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:356)
at com.cloudera.impala.hivecommon.api.HiveServer2ClientFactory.createTransport(HiveServer2ClientFactory.java:140)
at com.cloudera.impala.hivecommon.api.HiveServer2ClientFactory.createClient(HiveServer2ClientFactory.java:52)
at com.cloudera.impala.hivecommon.core.HiveJDBCConnection.connect(HiveJDBCConnection.java:597)
at com.cloudera.impala.jdbc.common.BaseConnectionFactory.doConnect(BaseConnectionFactory.java:219)
at com.cloudera.impala.jdbc.common.AbstractDriver.connect(AbstractDriver.java:216)
at java.sql.DriverManager.getConnection(DriverManager.java:571)
at java.sql.DriverManager.getConnection(DriverManager.java:233)
at ico.az.deploy.TestSuite.testTeradata(TestSuite.java:98)
at ico.az.deploy.TestSuite.run(TestSuite.java:311)
at ico.az.deploy.TestSuite.main(TestSuite.java:347)
Caused by: org.apache.thrift.transport.TTransportException
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:178)
at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:258)
at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
at com.cloudera.impala.hivecommon.api.HiveServer2ClientFactory$1.run(HiveServer2ClientFactory.java:146)
... 13 more

 

Regarding connection string parameters:

  • hostname : the host name where is running an Impala daemon, I took this one from Cloudera Manager->Impala->Instances->Impala daemon (there is one deamon running in each worker node, thus I've just choosen the first one).
  • port : taken from Impala Daemon HiveServer2 Port property property in the Impala Configuration.
  • AuthMech : according to the jdbc driver documentation 1 is for Kerberos authentication.
  • KrbRealm : I took this one from the param default_realm in the /etc/krb5.conf file on the edge node, is this correct?
  • KrbHostFQDN : same as Impala daemon hostname, correct?
  • KrbServiceName : should be "impala" the default, and it is also the nameof  Impala Kerberos Principal on the CM, correct?

These are the relevant properties I found on the Cloudera Manager (read only access) for Impala and Kerberos:

 

snp2.png

 

 

I am trying Kerberos authentication because it seems LDAP authentication is disabled for Impala:

 

snp1.png

 

 

 

What am I doing wrong?

 

 

 

5 REPLIES 5

avatar
Expert Contributor

Update:

 

I've tried to switch to ClouderaImpalaJDBC_2.5.43.1063 driver (using JDBC41). With the following connection string (to infer authentication):

 

jdbc:impala://trwor-dafb587f.azcloud.local:21050;AuthMech=1;KrbAuthType=0;KrbHostFQDN=trwor-dafb587f.azcloud.local;KrbServiceName=impala

Now the error shown is the following:

 

java.sql.SQLException: [Simba][ImpalaJDBCDriver](500164) Error initialized or created transport for authentication: [Simba][ImpalaJDBCDriver](500169) Unable to connect to server: [Simba][ImpalaJDBCDriver](500591) Kerberos Authentication failed..
        at com.cloudera.hivecommon.api.HiveServer2ClientFactory.createTransport(Unknown Source)
        at com.cloudera.hivecommon.api.HiveServer2ClientFactory.createClient(Unknown Source)
        at com.cloudera.hivecommon.core.HiveJDBCCommonConnection.establishConnection(Unknown Source)
        at com.cloudera.impala.core.ImpalaJDBCConnection.establishConnection(Unknown Source)
        at com.cloudera.jdbc.core.LoginTimeoutConnection.connect(Unknown Source)
        at com.cloudera.jdbc.common.BaseConnectionFactory.doConnect(Unknown Source)
        at com.cloudera.jdbc.common.AbstractDriver.connect(Unknown Source)
        at java.sql.DriverManager.getConnection(DriverManager.java:571)
        at java.sql.DriverManager.getConnection(DriverManager.java:233)
        at ico.az.deploy.TestSuite.testTeradata(TestSuite.java:101)
        at ico.az.deploy.TestSuite.run(TestSuite.java:314)
Caused by: com.cloudera.support.exceptions.GeneralException: [Simba][ImpalaJDBCDriver](500164) Error initialized or created transport for authentication: [Simba][ImpalaJDBCDriver](500169) Unable to connect to server: [Simba][ImpalaJDBCDriver](500591) Kerberos Authentication failed..
        ... 11 more
Caused by: java.lang.RuntimeException: [Simba][ImpalaJDBCDriver](500169) Unable to connect to server: [Simba][ImpalaJDBCDriver](500591) Kerberos Authentication failed.
        at com.cloudera.hivecommon.api.HiveServerPrivilegedAction.run(Unknown Source)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:356)
        at com.cloudera.hivecommon.api.HiveServer2ClientFactory.createTransport(Unknown Source)
        at com.cloudera.hivecommon.api.HiveServer2ClientFactory.createClient(Unknown Source)
        at com.cloudera.hivecommon.core.HiveJDBCCommonConnection.establishConnection(Unknown Source)
        at com.cloudera.impala.core.ImpalaJDBCConnection.establishConnection(Unknown Source)
        at com.cloudera.jdbc.core.LoginTimeoutConnection.connect(Unknown Source)
        at com.cloudera.jdbc.common.BaseConnectionFactory.doConnect(Unknown Source)
        at com.cloudera.jdbc.common.AbstractDriver.connect(Unknown Source)
        at java.sql.DriverManager.getConnection(DriverManager.java:571)
        at java.sql.DriverManager.getConnection(DriverManager.java:233)
        at ico.az.deploy.TestSuite.testTeradata(TestSuite.java:101)
        at ico.az.deploy.TestSuite.run(TestSuite.java:314)
        at ico.az.deploy.TestSuite.main(TestSuite.java:350)
Caused by: org.apache.thrift.transport.TTransportException
        at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
        at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
        at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:178)
        at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:258)
        at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
        ... 15 more

Please let me know if is there anything else I can try.

 

avatar
Super Guru
Did you kinit before running impala-shell command? Please run "klist" to confirm, or simply kinit again.

You can also turn on Kerberos debugging to see if it is kerberos related error:

KRB5_TRACE=/tmp/impala-kerberos-debug.log impala-shell -k -i trwor-b9a4f2a7.azcloud.local

Then please share the output in the log file /tmp/impala-kerberos-debug.log.

avatar
Expert Contributor

I've solved the jdbc issue by enabling SSL in the connection string:

 

jdbc:impala://trwor-dafb587f.azcloud.local:21050;SSL=1;AuthMech=1;KrbAuthType=0;KrbHostFQDN=trwor-dafb587f.azcloud.local;KrbServiceName=impala

 

Still not luck with impala-shell connection. If I run "klist" I got:

 

Ticket cache: FILE:/tmp/krb5cc_699006375_ASnf44
Default principal: icon0104@AZCLOUD.LOCAL

Valid starting       Expires              Service principal
04/06/2018 08:38:44  04/06/2018 18:38:44  krbtgt/AZCLOUD.LOCAL@AZCLOUD.LOCAL
        renew until 04/13/2018 08:38:44

Thanks for the support

 

avatar
Super Guru
What' the command you used to run impala-shell? Did you have "--ssl" options set for impala-shell command?

avatar
New Contributor

Hello,

We seem to have a similar problem with impala-shell also on CDH 5.13.2 (and now 5.13.3).

We are using Active Directory KDCs with a one-way trust established between a writable Hadoop AD (KRB.DOMAIN.COM) and our main user AD (USERS.DOMAIN.COM).  In addition, our servers are deployed to a third domain (server.domain.com) which does not have an associated KDC.

krb5.conf file:

 

[libdefaults]
default_realm = KRB.DOMAIN.COM
dns_lookup_kdc = true
dns_lookup_realm = false
ticket_lifetime = 86400
renew_lifetime = 604800
forwardable = true
default_tgs_enctypes = aes256-cts aes128-cts rc4-hmac
default_tkt_enctypes = aes256-cts aes128-cts rc4-hmac
permitted_enctypes = aes256-cts aes128-cts rc4-hmac
udp_preference_limit = 1
kdc_timeout = 3000
rdns = false
#default_ccache_name = KEYRING:persistent:%{uid}

[realms]
KRB.DOMAIN.COM = {
kdc = krb.domain.com
admin_server = krb.domain.com
default_domain = krb.domain.com
}
USERS.DOMAIN.COM = {
kdc = users.domain.com
admin_server = users.domain.com
}

[domain_realm]
krb.domain.com = KRB.DOMAIN.COM
.krb.domain.com = KRB.DOMAIN.COM
users.domain.com = USERS.DOMAIN.COM
.users.domain.com = USERS.DOMAIN.COM

 

Below is how we invoke impala-shell and the tracing statements I'm seeing after manipulating the KRB5_TRACE env variable:

 

[user1@edgenode ~]$ \impala-shell -k --ssl -i daemonnode.server.domain.com:21000
Starting Impala Shell using Kerberos authentication
Using service name 'impala'
SSL is enabled. Impala server certificates will NOT be verified (set --ca_cert to change)
[22712] 1524768162.661368: ccselect can't find appropriate cache for server principal impala/daemonnode.server.domain.com@
[22712] 1524768162.661450: Getting credentials user1@USERS.DOMAIN.COM -> impala/daemonnode.server.domain.com@ using ccache FILE:/tmp/krb5cc_738475
[22712] 1524768162.661513: Retrieving user1@USERS.DOMAIN.COM -> impala/daemonnode.server.domain.com@ from FILE:/tmp/krb5cc_738475 with result: -1765328243/Matching credential not found (filename: /tmp/krb5cc_738475)
[22712] 1524768162.661558: Retrying user1@USERS.DOMAIN.COM -> impala/daemonnode.server.domain.com@USERS.DOMAIN.COM with result: -1765328243/Matching credential not found (filename: /tmp/krb5cc_738475)
[22712] 1524768162.661563: Server has referral realm; starting with impala/daemonnode.server.domain.com@USERS.DOMAIN.COM
[22712] 1524768162.661627: Retrieving user1@USERS.DOMAIN.COM -> krbtgt/USERS.DOMAIN.COM@USERS.DOMAIN.COM from FILE:/tmp/krb5cc_738475 with result: 0/Success
[22712] 1524768162.661633: Starting with TGT for client realm: user1@USERS.DOMAIN.COM -> krbtgt/USERS.DOMAIN.COM@USERS.DOMAIN.COM
[22712] 1524768162.661640: Requesting tickets for impala/daemonnode.server.domain.com@USERS.DOMAIN.COM, referrals on
[22712] 1524768162.661662: Generated subkey for TGS request: aes256-cts/56A9
[22712] 1524768162.661696: etypes requested in TGS request: aes256-cts, aes128-cts, rc4-hmac
[22712] 1524768162.661790: Encoding request body and padata into FAST request
[22712] 1524768162.661917: Sending request (9771 bytes) to USERS.DOMAIN.COM
[22712] 1524768162.662009: Resolving hostname users.domain.com
[22712] 1524768162.701885: Initiating TCP connection to stream XXX.XXX.XXX.XXX:88
[22712] 1524768162.739233: Sending TCP request to stream XXX.XXX.XXX.XXX:88
[22712] 1524768162.836656: Received answer (351 bytes) from stream XXX.XXX.XXX.XXX:88
[22712] 1524768162.836673: Terminating TCP connection to stream XXX.XXX.XXX.XXX:88
[22712] 1524768164.121113: Response was not from master KDC
[22712] 1524768164.121153: Decoding FAST response
[22712] 1524768164.121209: TGS request result: -1765328377/Server not found in Kerberos database
[22712] 1524768164.121230: Local realm referral failed; trying fallback realm SERVER.DOMAIN.COM
[22712] 1524768164.121313: Retrieving user1@USERS.DOMAIN.COM -> krbtgt/SERVER.DOMAIN.COM@SERVER.DOMAIN.COM from FILE:/tmp/krb5cc_738475 with result: -1765328243/Matching credential not found (filename: /tmp/krb5cc_738475)
[22712] 1524768164.121366: Retrieving user1@USERS.DOMAIN.COM -> krbtgt/USERS.DOMAIN.COM@USERS.DOMAIN.COM from FILE:/tmp/krb5cc_738475 with result: 0/Success
[22712] 1524768164.121372: Starting with TGT for client realm: user1@USERS.DOMAIN.COM -> krbtgt/USERS.DOMAIN.COM@USERS.DOMAIN.COM
[22712] 1524768164.121417: Retrieving user1@USERS.DOMAIN.COM -> krbtgt/SERVER.DOMAIN.COM@SERVER.DOMAIN.COM from FILE:/tmp/krb5cc_738475 with result: -1765328243/Matching credential not found (filename: /tmp/krb5cc_738475)
[22712] 1524768164.121422: Requesting TGT krbtgt/SERVER.DOMAIN.COM@USERS.DOMAIN.COM using TGT krbtgt/USERS.DOMAIN.COM@USERS.DOMAIN.COM
[22712] 1524768164.121434: Generated subkey for TGS request: aes256-cts/31FF
[22712] 1524768164.121454: etypes requested in TGS request: aes256-cts, aes128-cts, rc4-hmac
[22712] 1524768164.121530: Encoding request body and padata into FAST request
[22712] 1524768164.121623: Sending request (9748 bytes) to USERS.DOMAIN.COM
[22712] 1524768164.121649: Resolving hostname users.domain.com
[22712] 1524768164.122515: Initiating TCP connection to stream YYY.YYY.YYY.YYY:88
[22712] 1524768164.126676: Sending TCP request to stream YYY.YYY.YYY.YYY:88
[22712] 1524768164.135666: Received answer (329 bytes) from stream YYY.YYY.YYY.YYY:88
[22712] 1524768164.135673: Terminating TCP connection to stream YYY.YYY.YYY.YYY:88
[22712] 1524768164.137693: Response was not from master KDC
[22712] 1524768164.137706: Decoding FAST response
[22712] 1524768164.137731: TGS request result: -1765328377/Server not found in Kerberos database
Error connecting: TTransportException, Could not start SASL: Error in sasl_client_start (-1) SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Server not found in Kerberos database)
***********************************************************************************
Welcome to the Impala shell.
(Impala Shell v2.10.0-cdh5.13.3 (15a453e) built on Sat Mar 17 03:48:31 PDT 2018)

Want to know what version of Impala you're connected to? Run the VERSION command to
find out!
***********************************************************************************
[Not connected] > exit;
Goodbye user1

I think the issue may be in what service principal impala-shell is requesting a ticket for.  It should be asking for impala/daemonnode.server.domain.com@KRB.DOMAIN.COM but I'm not sure if it ever does.

When I obtain the keytab for that service principal and kinit against it, impala-shell connects.

Additionally, I've been able to authenticate via Kerberos using Tableau (and the Cloudera ODBC driver), which grants me finer-grain control over what specific service principal needs to be authenticated.