Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here. Want to know more about what has changed? Check out the Community News blog.

Help connecting to Impala through impala-shell and jdbc in Kerberos/LDAP cluster

Help connecting to Impala through impala-shell and jdbc in Kerberos/LDAP cluster

Expert Contributor

Hello everybody

 

I am working on a CDH 5.13.2 cluster configured with Kerberos and LDAP authentication.

 

I need to connect to Impala thorugh jdbc and impala-shell, but I am having problems on both (Impala queries on HUE work fine).

 

For impala-shell I've tried:

 

 

impala-shell -k -i trwor-b9a4f2a7.azcloud.local

--->

Starting Impala Shell using Kerberos authentication
Using service name 'impala'
Error connecting: TTransportException, TSocket read 0 bytes
***********************************************************************************
Welcome to the Impala shell.
(Impala Shell v2.10.0-cdh5.13.2 (dc867db) built on Fri Feb 2 10:46:38 PST 2018)

 

I've also tried without Kerberos:

 

impala-shell -i trwor-b9a4f2a7.azcloud.local

--->


Starting Impala Shell without Kerberos authentication
Error connecting: TTransportException, TSocket read 0 bytes
Kerberos ticket found in the credentials cache, retrying the connection with a secure transport.
Error connecting: TTransportException, TSocket read 0 bytes
***********************************************************************************
Welcome to the Impala shell.
(Impala Shell v2.10.0-cdh5.13.2 (dc867db) built on Fri Feb 2 10:46:38 PST 2018)

 

In both cases I got a TTransportException.

 

 

I am having trouble also for connecting to Impala through jdbc (using Cloudera_ImpalaJDBC4_2.5.5.1007 driver):

 

 

String impalaConnectionUrl = "jdbc:impala://trwor-dafb587f.azcloud.local:21050;AuthMech=1;KrbRealm=AZCLOUD.LOCAL;KrbHostFQDN=trwor-dafb587f.azcloud.local;KrbServiceName=impala";

        try {
            Connection impalaConn = DriverManager.getConnection(impalaConnectionUrl);
            [...]
        }
        catch (SQLEception ex) {
            [...]
        }



---->


java.sql.SQLException: [Simba][ImpalaJDBCDriver](500310) Invalid operation: Unable to connect to server:;
at com.cloudera.impala.hivecommon.api.HiveServer2ClientFactory.createTransport(HiveServer2ClientFactory.java:224)
at com.cloudera.impala.hivecommon.api.HiveServer2ClientFactory.createClient(HiveServer2ClientFactory.java:52)
at com.cloudera.impala.hivecommon.core.HiveJDBCConnection.connect(HiveJDBCConnection.java:597)
at com.cloudera.impala.jdbc.common.BaseConnectionFactory.doConnect(BaseConnectionFactory.java:219)
at com.cloudera.impala.jdbc.common.AbstractDriver.connect(AbstractDriver.java:216)
at java.sql.DriverManager.getConnection(DriverManager.java:571)
at java.sql.DriverManager.getConnection(DriverManager.java:233)
at ico.az.deploy.TestSuite.testTeradata(TestSuite.java:98)
at ico.az.deploy.TestSuite.run(TestSuite.java:311)
Caused by: com.cloudera.impala.support.exceptions.GeneralException: [Simba][ImpalaJDBCDriver](500310) Invalid operation: Unable to connect to server:;
... 9 more
Caused by: java.lang.RuntimeException: Unable to connect to server:
at com.cloudera.impala.hivecommon.api.HiveServer2ClientFactory$1.run(HiveServer2ClientFactory.java:150)
at com.cloudera.impala.hivecommon.api.HiveServer2ClientFactory$1.run(HiveServer2ClientFactory.java:141)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:356)
at com.cloudera.impala.hivecommon.api.HiveServer2ClientFactory.createTransport(HiveServer2ClientFactory.java:140)
at com.cloudera.impala.hivecommon.api.HiveServer2ClientFactory.createClient(HiveServer2ClientFactory.java:52)
at com.cloudera.impala.hivecommon.core.HiveJDBCConnection.connect(HiveJDBCConnection.java:597)
at com.cloudera.impala.jdbc.common.BaseConnectionFactory.doConnect(BaseConnectionFactory.java:219)
at com.cloudera.impala.jdbc.common.AbstractDriver.connect(AbstractDriver.java:216)
at java.sql.DriverManager.getConnection(DriverManager.java:571)
at java.sql.DriverManager.getConnection(DriverManager.java:233)
at ico.az.deploy.TestSuite.testTeradata(TestSuite.java:98)
at ico.az.deploy.TestSuite.run(TestSuite.java:311)
at ico.az.deploy.TestSuite.main(TestSuite.java:347)
Caused by: org.apache.thrift.transport.TTransportException
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:178)
at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:258)
at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
at com.cloudera.impala.hivecommon.api.HiveServer2ClientFactory$1.run(HiveServer2ClientFactory.java:146)
... 13 more

 

Regarding connection string parameters:

  • hostname : the host name where is running an Impala daemon, I took this one from Cloudera Manager->Impala->Instances->Impala daemon (there is one deamon running in each worker node, thus I've just choosen the first one).
  • port : taken from Impala Daemon HiveServer2 Port property property in the Impala Configuration.
  • AuthMech : according to the jdbc driver documentation 1 is for Kerberos authentication.
  • KrbRealm : I took this one from the param default_realm in the /etc/krb5.conf file on the edge node, is this correct?
  • KrbHostFQDN : same as Impala daemon hostname, correct?
  • KrbServiceName : should be "impala" the default, and it is also the nameof  Impala Kerberos Principal on the CM, correct?

These are the relevant properties I found on the Cloudera Manager (read only access) for Impala and Kerberos:

 

snp2.png

 

 

I am trying Kerberos authentication because it seems LDAP authentication is disabled for Impala:

 

snp1.png

 

 

 

What am I doing wrong?

 

 

 

5 REPLIES 5

Re: Help connecting to Impala through impala-shell and jdbc in Kerberos/LDAP cluster

Expert Contributor

Update:

 

I've tried to switch to ClouderaImpalaJDBC_2.5.43.1063 driver (using JDBC41). With the following connection string (to infer authentication):

 

jdbc:impala://trwor-dafb587f.azcloud.local:21050;AuthMech=1;KrbAuthType=0;KrbHostFQDN=trwor-dafb587f.azcloud.local;KrbServiceName=impala

Now the error shown is the following:

 

java.sql.SQLException: [Simba][ImpalaJDBCDriver](500164) Error initialized or created transport for authentication: [Simba][ImpalaJDBCDriver](500169) Unable to connect to server: [Simba][ImpalaJDBCDriver](500591) Kerberos Authentication failed..
        at com.cloudera.hivecommon.api.HiveServer2ClientFactory.createTransport(Unknown Source)
        at com.cloudera.hivecommon.api.HiveServer2ClientFactory.createClient(Unknown Source)
        at com.cloudera.hivecommon.core.HiveJDBCCommonConnection.establishConnection(Unknown Source)
        at com.cloudera.impala.core.ImpalaJDBCConnection.establishConnection(Unknown Source)
        at com.cloudera.jdbc.core.LoginTimeoutConnection.connect(Unknown Source)
        at com.cloudera.jdbc.common.BaseConnectionFactory.doConnect(Unknown Source)
        at com.cloudera.jdbc.common.AbstractDriver.connect(Unknown Source)
        at java.sql.DriverManager.getConnection(DriverManager.java:571)
        at java.sql.DriverManager.getConnection(DriverManager.java:233)
        at ico.az.deploy.TestSuite.testTeradata(TestSuite.java:101)
        at ico.az.deploy.TestSuite.run(TestSuite.java:314)
Caused by: com.cloudera.support.exceptions.GeneralException: [Simba][ImpalaJDBCDriver](500164) Error initialized or created transport for authentication: [Simba][ImpalaJDBCDriver](500169) Unable to connect to server: [Simba][ImpalaJDBCDriver](500591) Kerberos Authentication failed..
        ... 11 more
Caused by: java.lang.RuntimeException: [Simba][ImpalaJDBCDriver](500169) Unable to connect to server: [Simba][ImpalaJDBCDriver](500591) Kerberos Authentication failed.
        at com.cloudera.hivecommon.api.HiveServerPrivilegedAction.run(Unknown Source)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:356)
        at com.cloudera.hivecommon.api.HiveServer2ClientFactory.createTransport(Unknown Source)
        at com.cloudera.hivecommon.api.HiveServer2ClientFactory.createClient(Unknown Source)
        at com.cloudera.hivecommon.core.HiveJDBCCommonConnection.establishConnection(Unknown Source)
        at com.cloudera.impala.core.ImpalaJDBCConnection.establishConnection(Unknown Source)
        at com.cloudera.jdbc.core.LoginTimeoutConnection.connect(Unknown Source)
        at com.cloudera.jdbc.common.BaseConnectionFactory.doConnect(Unknown Source)
        at com.cloudera.jdbc.common.AbstractDriver.connect(Unknown Source)
        at java.sql.DriverManager.getConnection(DriverManager.java:571)
        at java.sql.DriverManager.getConnection(DriverManager.java:233)
        at ico.az.deploy.TestSuite.testTeradata(TestSuite.java:101)
        at ico.az.deploy.TestSuite.run(TestSuite.java:314)
        at ico.az.deploy.TestSuite.main(TestSuite.java:350)
Caused by: org.apache.thrift.transport.TTransportException
        at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
        at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
        at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:178)
        at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:258)
        at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
        ... 15 more

Please let me know if is there anything else I can try.

 

Re: Help connecting to Impala through impala-shell and jdbc in Kerberos/LDAP cluster

Guru
Did you kinit before running impala-shell command? Please run "klist" to confirm, or simply kinit again.

You can also turn on Kerberos debugging to see if it is kerberos related error:

KRB5_TRACE=/tmp/impala-kerberos-debug.log impala-shell -k -i trwor-b9a4f2a7.azcloud.local

Then please share the output in the log file /tmp/impala-kerberos-debug.log.

Re: Help connecting to Impala through impala-shell and jdbc in Kerberos/LDAP cluster

Expert Contributor

I've solved the jdbc issue by enabling SSL in the connection string:

 

jdbc:impala://trwor-dafb587f.azcloud.local:21050;SSL=1;AuthMech=1;KrbAuthType=0;KrbHostFQDN=trwor-dafb587f.azcloud.local;KrbServiceName=impala

 

Still not luck with impala-shell connection. If I run "klist" I got:

 

Ticket cache: FILE:/tmp/krb5cc_699006375_ASnf44
Default principal: icon0104@AZCLOUD.LOCAL

Valid starting       Expires              Service principal
04/06/2018 08:38:44  04/06/2018 18:38:44  krbtgt/AZCLOUD.LOCAL@AZCLOUD.LOCAL
        renew until 04/13/2018 08:38:44

Thanks for the support

 

Re: Help connecting to Impala through impala-shell and jdbc in Kerberos/LDAP cluster

Guru
What' the command you used to run impala-shell? Did you have "--ssl" options set for impala-shell command?

Re: Help connecting to Impala through impala-shell and jdbc in Kerberos/LDAP cluster

New Contributor

Hello,

We seem to have a similar problem with impala-shell also on CDH 5.13.2 (and now 5.13.3).

We are using Active Directory KDCs with a one-way trust established between a writable Hadoop AD (KRB.DOMAIN.COM) and our main user AD (USERS.DOMAIN.COM).  In addition, our servers are deployed to a third domain (server.domain.com) which does not have an associated KDC.

krb5.conf file:

 

[libdefaults]
default_realm = KRB.DOMAIN.COM
dns_lookup_kdc = true
dns_lookup_realm = false
ticket_lifetime = 86400
renew_lifetime = 604800
forwardable = true
default_tgs_enctypes = aes256-cts aes128-cts rc4-hmac
default_tkt_enctypes = aes256-cts aes128-cts rc4-hmac
permitted_enctypes = aes256-cts aes128-cts rc4-hmac
udp_preference_limit = 1
kdc_timeout = 3000
rdns = false
#default_ccache_name = KEYRING:persistent:%{uid}

[realms]
KRB.DOMAIN.COM = {
kdc = krb.domain.com
admin_server = krb.domain.com
default_domain = krb.domain.com
}
USERS.DOMAIN.COM = {
kdc = users.domain.com
admin_server = users.domain.com
}

[domain_realm]
krb.domain.com = KRB.DOMAIN.COM
.krb.domain.com = KRB.DOMAIN.COM
users.domain.com = USERS.DOMAIN.COM
.users.domain.com = USERS.DOMAIN.COM

 

Below is how we invoke impala-shell and the tracing statements I'm seeing after manipulating the KRB5_TRACE env variable:

 

[user1@edgenode ~]$ \impala-shell -k --ssl -i daemonnode.server.domain.com:21000
Starting Impala Shell using Kerberos authentication
Using service name 'impala'
SSL is enabled. Impala server certificates will NOT be verified (set --ca_cert to change)
[22712] 1524768162.661368: ccselect can't find appropriate cache for server principal impala/daemonnode.server.domain.com@
[22712] 1524768162.661450: Getting credentials user1@USERS.DOMAIN.COM -> impala/daemonnode.server.domain.com@ using ccache FILE:/tmp/krb5cc_738475
[22712] 1524768162.661513: Retrieving user1@USERS.DOMAIN.COM -> impala/daemonnode.server.domain.com@ from FILE:/tmp/krb5cc_738475 with result: -1765328243/Matching credential not found (filename: /tmp/krb5cc_738475)
[22712] 1524768162.661558: Retrying user1@USERS.DOMAIN.COM -> impala/daemonnode.server.domain.com@USERS.DOMAIN.COM with result: -1765328243/Matching credential not found (filename: /tmp/krb5cc_738475)
[22712] 1524768162.661563: Server has referral realm; starting with impala/daemonnode.server.domain.com@USERS.DOMAIN.COM
[22712] 1524768162.661627: Retrieving user1@USERS.DOMAIN.COM -> krbtgt/USERS.DOMAIN.COM@USERS.DOMAIN.COM from FILE:/tmp/krb5cc_738475 with result: 0/Success
[22712] 1524768162.661633: Starting with TGT for client realm: user1@USERS.DOMAIN.COM -> krbtgt/USERS.DOMAIN.COM@USERS.DOMAIN.COM
[22712] 1524768162.661640: Requesting tickets for impala/daemonnode.server.domain.com@USERS.DOMAIN.COM, referrals on
[22712] 1524768162.661662: Generated subkey for TGS request: aes256-cts/56A9
[22712] 1524768162.661696: etypes requested in TGS request: aes256-cts, aes128-cts, rc4-hmac
[22712] 1524768162.661790: Encoding request body and padata into FAST request
[22712] 1524768162.661917: Sending request (9771 bytes) to USERS.DOMAIN.COM
[22712] 1524768162.662009: Resolving hostname users.domain.com
[22712] 1524768162.701885: Initiating TCP connection to stream XXX.XXX.XXX.XXX:88
[22712] 1524768162.739233: Sending TCP request to stream XXX.XXX.XXX.XXX:88
[22712] 1524768162.836656: Received answer (351 bytes) from stream XXX.XXX.XXX.XXX:88
[22712] 1524768162.836673: Terminating TCP connection to stream XXX.XXX.XXX.XXX:88
[22712] 1524768164.121113: Response was not from master KDC
[22712] 1524768164.121153: Decoding FAST response
[22712] 1524768164.121209: TGS request result: -1765328377/Server not found in Kerberos database
[22712] 1524768164.121230: Local realm referral failed; trying fallback realm SERVER.DOMAIN.COM
[22712] 1524768164.121313: Retrieving user1@USERS.DOMAIN.COM -> krbtgt/SERVER.DOMAIN.COM@SERVER.DOMAIN.COM from FILE:/tmp/krb5cc_738475 with result: -1765328243/Matching credential not found (filename: /tmp/krb5cc_738475)
[22712] 1524768164.121366: Retrieving user1@USERS.DOMAIN.COM -> krbtgt/USERS.DOMAIN.COM@USERS.DOMAIN.COM from FILE:/tmp/krb5cc_738475 with result: 0/Success
[22712] 1524768164.121372: Starting with TGT for client realm: user1@USERS.DOMAIN.COM -> krbtgt/USERS.DOMAIN.COM@USERS.DOMAIN.COM
[22712] 1524768164.121417: Retrieving user1@USERS.DOMAIN.COM -> krbtgt/SERVER.DOMAIN.COM@SERVER.DOMAIN.COM from FILE:/tmp/krb5cc_738475 with result: -1765328243/Matching credential not found (filename: /tmp/krb5cc_738475)
[22712] 1524768164.121422: Requesting TGT krbtgt/SERVER.DOMAIN.COM@USERS.DOMAIN.COM using TGT krbtgt/USERS.DOMAIN.COM@USERS.DOMAIN.COM
[22712] 1524768164.121434: Generated subkey for TGS request: aes256-cts/31FF
[22712] 1524768164.121454: etypes requested in TGS request: aes256-cts, aes128-cts, rc4-hmac
[22712] 1524768164.121530: Encoding request body and padata into FAST request
[22712] 1524768164.121623: Sending request (9748 bytes) to USERS.DOMAIN.COM
[22712] 1524768164.121649: Resolving hostname users.domain.com
[22712] 1524768164.122515: Initiating TCP connection to stream YYY.YYY.YYY.YYY:88
[22712] 1524768164.126676: Sending TCP request to stream YYY.YYY.YYY.YYY:88
[22712] 1524768164.135666: Received answer (329 bytes) from stream YYY.YYY.YYY.YYY:88
[22712] 1524768164.135673: Terminating TCP connection to stream YYY.YYY.YYY.YYY:88
[22712] 1524768164.137693: Response was not from master KDC
[22712] 1524768164.137706: Decoding FAST response
[22712] 1524768164.137731: TGS request result: -1765328377/Server not found in Kerberos database
Error connecting: TTransportException, Could not start SASL: Error in sasl_client_start (-1) SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Server not found in Kerberos database)
***********************************************************************************
Welcome to the Impala shell.
(Impala Shell v2.10.0-cdh5.13.3 (15a453e) built on Sat Mar 17 03:48:31 PDT 2018)

Want to know what version of Impala you're connected to? Run the VERSION command to
find out!
***********************************************************************************
[Not connected] > exit;
Goodbye user1

I think the issue may be in what service principal impala-shell is requesting a ticket for.  It should be asking for impala/daemonnode.server.domain.com@KRB.DOMAIN.COM but I'm not sure if it ever does.

When I obtain the keytab for that service principal and kinit against it, impala-shell connects.

Additionally, I've been able to authenticate via Kerberos using Tableau (and the Cloudera ODBC driver), which grants me finer-grain control over what specific service principal needs to be authenticated.