Created on 04-02-2018 11:15 AM - edited 09-16-2022 06:03 AM
Hello everybody
I am working on a CDH 5.13.2 cluster configured with Kerberos and LDAP authentication.
I need to connect to Impala thorugh jdbc and impala-shell, but I am having problems on both (Impala queries on HUE work fine).
For impala-shell I've tried:
impala-shell -k -i trwor-b9a4f2a7.azcloud.local
--->
Starting Impala Shell using Kerberos authentication
Using service name 'impala'
Error connecting: TTransportException, TSocket read 0 bytes
***********************************************************************************
Welcome to the Impala shell.
(Impala Shell v2.10.0-cdh5.13.2 (dc867db) built on Fri Feb 2 10:46:38 PST 2018)
I've also tried without Kerberos:
impala-shell -i trwor-b9a4f2a7.azcloud.local ---> Starting Impala Shell without Kerberos authentication Error connecting: TTransportException, TSocket read 0 bytes Kerberos ticket found in the credentials cache, retrying the connection with a secure transport. Error connecting: TTransportException, TSocket read 0 bytes ***********************************************************************************
Welcome to the Impala shell.
(Impala Shell v2.10.0-cdh5.13.2 (dc867db) built on Fri Feb 2 10:46:38 PST 2018)
In both cases I got a TTransportException.
I am having trouble also for connecting to Impala through jdbc (using Cloudera_ImpalaJDBC4_2.5.5.1007 driver):
String impalaConnectionUrl = "jdbc:impala://trwor-dafb587f.azcloud.local:21050;AuthMech=1;KrbRealm=AZCLOUD.LOCAL;KrbHostFQDN=trwor-dafb587f.azcloud.local;KrbServiceName=impala"; try { Connection impalaConn = DriverManager.getConnection(impalaConnectionUrl); [...] } catch (SQLEception ex) { [...] }
---->
java.sql.SQLException: [Simba][ImpalaJDBCDriver](500310) Invalid operation: Unable to connect to server:;
at com.cloudera.impala.hivecommon.api.HiveServer2ClientFactory.createTransport(HiveServer2ClientFactory.java:224)
at com.cloudera.impala.hivecommon.api.HiveServer2ClientFactory.createClient(HiveServer2ClientFactory.java:52)
at com.cloudera.impala.hivecommon.core.HiveJDBCConnection.connect(HiveJDBCConnection.java:597)
at com.cloudera.impala.jdbc.common.BaseConnectionFactory.doConnect(BaseConnectionFactory.java:219)
at com.cloudera.impala.jdbc.common.AbstractDriver.connect(AbstractDriver.java:216)
at java.sql.DriverManager.getConnection(DriverManager.java:571)
at java.sql.DriverManager.getConnection(DriverManager.java:233)
at ico.az.deploy.TestSuite.testTeradata(TestSuite.java:98)
at ico.az.deploy.TestSuite.run(TestSuite.java:311)
Caused by: com.cloudera.impala.support.exceptions.GeneralException: [Simba][ImpalaJDBCDriver](500310) Invalid operation: Unable to connect to server:;
... 9 more
Caused by: java.lang.RuntimeException: Unable to connect to server:
at com.cloudera.impala.hivecommon.api.HiveServer2ClientFactory$1.run(HiveServer2ClientFactory.java:150)
at com.cloudera.impala.hivecommon.api.HiveServer2ClientFactory$1.run(HiveServer2ClientFactory.java:141)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:356)
at com.cloudera.impala.hivecommon.api.HiveServer2ClientFactory.createTransport(HiveServer2ClientFactory.java:140)
at com.cloudera.impala.hivecommon.api.HiveServer2ClientFactory.createClient(HiveServer2ClientFactory.java:52)
at com.cloudera.impala.hivecommon.core.HiveJDBCConnection.connect(HiveJDBCConnection.java:597)
at com.cloudera.impala.jdbc.common.BaseConnectionFactory.doConnect(BaseConnectionFactory.java:219)
at com.cloudera.impala.jdbc.common.AbstractDriver.connect(AbstractDriver.java:216)
at java.sql.DriverManager.getConnection(DriverManager.java:571)
at java.sql.DriverManager.getConnection(DriverManager.java:233)
at ico.az.deploy.TestSuite.testTeradata(TestSuite.java:98)
at ico.az.deploy.TestSuite.run(TestSuite.java:311)
at ico.az.deploy.TestSuite.main(TestSuite.java:347)
Caused by: org.apache.thrift.transport.TTransportException
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:178)
at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:258)
at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
at com.cloudera.impala.hivecommon.api.HiveServer2ClientFactory$1.run(HiveServer2ClientFactory.java:146)
... 13 more
Regarding connection string parameters:
These are the relevant properties I found on the Cloudera Manager (read only access) for Impala and Kerberos:
I am trying Kerberos authentication because it seems LDAP authentication is disabled for Impala:
What am I doing wrong?
Created 04-02-2018 12:41 PM
Update:
I've tried to switch to ClouderaImpalaJDBC_2.5.43.1063 driver (using JDBC41). With the following connection string (to infer authentication):
jdbc:impala://trwor-dafb587f.azcloud.local:21050;AuthMech=1;KrbAuthType=0;KrbHostFQDN=trwor-dafb587f.azcloud.local;KrbServiceName=impala
Now the error shown is the following:
java.sql.SQLException: [Simba][ImpalaJDBCDriver](500164) Error initialized or created transport for authentication: [Simba][ImpalaJDBCDriver](500169) Unable to connect to server: [Simba][ImpalaJDBCDriver](500591) Kerberos Authentication failed.. at com.cloudera.hivecommon.api.HiveServer2ClientFactory.createTransport(Unknown Source) at com.cloudera.hivecommon.api.HiveServer2ClientFactory.createClient(Unknown Source) at com.cloudera.hivecommon.core.HiveJDBCCommonConnection.establishConnection(Unknown Source) at com.cloudera.impala.core.ImpalaJDBCConnection.establishConnection(Unknown Source) at com.cloudera.jdbc.core.LoginTimeoutConnection.connect(Unknown Source) at com.cloudera.jdbc.common.BaseConnectionFactory.doConnect(Unknown Source) at com.cloudera.jdbc.common.AbstractDriver.connect(Unknown Source) at java.sql.DriverManager.getConnection(DriverManager.java:571) at java.sql.DriverManager.getConnection(DriverManager.java:233) at ico.az.deploy.TestSuite.testTeradata(TestSuite.java:101) at ico.az.deploy.TestSuite.run(TestSuite.java:314) Caused by: com.cloudera.support.exceptions.GeneralException: [Simba][ImpalaJDBCDriver](500164) Error initialized or created transport for authentication: [Simba][ImpalaJDBCDriver](500169) Unable to connect to server: [Simba][ImpalaJDBCDriver](500591) Kerberos Authentication failed.. ... 11 more Caused by: java.lang.RuntimeException: [Simba][ImpalaJDBCDriver](500169) Unable to connect to server: [Simba][ImpalaJDBCDriver](500591) Kerberos Authentication failed. at com.cloudera.hivecommon.api.HiveServerPrivilegedAction.run(Unknown Source) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:356) at com.cloudera.hivecommon.api.HiveServer2ClientFactory.createTransport(Unknown Source) at com.cloudera.hivecommon.api.HiveServer2ClientFactory.createClient(Unknown Source) at com.cloudera.hivecommon.core.HiveJDBCCommonConnection.establishConnection(Unknown Source) at com.cloudera.impala.core.ImpalaJDBCConnection.establishConnection(Unknown Source) at com.cloudera.jdbc.core.LoginTimeoutConnection.connect(Unknown Source) at com.cloudera.jdbc.common.BaseConnectionFactory.doConnect(Unknown Source) at com.cloudera.jdbc.common.AbstractDriver.connect(Unknown Source) at java.sql.DriverManager.getConnection(DriverManager.java:571) at java.sql.DriverManager.getConnection(DriverManager.java:233) at ico.az.deploy.TestSuite.testTeradata(TestSuite.java:101) at ico.az.deploy.TestSuite.run(TestSuite.java:314) at ico.az.deploy.TestSuite.main(TestSuite.java:350) Caused by: org.apache.thrift.transport.TTransportException at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:178) at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:258) at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37) ... 15 more
Please let me know if is there anything else I can try.
Created 04-04-2018 09:26 PM
Created 04-06-2018 01:43 AM
I've solved the jdbc issue by enabling SSL in the connection string:
jdbc:impala://trwor-dafb587f.azcloud.local:21050;SSL=1;AuthMech=1;KrbAuthType=0;KrbHostFQDN=trwor-dafb587f.azcloud.local;KrbServiceName=impala
Still not luck with impala-shell connection. If I run "klist" I got:
Ticket cache: FILE:/tmp/krb5cc_699006375_ASnf44 Default principal: icon0104@AZCLOUD.LOCAL Valid starting Expires Service principal 04/06/2018 08:38:44 04/06/2018 18:38:44 krbtgt/AZCLOUD.LOCAL@AZCLOUD.LOCAL renew until 04/13/2018 08:38:44
Thanks for the support
Created 04-08-2018 05:11 PM
Created 04-26-2018 12:54 PM
Hello,
We seem to have a similar problem with impala-shell also on CDH 5.13.2 (and now 5.13.3).
We are using Active Directory KDCs with a one-way trust established between a writable Hadoop AD (KRB.DOMAIN.COM) and our main user AD (USERS.DOMAIN.COM). In addition, our servers are deployed to a third domain (server.domain.com) which does not have an associated KDC.
krb5.conf file:
[libdefaults] default_realm = KRB.DOMAIN.COM dns_lookup_kdc = true dns_lookup_realm = false ticket_lifetime = 86400 renew_lifetime = 604800 forwardable = true default_tgs_enctypes = aes256-cts aes128-cts rc4-hmac default_tkt_enctypes = aes256-cts aes128-cts rc4-hmac permitted_enctypes = aes256-cts aes128-cts rc4-hmac udp_preference_limit = 1 kdc_timeout = 3000 rdns = false #default_ccache_name = KEYRING:persistent:%{uid} [realms] KRB.DOMAIN.COM = { kdc = krb.domain.com admin_server = krb.domain.com default_domain = krb.domain.com } USERS.DOMAIN.COM = { kdc = users.domain.com admin_server = users.domain.com } [domain_realm] krb.domain.com = KRB.DOMAIN.COM .krb.domain.com = KRB.DOMAIN.COM users.domain.com = USERS.DOMAIN.COM .users.domain.com = USERS.DOMAIN.COM
Below is how we invoke impala-shell and the tracing statements I'm seeing after manipulating the KRB5_TRACE env variable:
[user1@edgenode ~]$ \impala-shell -k --ssl -i daemonnode.server.domain.com:21000 Starting Impala Shell using Kerberos authentication Using service name 'impala' SSL is enabled. Impala server certificates will NOT be verified (set --ca_cert to change) [22712] 1524768162.661368: ccselect can't find appropriate cache for server principal impala/daemonnode.server.domain.com@ [22712] 1524768162.661450: Getting credentials user1@USERS.DOMAIN.COM -> impala/daemonnode.server.domain.com@ using ccache FILE:/tmp/krb5cc_738475 [22712] 1524768162.661513: Retrieving user1@USERS.DOMAIN.COM -> impala/daemonnode.server.domain.com@ from FILE:/tmp/krb5cc_738475 with result: -1765328243/Matching credential not found (filename: /tmp/krb5cc_738475) [22712] 1524768162.661558: Retrying user1@USERS.DOMAIN.COM -> impala/daemonnode.server.domain.com@USERS.DOMAIN.COM with result: -1765328243/Matching credential not found (filename: /tmp/krb5cc_738475) [22712] 1524768162.661563: Server has referral realm; starting with impala/daemonnode.server.domain.com@USERS.DOMAIN.COM [22712] 1524768162.661627: Retrieving user1@USERS.DOMAIN.COM -> krbtgt/USERS.DOMAIN.COM@USERS.DOMAIN.COM from FILE:/tmp/krb5cc_738475 with result: 0/Success [22712] 1524768162.661633: Starting with TGT for client realm: user1@USERS.DOMAIN.COM -> krbtgt/USERS.DOMAIN.COM@USERS.DOMAIN.COM [22712] 1524768162.661640: Requesting tickets for impala/daemonnode.server.domain.com@USERS.DOMAIN.COM, referrals on [22712] 1524768162.661662: Generated subkey for TGS request: aes256-cts/56A9 [22712] 1524768162.661696: etypes requested in TGS request: aes256-cts, aes128-cts, rc4-hmac [22712] 1524768162.661790: Encoding request body and padata into FAST request [22712] 1524768162.661917: Sending request (9771 bytes) to USERS.DOMAIN.COM [22712] 1524768162.662009: Resolving hostname users.domain.com [22712] 1524768162.701885: Initiating TCP connection to stream XXX.XXX.XXX.XXX:88 [22712] 1524768162.739233: Sending TCP request to stream XXX.XXX.XXX.XXX:88 [22712] 1524768162.836656: Received answer (351 bytes) from stream XXX.XXX.XXX.XXX:88 [22712] 1524768162.836673: Terminating TCP connection to stream XXX.XXX.XXX.XXX:88 [22712] 1524768164.121113: Response was not from master KDC [22712] 1524768164.121153: Decoding FAST response [22712] 1524768164.121209: TGS request result: -1765328377/Server not found in Kerberos database [22712] 1524768164.121230: Local realm referral failed; trying fallback realm SERVER.DOMAIN.COM [22712] 1524768164.121313: Retrieving user1@USERS.DOMAIN.COM -> krbtgt/SERVER.DOMAIN.COM@SERVER.DOMAIN.COM from FILE:/tmp/krb5cc_738475 with result: -1765328243/Matching credential not found (filename: /tmp/krb5cc_738475) [22712] 1524768164.121366: Retrieving user1@USERS.DOMAIN.COM -> krbtgt/USERS.DOMAIN.COM@USERS.DOMAIN.COM from FILE:/tmp/krb5cc_738475 with result: 0/Success [22712] 1524768164.121372: Starting with TGT for client realm: user1@USERS.DOMAIN.COM -> krbtgt/USERS.DOMAIN.COM@USERS.DOMAIN.COM [22712] 1524768164.121417: Retrieving user1@USERS.DOMAIN.COM -> krbtgt/SERVER.DOMAIN.COM@SERVER.DOMAIN.COM from FILE:/tmp/krb5cc_738475 with result: -1765328243/Matching credential not found (filename: /tmp/krb5cc_738475) [22712] 1524768164.121422: Requesting TGT krbtgt/SERVER.DOMAIN.COM@USERS.DOMAIN.COM using TGT krbtgt/USERS.DOMAIN.COM@USERS.DOMAIN.COM [22712] 1524768164.121434: Generated subkey for TGS request: aes256-cts/31FF [22712] 1524768164.121454: etypes requested in TGS request: aes256-cts, aes128-cts, rc4-hmac [22712] 1524768164.121530: Encoding request body and padata into FAST request [22712] 1524768164.121623: Sending request (9748 bytes) to USERS.DOMAIN.COM [22712] 1524768164.121649: Resolving hostname users.domain.com [22712] 1524768164.122515: Initiating TCP connection to stream YYY.YYY.YYY.YYY:88 [22712] 1524768164.126676: Sending TCP request to stream YYY.YYY.YYY.YYY:88 [22712] 1524768164.135666: Received answer (329 bytes) from stream YYY.YYY.YYY.YYY:88 [22712] 1524768164.135673: Terminating TCP connection to stream YYY.YYY.YYY.YYY:88 [22712] 1524768164.137693: Response was not from master KDC [22712] 1524768164.137706: Decoding FAST response [22712] 1524768164.137731: TGS request result: -1765328377/Server not found in Kerberos database Error connecting: TTransportException, Could not start SASL: Error in sasl_client_start (-1) SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Server not found in Kerberos database) *********************************************************************************** Welcome to the Impala shell. (Impala Shell v2.10.0-cdh5.13.3 (15a453e) built on Sat Mar 17 03:48:31 PDT 2018) Want to know what version of Impala you're connected to? Run the VERSION command to find out! *********************************************************************************** [Not connected] > exit; Goodbye user1
I think the issue may be in what service principal impala-shell is requesting a ticket for. It should be asking for impala/daemonnode.server.domain.com@KRB.DOMAIN.COM but I'm not sure if it ever does.
When I obtain the keytab for that service principal and kinit against it, impala-shell connects.
Additionally, I've been able to authenticate via Kerberos using Tableau (and the Cloudera ODBC driver), which grants me finer-grain control over what specific service principal needs to be authenticated.