Member since
09-27-2016
1
Post
1
Kudos Received
0
Solutions
04-26-2018
12:54 PM
1 Kudo
Hello, We seem to have a similar problem with impala-shell also on CDH 5.13.2 (and now 5.13.3). We are using Active Directory KDCs with a one-way trust established between a writable Hadoop AD (KRB.DOMAIN.COM) and our main user AD (USERS.DOMAIN.COM). In addition, our servers are deployed to a third domain (server.domain.com) which does not have an associated KDC. krb5.conf file: [libdefaults]
default_realm = KRB.DOMAIN.COM
dns_lookup_kdc = true
dns_lookup_realm = false
ticket_lifetime = 86400
renew_lifetime = 604800
forwardable = true
default_tgs_enctypes = aes256-cts aes128-cts rc4-hmac
default_tkt_enctypes = aes256-cts aes128-cts rc4-hmac
permitted_enctypes = aes256-cts aes128-cts rc4-hmac
udp_preference_limit = 1
kdc_timeout = 3000
rdns = false
#default_ccache_name = KEYRING:persistent:%{uid}
[realms]
KRB.DOMAIN.COM = {
kdc = krb.domain.com
admin_server = krb.domain.com
default_domain = krb.domain.com
}
USERS.DOMAIN.COM = {
kdc = users.domain.com
admin_server = users.domain.com
}
[domain_realm]
krb.domain.com = KRB.DOMAIN.COM
.krb.domain.com = KRB.DOMAIN.COM
users.domain.com = USERS.DOMAIN.COM
.users.domain.com = USERS.DOMAIN.COM Below is how we invoke impala-shell and the tracing statements I'm seeing after manipulating the KRB5_TRACE env variable: [user1@edgenode ~]$ \impala-shell -k --ssl -i daemonnode.server.domain.com:21000
Starting Impala Shell using Kerberos authentication
Using service name 'impala'
SSL is enabled. Impala server certificates will NOT be verified (set --ca_cert to change)
[22712] 1524768162.661368: ccselect can't find appropriate cache for server principal impala/daemonnode.server.domain.com@
[22712] 1524768162.661450: Getting credentials user1@USERS.DOMAIN.COM -> impala/daemonnode.server.domain.com@ using ccache FILE:/tmp/krb5cc_738475
[22712] 1524768162.661513: Retrieving user1@USERS.DOMAIN.COM -> impala/daemonnode.server.domain.com@ from FILE:/tmp/krb5cc_738475 with result: -1765328243/Matching credential not found (filename: /tmp/krb5cc_738475)
[22712] 1524768162.661558: Retrying user1@USERS.DOMAIN.COM -> impala/daemonnode.server.domain.com@USERS.DOMAIN.COM with result: -1765328243/Matching credential not found (filename: /tmp/krb5cc_738475)
[22712] 1524768162.661563: Server has referral realm; starting with impala/daemonnode.server.domain.com@USERS.DOMAIN.COM
[22712] 1524768162.661627: Retrieving user1@USERS.DOMAIN.COM -> krbtgt/USERS.DOMAIN.COM@USERS.DOMAIN.COM from FILE:/tmp/krb5cc_738475 with result: 0/Success
[22712] 1524768162.661633: Starting with TGT for client realm: user1@USERS.DOMAIN.COM -> krbtgt/USERS.DOMAIN.COM@USERS.DOMAIN.COM
[22712] 1524768162.661640: Requesting tickets for impala/daemonnode.server.domain.com@USERS.DOMAIN.COM, referrals on
[22712] 1524768162.661662: Generated subkey for TGS request: aes256-cts/56A9
[22712] 1524768162.661696: etypes requested in TGS request: aes256-cts, aes128-cts, rc4-hmac
[22712] 1524768162.661790: Encoding request body and padata into FAST request
[22712] 1524768162.661917: Sending request (9771 bytes) to USERS.DOMAIN.COM
[22712] 1524768162.662009: Resolving hostname users.domain.com
[22712] 1524768162.701885: Initiating TCP connection to stream XXX.XXX.XXX.XXX:88
[22712] 1524768162.739233: Sending TCP request to stream XXX.XXX.XXX.XXX:88
[22712] 1524768162.836656: Received answer (351 bytes) from stream XXX.XXX.XXX.XXX:88
[22712] 1524768162.836673: Terminating TCP connection to stream XXX.XXX.XXX.XXX:88
[22712] 1524768164.121113: Response was not from master KDC
[22712] 1524768164.121153: Decoding FAST response
[22712] 1524768164.121209: TGS request result: -1765328377/Server not found in Kerberos database
[22712] 1524768164.121230: Local realm referral failed; trying fallback realm SERVER.DOMAIN.COM
[22712] 1524768164.121313: Retrieving user1@USERS.DOMAIN.COM -> krbtgt/SERVER.DOMAIN.COM@SERVER.DOMAIN.COM from FILE:/tmp/krb5cc_738475 with result: -1765328243/Matching credential not found (filename: /tmp/krb5cc_738475)
[22712] 1524768164.121366: Retrieving user1@USERS.DOMAIN.COM -> krbtgt/USERS.DOMAIN.COM@USERS.DOMAIN.COM from FILE:/tmp/krb5cc_738475 with result: 0/Success
[22712] 1524768164.121372: Starting with TGT for client realm: user1@USERS.DOMAIN.COM -> krbtgt/USERS.DOMAIN.COM@USERS.DOMAIN.COM
[22712] 1524768164.121417: Retrieving user1@USERS.DOMAIN.COM -> krbtgt/SERVER.DOMAIN.COM@SERVER.DOMAIN.COM from FILE:/tmp/krb5cc_738475 with result: -1765328243/Matching credential not found (filename: /tmp/krb5cc_738475)
[22712] 1524768164.121422: Requesting TGT krbtgt/SERVER.DOMAIN.COM@USERS.DOMAIN.COM using TGT krbtgt/USERS.DOMAIN.COM@USERS.DOMAIN.COM
[22712] 1524768164.121434: Generated subkey for TGS request: aes256-cts/31FF
[22712] 1524768164.121454: etypes requested in TGS request: aes256-cts, aes128-cts, rc4-hmac
[22712] 1524768164.121530: Encoding request body and padata into FAST request
[22712] 1524768164.121623: Sending request (9748 bytes) to USERS.DOMAIN.COM
[22712] 1524768164.121649: Resolving hostname users.domain.com
[22712] 1524768164.122515: Initiating TCP connection to stream YYY.YYY.YYY.YYY:88
[22712] 1524768164.126676: Sending TCP request to stream YYY.YYY.YYY.YYY:88
[22712] 1524768164.135666: Received answer (329 bytes) from stream YYY.YYY.YYY.YYY:88
[22712] 1524768164.135673: Terminating TCP connection to stream YYY.YYY.YYY.YYY:88
[22712] 1524768164.137693: Response was not from master KDC
[22712] 1524768164.137706: Decoding FAST response
[22712] 1524768164.137731: TGS request result: -1765328377/Server not found in Kerberos database
Error connecting: TTransportException, Could not start SASL: Error in sasl_client_start (-1) SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Server not found in Kerberos database)
***********************************************************************************
Welcome to the Impala shell.
(Impala Shell v2.10.0-cdh5.13.3 (15a453e) built on Sat Mar 17 03:48:31 PDT 2018)
Want to know what version of Impala you're connected to? Run the VERSION command to
find out!
***********************************************************************************
[Not connected] > exit;
Goodbye user1 I think the issue may be in what service principal impala-shell is requesting a ticket for. It should be asking for impala/daemonnode.server.domain.com@KRB.DOMAIN.COM but I'm not sure if it ever does. When I obtain the keytab for that service principal and kinit against it, impala-shell connects. Additionally, I've been able to authenticate via Kerberos using Tableau (and the Cloudera ODBC driver), which grants me finer-grain control over what specific service principal needs to be authenticated.
... View more