Kerberos Authentication Issues with Impala from edgeserver

[#1]

Hi All,

 We have upgrader he VM resources , like CPU and storage is added to each VM in the Cloudera manager. 

  we have done this one by one VM , and also we have upgraded the edgeserv as well, where we have proxy is installed to access the datanodes. our appliaction is using the impala api URL to access the data from the datanode via the proxy using edgeserver.

from java side we can see the the error 

java.sql.SQLException: [Simba][ImpalaJDBCDriver](500310) Invalid operation: Unable to connect to server:;

and checked the edgeserver logs request is recived in proxy and redirct to the data nodes.

in the data nodes logs we can see error logs

E0327 11:57:09.115049 32288 authentication.cc:159] SASL message (Kerberos (external)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request)
I0327 11:57:09.115571 32288 thrift-util.cc:123] TThreadPoolServer: Caught TException: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context
E0327 11:57:09.199386 32288 authentication.cc:159] SASL message (Kerberos (external)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request)
I0327 11:57:09.199843 32288 thrift-util.cc:123] TThreadPoolServer: Caught TException: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context

i have cheked the kerbros ticked it is valid from both the places. from the appliaction serevr and the edgeserver i am using same .keytab ,file and kerboros ticket is valid.

 please help me out to fix this issues, its very urgent to resolve this issue.

Reagrds,

pandu

[#2]

The kerberos ticket is valid but the DN complains about Wrong principal in request

Please review your application configuration or proxy software for correct kerberos principal configured.

As you upgraded your proxy server host, may it be that some settings changed like hostname or krb5.conf?

Another possibility is that there is a mismatch of encryption types, see this KB article

[#3]

Hi gzigldurm,

Thank you fro the reply, 

I have review the proxy server and hostname configuration is same ,where i can see the request is coming to proxy and redirect to any onfe of the datanodes where i can see the kerberos ticket and conf file for the proxy and dtanode is mention below

proxy server krb5.conf
--------------------------

~]$ cat /etc/krb5.conf
[libdefaults]
default_realm = DEV.SIT.COM
dns_lookup_kdc = false
dns_lookup_realm = false
ticket_lifetime = 86400
renew_lifetime = 604800
forwardable = true
default_tgs_enctypes = rc4-hmac aes256-cts-hmac-sha1-96
default_tkt_enctypes = rc4-hmac aes256-cts-hmac-sha1-96
permitted_enctypes = rc4-hmac aes256-cts-hmac-sha1-96
udp_preference_limit = 1
kdc_timeout = 3000
[realms]
DEV.SIT.COM = {
kdc = clouderamanager.hadoop-inventory.local
admin_server = clouderamanager.hadoop-inventory.local
}
[domain_realm]

-----------
kerberos Ticket

# klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: dbadmin@DEV.SIT.COM

Valid starting Expires Service principal
03/29/19 12:00:01 03/30/19 12:00:01 krbtgt/DEV.SIT.COM@DEV.SIT.COM
renew until 04/05/19 13:00:01

principals:
------------

$] klist -ket /home/sit/dbadmin.keytab
Keytab name: FILE:/home/sit/dbadmin.keytab
KVNO Timestamp Principal
---- ----------------- --------------------------------------------------------
1 03/18/19 10:38:24 dbadmin@DEV.SIT.COM (arcfour-hmac)

------------------------------------------------------------------------------

the request is redirect to datanode1 and its krb5 and principals

]# cat /etc/krb5.conf
[libdefaults]
default_realm = DEV.SIT.COM
dns_lookup_kdc = false
dns_lookup_realm = false
ticket_lifetime = 86400
renew_lifetime = 604800
forwardable = true
default_tgs_enctypes = rc4-hmac aes256-cts-hmac-sha1-96
default_tkt_enctypes = rc4-hmac aes256-cts-hmac-sha1-96
permitted_enctypes = rc4-hmac aes256-cts-hmac-sha1-96
udp_preference_limit = 1
kdc_timeout = 3000
[realms]
DEV.SIT.COM = {
kdc = clouderamanager.hadoop-inventory.local
admin_server = clouderamanager.hadoop-inventory.local
}
[domain_realm]

kerberos Ticket in datanode
-----------------------------------------------

# klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: impala/data1@DEV.SIT.COM

Valid starting Expires Service principal
03/25/19 12:02:59 03/26/19 12:02:59 krbtgt/DEV.SIT.COM@DEV.SIT.COM
renew until 04/01/19 13:02:59

principals:
------------------------------------------
]# klist -ket impala.keytab
Keytab name: FILE:impala.keytab
KVNO Timestamp Principal
---- ----------------- --------------------------------------------------------
2 03/25/19 07:34:07 impala/data1@DEV.SIT.COM (aes256-cts-hmac-sha1-96)
2 03/25/19 07:34:07 impala/data1@DEV.SIT.COM (aes128-cts-hmac-sha1-96)
2 03/25/19 07:34:07 impala/data1@DEV.SIT.COM (des3-cbc-sha1)
2 03/25/19 07:34:07 impala/data1@DEV.SIT.COM (arcfour-hmac)
2 03/25/19 07:34:08 impala/data1@DEV.SIT.COM (des-hmac-sha1)
2 03/25/19 07:34:08 impala/data1@DEV.SIT.COM (des-cbc-md5)

--------------------------------------------------

Implad logs i can see this error

E0329 14:04:55.577369 32288 authentication.cc:159] SASL message (Kerberos (external)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request)
I0329 14:04:55.580916 32288 thrift-util.cc:123] TThreadPoolServer: Caught TException: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context
E0329 14:04:55.672466 32288 authentication.cc:159] SASL message (Kerberos (external)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request)
I0329 14:04:55.673733 32288 thrift-util.cc:123] TThreadPoolServer: Caught TException: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context

from the cloudmanegr the kerberos configuration 

]# cat /var/kerberos/krb5kdc/kdc.conf
[kdcdefaults]
kdc_ports = 88
kdc_tcp_ports = 88

[realms]
DEV.SIT.COM = {
#master_key_type = aes256-cts
acl_file = /var/kerberos/krb5kdc/kadm5.acl
dict_file = /usr/share/dict/words
max_renewable_life = 7d
max_life = 1d
admin_keytab = /var/kerberos/krb5kdc/kadm5.keytab
supported_enctypes = aes256-cts:normal aes128-cts:normal des3-hmac-sha1:normal arcfour-hmac:normal des-hmac-sha1:normal des-cbc-md5:normal des-cbc-crc:normal
default_principal_flags = +renewable, +forwardable
}

where in clouder portal  --> administartiom --> settings-- > kerberos 

kerberos encription types mentions as 

rc4-hmac

aes256-cts-hmac-sha1-96

------------------------------------------------------

can you please let me know where the issues accoued in the principals?

you mean to say rc4-hmac is not supported enscription types

Regards,

pandu.

[#4]

Can you kinit using this principal and then run hdfs commands like "hdfs dfs -l /"? Does this work?

[#5 · Accepted solution]

Hi Gzigldrum,

  Thank you for the reply, 

  I found the Route cause and resolved the issue myself,

  my proxy server principal is not listing in the impla demon keytab , impla/proxy@relim  in none of the demon nodes,

  i cross checked the Impala Daemons Load Balancer has confiured to Executor  group feild eariler with value " proxyhost:port"

this where its not reflecting ,because none of the impla demon nodes are present in executor group . 

All the impla demon nodes are present in "impala demon default group, i have added the value '"proxyhost:port" 

in this field and restarted the impala service, when is cross cehck the princaipl in demon node the impala keytab has the impla/proxy@relim, and the kerberos authentication from proxy server started working fro the impala.

Regards,

Pandu