Team,
This error has sucked up my entire week. I have now poured over 40 hours into troubleshooting this error and have made zero progress. I am still getting the same error messages. While not a critical tool for us I do not like things broken. So I really could use advice on how to troubleshoot this or even fix it. What else can I look at?
Versions: CentOS 6.7, Java 1.7, CDH 5.7, MIT Kerberos 5 1.10, Impala with yum download
Build: A 12 node cluster running in AWS. There is no Cloudera Manager. I have enabled HA for HDFS and YARN. I have installed Kerberos. I also have installed SSL using a Java keystore and Java truststore. These are signed with a self-signed cert. HDFS, YARN, MapReduce, Hive, Oozie, and HBase all work from the command line.
Impala fails. Impala did work well prior to installing Kerberos.
Error Message: This is the critical error message. It implies the impala-catalog is not initiating a kerberos ticket.
There are follow on messages about not reaching the metastore. But I have focused on the GSS error.
Java exception follows:
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
IllegalStateException. MetaException. could not connect to meta store using any of hte URIs provided. Most recent failure thrift.transport.TTransportException: GSS initiate failed.
Diag: The impala deamons are able to kinit a tgt but they may not be able to requesting service tickets correctly.
I believe this is a configuration error. Some parameter is not being passed into Impala correctly. Either from Kerberos or from Impala.
Troubleshooting:
1. Impala worked prior to the installation of Kerberos. It failed immediately after installing Kerberos. I have maded lots of different configuration changes on the default impala file. No change in errors.
2. Tested DNS. Valid
3. Validated the default impala file against Cloudera cdh 5.1 manual. Valid
4. Validated the JCE install of jars. Valid and working with KRB. I moved this up to AES256 and then down again. No change in errors.
5. Set default KRB5 to desc3-cbc-sha1 for all principals. Rebuilt the KRB db. Validated all principals are using the same encrypt. No change.
6. Limited the encrypts to only desc3-cbc-sha1. Rebuilt the KRB db. Validated all principals are using the same encyrpt. Allowed weak encryption. No change.
7. Started impala-catalog by hand on the master servers and by service script. Attempt to force other errors. No change.
8. Added the following line into the hadoop-env.sh. Restarted the cluster. No change. This was really an Easter egg, but by this time I was willing to try anything.
# WKD added due to Kerberos issues related to Impala.
export JAVA_OPTS="-Djavax.security.auth.useSubjectCredsOnly=false ${JAVA_OPTS}"
9. I hunted all of the support config files for some parameter that might affect only Impala. After all of the other apps worked. In particulary HBase came up with no hestiation or follow on troubleshooing.
10. I have tried the kinit -R several times and have valdiated I am getting newable tickets.
Current output:
****IMPALA PARAMETERS
--heap_profile_dir=
--hostname=master03.invalid
--keytab_file=/etc/impala/conf/impala.keytab
--krb5_conf=/etc/krb5.conf
--krb5_debug_file=
--mem_limit=80%
--principal=impala/master03.invalid@HADOOPREALM
*****PRINCIPALS
kadmin: getprinc impala/master03.invalid@HADOOPREALM
Principal: impala/master03.invalid@HADOOPREALM
Expiration date: [never]
Last password change: Fri May 27 19:38:20 UTC 2016
Password expiration date: [none]
Maximum ticket life: 2 days 00:00:00
Maximum renewable life: 7 days 00:00:00
Last modified: Fri May 27 19:38:20 UTC 2016 (hdadmin/admin@HADOOPREALM)
Last successful authentication: [never]
Last failed authentication: [never]
Failed password attempts: 0
Number of keys: 4
Key: vno 1, des3-cbc-sha1, no salt
Key: vno 1, arcfour-hmac, no salt
Key: vno 1, des-hmac-sha1, no salt
Key: vno 1, des-cbc-md5, no salt
kadmin: getprinc krbtgt/HADOOPREALM@HADOOPREALM
Principal: krbtgt/HADOOPREALM@HADOOPREALM
Expiration date: [never]
Last password change: [never]
Password expiration date: [none]
Maximum ticket life: 2 days 00:00:00
Maximum renewable life: 7 days 00:00:00
Last modified: Fri May 27 18:45:45 UTC 2016 (db_creation@HADOOPREALM)
Last successful authentication: [never]
Last failed authentication: [never]
Failed password attempts: 0
Number of keys: 5
Key: vno 1, des3-cbc-sha1, no salt
Key: vno 1, arcfour-hmac, no salt
Key: vno 1, des-hmac-sha1, no salt
Key: vno 1, des-cbc-md5, no salt
Key: vno 1, des-cbc-crc, no salt
impala$ klist -e
Ticket cache: FILE:/tmp/krb5cc_490
Default principal: impala/master03.invalid@HADOOPREALM
Valid starting Expires Service principal
05/27/16 21:46:31 05/29/16 21:46:31 krbtgt/HADOOPREALM@HADOOPREALM
renew until 06/03/16 21:45:50, Etype (skey, tkt): des3-cbc-sha1, des3-cbc-sha1
Current Config Files:
*****IMPALA DEFAULT
IMPALA_BACKEND_PORT=22000
IMPALA_STATE_STORE_HOST=master03.invalid
IMPALA_STATE_STORE_PORT=24000
IMPALA_CATALOG_SERVICE_HOST=master03.invalid
IMPALA_CATALOG_SERVICE_PORT=26000
IMPALA_LOG_DIR=/var/log/impala
IMPALA_STATE_STORE_ARGS=" -state_store_port=${IMPALA_STATE_STORE_PORT} -kerberos_reinit_interval=60 -principal=impala/${IMPALA_STATE_STORE_HOST}@HADOOPREALM -keytab_file=/etc/impala/conf/impala.keytab -log_dir=${IMPALA_LOG_DIR} "
IMPALA_CATALOG_ARGS=" -kerberos_reinit_interval=60 -principal=impala/${IMPALA_STATE_STORE_HOST}@HADOOPREALM -keytab_file=/etc/impala/conf/impala.keytab -log_dir=${IMPALA_LOG_DIR} "
IMPALA_SERVER_ARGS=" -be_port=${IMPALA_BACKEND_PORT} -use_statestore -state_store_host=${IMPALA_STATE_STORE_HOST} -state_store_port=${IMPALA_STATE_STORE_PORT} -catalog_service_host=${IMPALA_CATALOG_SERVICE_HOST} -kerberos_reinit_interval=60 -principal=impala/master03.invalid@HADOOPREALM -keytab_file=/etc/impala/conf/impala.keytab -log_dir=${IMPALA_LOG_DIR} "
ENABLE_CORE_DUMPS=false
****KRB5.CON
[libdefaults]
default_realm = HADOOPREALM
dns_lookup_realm = false
dns_lookup_kdc = false
ticket_lifetime = 2d
renew_lifetime = 7d
forwardable = true
renewable = true
allow_weak_crypto = true
default_tgs_enctypes = des3-hmac-sha1 arcfour-hmac des-hmac-sha1 des-cbc-md5 des-cbc-crc aes128-cts aes256-cts
default_tkt_enctypes = des3-hmac-sha1 arcfour-hmac des-hmac-sha1 des-cbc-md5 des-cbc-crc aes128-cts aes256-cts
[realms]
HADOOPREALM = {
kdc = admin01.invalid
admin_server = admin01.invalid
default_domain = invalid
}
[domain_realm]
.invalid = HADOOPREALM
invalid = HADOOPREALM
****KDC.CONF
[kdcdefaults]
kdc_ports = 88,750
[realms]
HADOOPREALM = {
database_name = /var/kerberos/krb5kdc/principal
acl_file = /var/kerberos/krb5kdc/kadm5.acl
dict_file = /var/kerberos/krb5kdc/kadm5.dict
admin_keytab = /var/kerberos/krb5kdc/kadm5.keytab
key_stash_file = /var/kerberos/krb5kdc/.k5.HADOOPREALM
kadmind_port = 749
allow-tickets = true
forwardable = true
renewable = true
max_life = 2d 0h 0m 0s
max_renewable_life = 7d 0h 0m 0s
master_key_type = des3-hmac-sha1
supported_enctypes = des3-hmac-sha1:normal arcfour-hmac:normal des-hmac-sha1:normal des-cbc-md5:normal des-cbc-crc:normal aes128-cbc:normal aes256-cbc:normal
default_principal_flags = +renewable,+forwardable,+postdateable,+proxiable,+tgt-based,+service
}
