Created on 09-21-2016 07:39 AM - edited 09-16-2022 03:40 AM
Hi,
after we enabled Kerberos (AD) on the cluster using the Wizard, Impala failed to start, meanwhile the other services didn't have problems.
It seems that the Principal passed at the StateStore from the Catalog Server is wrong, even if it's configuration has no issues, indeed, as you can see in the logs, the principal has the "at" at the end without the Realm.
Here there are some informations that can be useful:
OS: Red Hat Enterprise Linux Server release 7.2 (Maipo)
CDH Parcel: 5.8.0-1.cdh5.8.0.p0.42
Cloudera Manager: 5.8.1
Service hosts:
Configurations:
krb5.conf:
[libdefaults] default_realm = REALM.COM dns_lookup_kdc = false dns_lookup_realm = false ticket_lifetime = 86400 renew_lifetime = 604800 forwardable = true default_tgs_enctypes = aes256-cts-hmac-sha1-96 rc4-hmac des-cbc-crc des-cbc-md5 default_tkt_enctypes = aes256-cts-hmac-sha1-96 rc4-hmac des-cbc-crc des-cbc-md5 permitted_enctypes = aes256-cts-hmac-sha1-96 rc4-hmac des-cbc-crc des-cbc-md5 udp_preference_limit = 1 kdc_timeout = 3000 [realms] REALM.COM = { kdc = realm.com admin_server = realm.com }
Catalog Server stack trace:
F0921 12:46:36.956574 24185 catalogd-main.cc:76] RPC Error: write() send(): Broken pipe . Impalad exiting. *** Check failure stack trace: *** @ 0x1b465dd (unknown) @ 0x1b48f06 (unknown) @ 0x1b460fd (unknown) @ 0x1b499ae (unknown) @ 0x7f77db (unknown) @ 0x7c39c6 (unknown) @ 0x7f1289e7cb15 __libc_start_main @ 0x7f658d (unknown) Picked up JAVA_TOOL_OPTIONS: -Xms4294967296 -Xmx4294967296 -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh
StateStore stack trace:
E0921 12:46:36.836042 23719 authentication.cc:155] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (No key table entry found matching impala/clouderavalm1@)
Here there are the flags auto generated
Catalog:
-catalog_service_port=26000 -max_log_files=10 -enable_webserver=true -load_auth_to_local_rules=true -load_catalog_in_background=true -webserver_port=25020 -kerberos_reinit_interval=60 -principal=impala/CLOUDERAVALM1@REALM.COM -keytab_file=/run/cloudera-scm-agent/process/818-impala-CATALOGSERVER/impala.keytab -log_filename=catalogd -statestore_subscriber_timeout_seconds=30 -state_store_host=CLOUDERAVALM1 -state_store_port=24000 -minidump_path=/var/log/impala-minidumps -max_minidumps=9
StateStore:
-state_store_pending_task_count_max=0 -max_log_files=10 -state_store_port=24000 -enable_webserver=true -webserver_port=25010 -state_store_num_server_worker_threads=4 -kerberos_reinit_interval=60 -principal=impala/CLOUDERAVALM1@REALM.COM -keytab_file=/run/cloudera-scm-agent/process/820-impala-STATESTORE/impala.keytab -log_filename=statestored -minidump_path=/var/log/impala-minidumps -max_minidumps=9
Here the keytabs auto generated (in the directory: (/run/cloudera-scm-agent/process/...)
Catalog:
slot KVNO Principal ---- ---- --------------------------------------------------------------------- 1 1 impala/CLOUDERAVALM1@REALM.COM 2 1 impala/CLOUDERAVALM1@REALM.COM 3 1 impala/CLOUDERAVALM1@REALM.COM 4 1 impala/CLOUDERAVALM1@REALM.COM 5 1 impala/CLOUDERAVALM1@REALM.COM 6 1 impala/CLOUDERAVALM1@REALM.COM
State store:
slot KVNO Principal ---- ---- --------------------------------------------------------------------- 1 1 impala/CLOUDERAVALM1@REALM.COM 2 1 impala/CLOUDERAVALM1@REALM.COM 3 1 impala/CLOUDERAVALM1@REALM.COM 4 1 impala/CLOUDERAVALM1@REALM.COM 5 1 impala/CLOUDERAVALM1@REALM.COM 6 1 impala/CLOUDERAVALM1@REALM.COM
If necessary, we can provide also other logs.
Thanks.