Member since
09-21-2016
3
Posts
0
Kudos Received
0
Solutions
07-25-2019
10:00 AM
is a bit late but i post the solution that worked for me. the problem was the hostnames, impala with kerberos wants the hostnames in lowercase.
... View more
09-21-2016
07:39 AM
Hi, after we enabled Kerberos (AD) on the cluster using the Wizard, Impala failed to start, meanwhile the other services didn't have problems. It seems that the Principal passed at the StateStore from the Catalog Server is wrong, even if it's configuration has no issues, indeed, as you can see in the logs, the principal has the "at" at the end without the Realm. Here there are some informations that can be useful: OS: Red Hat Enterprise Linux Server release 7.2 (Maipo) CDH Parcel: 5.8.0-1.cdh5.8.0.p0.42 Cloudera Manager: 5.8.1 Service hosts: Master (clouderavalm1): Imapala Catalog Server - fails Impala StateStore - success (but gives the problem of the Principal when the catalog tries to authenticate) Other Nodes (clouderavaln0[1-3]): Impala Daemon - fails Configurations: krb5.conf: [libdefaults]
default_realm = REALM.COM
dns_lookup_kdc = false
dns_lookup_realm = false
ticket_lifetime = 86400
renew_lifetime = 604800
forwardable = true
default_tgs_enctypes = aes256-cts-hmac-sha1-96 rc4-hmac des-cbc-crc des-cbc-md5
default_tkt_enctypes = aes256-cts-hmac-sha1-96 rc4-hmac des-cbc-crc des-cbc-md5
permitted_enctypes = aes256-cts-hmac-sha1-96 rc4-hmac des-cbc-crc des-cbc-md5
udp_preference_limit = 1
kdc_timeout = 3000
[realms]
REALM.COM = {
kdc = realm.com
admin_server = realm.com
} Catalog Server stack trace: F0921 12:46:36.956574 24185 catalogd-main.cc:76] RPC Error: write() send(): Broken pipe
. Impalad exiting.
*** Check failure stack trace: ***
@ 0x1b465dd (unknown)
@ 0x1b48f06 (unknown)
@ 0x1b460fd (unknown)
@ 0x1b499ae (unknown)
@ 0x7f77db (unknown)
@ 0x7c39c6 (unknown)
@ 0x7f1289e7cb15 __libc_start_main
@ 0x7f658d (unknown)
Picked up JAVA_TOOL_OPTIONS: -Xms4294967296 -Xmx4294967296 -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh StateStore stack trace: E0921 12:46:36.836042 23719 authentication.cc:155] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure.
Minor code may provide more information (No key table entry found matching impala/clouderavalm1@) Here there are the flags auto generated Catalog: -catalog_service_port=26000
-max_log_files=10
-enable_webserver=true
-load_auth_to_local_rules=true
-load_catalog_in_background=true
-webserver_port=25020
-kerberos_reinit_interval=60
-principal=impala/CLOUDERAVALM1@REALM.COM
-keytab_file=/run/cloudera-scm-agent/process/818-impala-CATALOGSERVER/impala.keytab
-log_filename=catalogd
-statestore_subscriber_timeout_seconds=30
-state_store_host=CLOUDERAVALM1
-state_store_port=24000
-minidump_path=/var/log/impala-minidumps
-max_minidumps=9 StateStore: -state_store_pending_task_count_max=0
-max_log_files=10
-state_store_port=24000
-enable_webserver=true
-webserver_port=25010
-state_store_num_server_worker_threads=4
-kerberos_reinit_interval=60
-principal=impala/CLOUDERAVALM1@REALM.COM
-keytab_file=/run/cloudera-scm-agent/process/820-impala-STATESTORE/impala.keytab
-log_filename=statestored
-minidump_path=/var/log/impala-minidumps
-max_minidumps=9 Here the keytabs auto generated (in the directory: (/run/cloudera-scm-agent/process/...) Catalog: slot KVNO Principal
---- ---- ---------------------------------------------------------------------
1 1 impala/CLOUDERAVALM1@REALM.COM
2 1 impala/CLOUDERAVALM1@REALM.COM
3 1 impala/CLOUDERAVALM1@REALM.COM
4 1 impala/CLOUDERAVALM1@REALM.COM
5 1 impala/CLOUDERAVALM1@REALM.COM
6 1 impala/CLOUDERAVALM1@REALM.COM State store: slot KVNO Principal
---- ---- ---------------------------------------------------------------------
1 1 impala/CLOUDERAVALM1@REALM.COM
2 1 impala/CLOUDERAVALM1@REALM.COM
3 1 impala/CLOUDERAVALM1@REALM.COM
4 1 impala/CLOUDERAVALM1@REALM.COM
5 1 impala/CLOUDERAVALM1@REALM.COM
6 1 impala/CLOUDERAVALM1@REALM.COM If necessary, we can provide also other logs. Thanks.
... View more
Labels: