Created on 01-05-2015 07:05 PM - edited 09-16-2022 02:17 AM
Environment : CDH 5.3.0 Parcels + +kerberos security(MIT kerberos version 5)
Cloudera Manager -> enable Kerberos -> HDFS(ok) -> YARN (MR2 Included)(ok) -> Hive(ok) -> Impala (error)
Using internal kerberos principal "impala/master01.thadoop@THADOOP" Internal communication is authenticated with Kerberos Registering impala/master01.thadoop@THADOOP, keytab file /var/run/cloudera-scm-agent/process/210-impala-STATESTORE/impala.keytab Waiting for Kerberos ticket for principal: impala/master01.thadoop@THADOOP Kerberos ticket granted to impala/master01.thadoop@THADOOP Using external kerberos principal "impala/master01.thadoop@THADOOP" External communication is authenticated with Kerberos statestored version 2.1.0-cdh5 RELEASE (build e48c2b48c53ea9601b8f47a39373aa83ff7ca6e2) Built on Tue, 16 Dec 2014 19:25:34 PST Using hostname: master01.thadoop Flags (see also /varz are on debug webserver): --catalog_service_port=26000 --load_catalog_in_background=true --num_metadata_loading_threads=16 --sentry_config= --disable_optimization_passes=false --dump_ir=false --opt_module= --print_llvm_ir_instruction_count=false --unopt_module= --abort_on_config_error=true --be_port=22000 --be_principal= --compact_catalog_topic=false --disable_mem_pools=false --enable_process_lifetime_heap_profiling=false --heap_profile_dir= --hostname=master01.thadoop --keytab_file=/var/run/cloudera-scm-agent/process/210-impala-STATESTORE/impala.keytab --krb5_conf= --krb5_debug_file= --mem_limit=80% --principal=impala/master01.thadoop@THADOOP --log_filename=statestored --redirect_stdout_stderr=true --data_source_batch_size=1024 --exchg_node_buffer_size_bytes=10485760 --enable_partitioned_aggregation=true --enable_partitioned_hash_join=true --enable_probe_side_filtering=true --skip_lzo_version_check=false --max_row_batches=0 --debug_disable_streaming_gzip=false --enable_phj_probe_side_filtering=true --enable_ldap_auth=false --kerberos_reinit_interval=60 --ldap_allow_anonymous_binds=false --ldap_baseDN= --ldap_bind_pattern= --ldap_ca_certificate= --ldap_domain= --ldap_manual_config=false --ldap_passwords_in_clear_ok=false --ldap_tls=false --ldap_uri= --sasl_path=/usr/lib/sasl2:/usr/lib64/sasl2:/usr/local/lib/sasl2:/usr/lib/x86_64-linux-gnu/sasl2 --rpc_cnxn_attempts=10 --rpc_cnxn_retry_interval_ms=2000 --disk_spill_encryption=false --insert_inherit_permissions=false --max_free_io_buffers=128 --min_buffer_size=1024 --num_disks=0 --num_threads_per_disk=0 --read_size=8388608 --catalog_service_host=localhost --cgroup_hierarchy_path= --enable_rm=false --enable_webserver=true --llama_addresses= --llama_callback_port=28000 --llama_host= --llama_max_request_attempts=5 --llama_port=15000 --llama_registration_timeout_secs=30 --llama_registration_wait_secs=3 --num_hdfs_worker_threads=16 --resource_broker_cnxn_attempts=1 --resource_broker_cnxn_retry_interval_ms=3000 --resource_broker_recv_timeout=0 --resource_broker_send_timeout=0 --staging_cgroup=impala_staging --state_store_host=localhost --state_store_subscriber_port=23000 --use_statestore=true --local_library_dir=/tmp --serialize_batch=false --status_report_interval=5 --num_threads_per_core=3 --scratch_dirs=/tmp --queue_wait_timeout_ms=60000 --default_pool_max_queued=200 --default_pool_max_requests=200 --default_pool_mem_limit= --disable_pool_max_requests=false --disable_pool_mem_limits=false --fair_scheduler_allocation_path= --llama_site_path= --log_mem_usage_interval=0 --authorization_policy_file= --authorization_policy_provider_class=org.apache.sentry.provider.common.HadoopGroupResourceAuthorizationProvider --authorized_proxy_user_config= --load_catalog_at_startup=false --server_name= --abort_on_failed_audit_event=true --audit_event_log_dir= --be_service_threads=64 --beeswax_port=21000 --cancellation_thread_pool_size=5 --default_query_options= --fe_service_threads=64 --hs2_port=21050 --idle_query_timeout=0 --idle_session_timeout=0 --local_nodemanager_url= --log_query_to_file=true --max_audit_event_log_file_size=5000 --max_profile_log_file_size=5000 --max_result_cache_size=100000 --profile_log_dir= --query_log_size=25 --ssl_client_ca_certificate= --ssl_private_key= --ssl_server_certificate= --max_vcore_oversubscription_ratio=2.5 --rm_always_use_defaults=false --rm_default_cpu_vcores=2 --rm_default_memory=4G --disable_admission_control=true --require_username=false --statestore_subscriber_cnxn_attempts=10 --statestore_subscriber_cnxn_retry_interval_ms=3000 --statestore_subscriber_timeout_seconds=30 --state_store_port=24000 --statestore_heartbeat_frequency_ms=1000 --statestore_max_missed_heartbeats=10 --statestore_num_heartbeat_threads=10 --statestore_num_update_threads=10 --statestore_update_frequency_ms=2000 --force_lowercase_usernames=false --num_cores=0 --web_log_bytes=1048576 --non_impala_java_vlog=0 --periodic_counter_update_period_ms=500 --enable_webserver_doc_root=true --webserver_authentication_domain= --webserver_certificate_file= --webserver_doc_root=/opt/cloudera/parcels/CDH-5.3.0-1.cdh5.3.0.p0.30/lib/impala --webserver_interface= --webserver_password_file= --webserver_port=25010 --flagfile=/var/run/cloudera-scm-agent/process/210-impala-STATESTORE/impala-conf/state_store_flags --fromenv= --tryfromenv= --undefok= --tab_completion_columns=80 --tab_completion_word= --help=false --helpfull=false --helpmatch= --helpon= --helppackage=false --helpshort=false --helpxml=false --version=false --alsologtoemail= --alsologtostderr=false --drop_log_memory=true --log_backtrace_at= --log_dir=/var/log/statestore --log_link= --log_prefix=true --logbuflevel=0 --logbufsecs=30 --logbufvlevel=1 --logemaillevel=999 --logmailer=/bin/mail --logtostderr=false --max_log_size=200 --minloglevel=0 --stderrthreshold=4 --stop_logging_if_full_disk=false --symbolize_stacktrace=true --v=1 --vmodule= Cpu Info: Model: QEMU Virtual CPU version 0.14.1 Cores: 4 L1 Cache: 32.00 KB L2 Cache: 2.00 MB L3 Cache: 0 Hardware Supports: popcnt Disk Info: Num disks 1: vda (rotational=true) Physical Memory: 7.69 GB OS version: Linux version 2.6.32-431.el6.x86_64 (mockbuild@c6b8.bsys.dev.centos.org) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-4) (GCC) ) #1 SMP Fri Nov 22 03:15:09 UTC 2013 Process ID: 22645 Starting webserver on 0.0.0.0:25010 Document root: /opt/cloudera/parcels/CDH-5.3.0-1.cdh5.3.0.p0.30/lib/impala Webserver started ThriftServer 'StatestoreService' started on port: 24000 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request) TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request) TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request) TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context Failed to extend Kerberos ticket. Error: Shell cmd: 'kinit -R' exited with an error: ''. Output was: ''. Failure count: 1 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request) TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request) TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request) TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request) TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request) TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request) TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request) TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request) TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request) TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request) TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wr TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request) TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request) TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context
path : /var/kerberos/krb5kdc/kdc.conf
[kdcdefaults]
kdc_ports = 88
kdc_tcp_ports = 88
[realms]
THADOOP.COM = {
#master_key_type = aes256-cts
acl_file = /var/kerberos/krb5kdc/kadm5.acl
dict_file = /usr/share/dict/words
admin_keytab = /var/kerberos/krb5kdc/kadm5.keytab
supported_enctypes = des3-hmac-sha1:normal arcfour-hmac:normal des-hmac-sha1:normal des-cbc-md5:normal des-cbc-crc:normal des-cbc-crc:v4 des-cbc-crc:afs3
max_life = 30d
max_renewable_life = 30d
default_principal_flags = +renewable, +forwardable
}
path : /etc/krb5.conf
[logging]
default = FILE:/var/log/krb5libs.log
kdc = FILE:/var/log/krb5kdc.log
admin_server = FILE:/var/log/kadmind.log
[libdefaults]
default_realm = THADOOP
dns_lookup_realm = false
dns_lookup_kdc = false
ticket_lifetime = 32d
renew_lifetime = 32d
forwardable = true
renewable = true
udp_preference_limit = 1
default_tgs_enctypes = arcfour-hmac
default_tkt_enctypes = arcfour-hmac
[realms]
THADOOP = {
kdc = kerberos.thadoop
admin_server = kerberos.thadoop
}
[domain_realm]
.thadoop = THADOOP
thadoop = THADOOP
path : /var/kerberos/krb5kdc/kadm5.acl
*/admin@THADOOP *
and...
[root@master01 210-impala-STATESTORE]# klist -e
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: admin/admin@THADOOP
Valid starting Expires Service principal
01/06/15 10:08:42 01/07/15 10:08:42 krbtgt/THADOOP@THADOOP
renew until 01/06/15 10:08:42, Etype (skey, tkt): des3-cbc-sha1, aes256-cts-hmac-sha1-96
============================================================
[root@master01 210-impala-STATESTORE]# pwd
/var/run/cloudera-scm-agent/process/210-impala-STATESTORE
[root@master01 210-impala-STATESTORE]# klist -ket impala.keytab
Keytab name: FILE:impala.keytab
KVNO Timestamp Principal
---- ----------------- --------------------------------------------------------
2 01/06/15 10:13:43 impala/master01.thadoop@THADOOP (aes256-cts-hmac-sha1-96)
2 01/06/15 10:13:43 impala/master01.thadoop@THADOOP (aes128-cts-hmac-sha1-96)
2 01/06/15 10:13:44 impala/master01.thadoop@THADOOP (des3-cbc-sha1)
2 01/06/15 10:13:44 impala/master01.thadoop@THADOOP (arcfour-hmac)
(There is no HTTP.keytab this is normal?)
by the way...
Kerberos Encryption Types : des3-cbc-sha1 (default rc4-hmac)
Anyone have any suggestions how to resolve this problem?
Created on 02-15-2015 05:09 PM - edited 02-15-2015 05:10 PM
Hi Jason,
This might be due to the mismatch of encryption types between clients and the KDC server. Please follow the below steps and see if it helps.
1. Stop the cluster through CM
2. Go to CM --> Administration --> Kerberos --> 'Kerberos Encryption Types', then add the following encryption types:
des3-hmac-sha1
arcfour-hmac
des-hmac-sha1
des-cbc-md5
des-cbc-crc
3. Do redeploy krb5.conf through CM, and restart the cluster.
Daisuke
Created 06-01-2018 01:31 PM
@vvinaga It looks like it cannot talk to the HDFS NameNode from the logs. Could you check if HDFS is configured correctly to use Kerberos?
Created 08-21-2017 12:43 AM
HI,
we are now facing same kerberos authentication failure issue due to not using all below encryption types while generating the keytab.
Because, now IT infrastructure/AD team not allowed to use below weak encryption types while generating keytab as it's denied by corporate policy. And we are only allowed to use below string encryptions.
Could you please advise for the alternate solutions for this authentication issue?
Weak encryptions:
DES-CBC-CRC | DES-CBC-MD5 | RC4-HMAC-NT
Strong encryptions:
AES256-SHA1 | AES128-SHA1
Thanks in Adv,
Sanjay
Created on 08-21-2017 12:45 AM - edited 08-21-2017 12:49 AM
HI,
we are now facing same kerberos authentication failure issue due to not using all below encryption types while generating the keytab.
Because, now IT infrastructure/AD team not allowed to use below weak encryption types while generating keytab as it's being denied by corporate policy. And we are only allowed to use below strong encryptions.
Could you please advise for the alternate solutions for this authentication issue?
Weak encryptions:
DES-CBC-CRC | DES-CBC-MD5 | RC4-HMAC-NT
Strong encryptions:
AES256-SHA1 | AES128-SHA1