Support Questions

Find answers, ask questions, and share your expertise

kerberos authentication failure: GSSAPI Failure: gss_accept_sec_context

avatar
New Contributor

Environment : CDH 5.3.0 Parcels + +kerberos security(MIT kerberos version 5)

 

Cloudera Manager -> enable Kerberos -> HDFS(ok) -> YARN (MR2 Included)(ok) -> Hive(ok) -> Impala (error)

 

 

	
Using internal kerberos principal "impala/master01.thadoop@THADOOP"
	
Internal communication is authenticated with Kerberos
	
Registering impala/master01.thadoop@THADOOP, keytab file /var/run/cloudera-scm-agent/process/210-impala-STATESTORE/impala.keytab
	
Waiting for Kerberos ticket for principal: impala/master01.thadoop@THADOOP

Kerberos ticket granted to impala/master01.thadoop@THADOOP

Using external kerberos principal "impala/master01.thadoop@THADOOP"
	
External communication is authenticated with Kerberos
	
statestored version 2.1.0-cdh5 RELEASE (build e48c2b48c53ea9601b8f47a39373aa83ff7ca6e2)
Built on Tue, 16 Dec 2014 19:25:34 PST
	
Using hostname: master01.thadoop
	
Flags (see also /varz are on debug webserver):
--catalog_service_port=26000
--load_catalog_in_background=true
--num_metadata_loading_threads=16
--sentry_config=
--disable_optimization_passes=false
--dump_ir=false
--opt_module=
--print_llvm_ir_instruction_count=false
--unopt_module=
--abort_on_config_error=true
--be_port=22000
--be_principal=
--compact_catalog_topic=false
--disable_mem_pools=false
--enable_process_lifetime_heap_profiling=false
--heap_profile_dir=
--hostname=master01.thadoop
--keytab_file=/var/run/cloudera-scm-agent/process/210-impala-STATESTORE/impala.keytab
--krb5_conf=
--krb5_debug_file=
--mem_limit=80%
--principal=impala/master01.thadoop@THADOOP
--log_filename=statestored
--redirect_stdout_stderr=true
--data_source_batch_size=1024
--exchg_node_buffer_size_bytes=10485760
--enable_partitioned_aggregation=true
--enable_partitioned_hash_join=true
--enable_probe_side_filtering=true
--skip_lzo_version_check=false
--max_row_batches=0
--debug_disable_streaming_gzip=false
--enable_phj_probe_side_filtering=true
--enable_ldap_auth=false
--kerberos_reinit_interval=60
--ldap_allow_anonymous_binds=false
--ldap_baseDN=
--ldap_bind_pattern=
--ldap_ca_certificate=
--ldap_domain=
--ldap_manual_config=false
--ldap_passwords_in_clear_ok=false
--ldap_tls=false
--ldap_uri=
--sasl_path=/usr/lib/sasl2:/usr/lib64/sasl2:/usr/local/lib/sasl2:/usr/lib/x86_64-linux-gnu/sasl2
--rpc_cnxn_attempts=10
--rpc_cnxn_retry_interval_ms=2000
--disk_spill_encryption=false
--insert_inherit_permissions=false
--max_free_io_buffers=128
--min_buffer_size=1024
--num_disks=0
--num_threads_per_disk=0
--read_size=8388608
--catalog_service_host=localhost
--cgroup_hierarchy_path=
--enable_rm=false
--enable_webserver=true
--llama_addresses=
--llama_callback_port=28000
--llama_host=
--llama_max_request_attempts=5
--llama_port=15000
--llama_registration_timeout_secs=30
--llama_registration_wait_secs=3
--num_hdfs_worker_threads=16
--resource_broker_cnxn_attempts=1
--resource_broker_cnxn_retry_interval_ms=3000
--resource_broker_recv_timeout=0
--resource_broker_send_timeout=0
--staging_cgroup=impala_staging
--state_store_host=localhost
--state_store_subscriber_port=23000
--use_statestore=true
--local_library_dir=/tmp
--serialize_batch=false
--status_report_interval=5
--num_threads_per_core=3
--scratch_dirs=/tmp
--queue_wait_timeout_ms=60000
--default_pool_max_queued=200
--default_pool_max_requests=200
--default_pool_mem_limit=
--disable_pool_max_requests=false
--disable_pool_mem_limits=false
--fair_scheduler_allocation_path=
--llama_site_path=
--log_mem_usage_interval=0
--authorization_policy_file=
--authorization_policy_provider_class=org.apache.sentry.provider.common.HadoopGroupResourceAuthorizationProvider
--authorized_proxy_user_config=
--load_catalog_at_startup=false
--server_name=
--abort_on_failed_audit_event=true
--audit_event_log_dir=
--be_service_threads=64
--beeswax_port=21000
--cancellation_thread_pool_size=5
--default_query_options=
--fe_service_threads=64
--hs2_port=21050
--idle_query_timeout=0
--idle_session_timeout=0
--local_nodemanager_url=
--log_query_to_file=true
--max_audit_event_log_file_size=5000
--max_profile_log_file_size=5000
--max_result_cache_size=100000
--profile_log_dir=
--query_log_size=25
--ssl_client_ca_certificate=
--ssl_private_key=
--ssl_server_certificate=
--max_vcore_oversubscription_ratio=2.5
--rm_always_use_defaults=false
--rm_default_cpu_vcores=2
--rm_default_memory=4G
--disable_admission_control=true
--require_username=false
--statestore_subscriber_cnxn_attempts=10
--statestore_subscriber_cnxn_retry_interval_ms=3000
--statestore_subscriber_timeout_seconds=30
--state_store_port=24000
--statestore_heartbeat_frequency_ms=1000
--statestore_max_missed_heartbeats=10
--statestore_num_heartbeat_threads=10
--statestore_num_update_threads=10
--statestore_update_frequency_ms=2000
--force_lowercase_usernames=false
--num_cores=0
--web_log_bytes=1048576
--non_impala_java_vlog=0
--periodic_counter_update_period_ms=500
--enable_webserver_doc_root=true
--webserver_authentication_domain=
--webserver_certificate_file=
--webserver_doc_root=/opt/cloudera/parcels/CDH-5.3.0-1.cdh5.3.0.p0.30/lib/impala
--webserver_interface=
--webserver_password_file=
--webserver_port=25010
--flagfile=/var/run/cloudera-scm-agent/process/210-impala-STATESTORE/impala-conf/state_store_flags
--fromenv=
--tryfromenv=
--undefok=
--tab_completion_columns=80
--tab_completion_word=
--help=false
--helpfull=false
--helpmatch=
--helpon=
--helppackage=false
--helpshort=false
--helpxml=false
--version=false
--alsologtoemail=
--alsologtostderr=false
--drop_log_memory=true
--log_backtrace_at=
--log_dir=/var/log/statestore
--log_link=
--log_prefix=true
--logbuflevel=0
--logbufsecs=30
--logbufvlevel=1
--logemaillevel=999
--logmailer=/bin/mail
--logtostderr=false
--max_log_size=200
--minloglevel=0
--stderrthreshold=4
--stop_logging_if_full_disk=false
--symbolize_stacktrace=true
--v=1
--vmodule=
	
Cpu Info:
  Model: QEMU Virtual CPU version 0.14.1
  Cores: 4
  L1 Cache: 32.00 KB
  L2 Cache: 2.00 MB
  L3 Cache: 0
  Hardware Supports:
    popcnt
	
Disk Info: 
  Num disks 1: 
    vda (rotational=true)

	
Physical Memory: 7.69 GB
	
OS version: Linux version 2.6.32-431.el6.x86_64 (mockbuild@c6b8.bsys.dev.centos.org) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-4) (GCC) ) #1 SMP Fri Nov 22 03:15:09 UTC 2013
	
Process ID: 22645
	
Starting webserver on 0.0.0.0:25010
	
Document root: /opt/cloudera/parcels/CDH-5.3.0-1.cdh5.3.0.p0.30/lib/impala
	
Webserver started

ThriftServer 'StatestoreService' started on port: 24000
	
SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Wrong principal in request)
	
TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context
	
SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Wrong principal in request)

TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context
	
SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Wrong principal in request)

TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context
	
Failed to extend Kerberos ticket. Error: Shell cmd: 'kinit -R' exited with an error: ''. Output was: ''. Failure count: 1
	
SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Wrong principal in request)
	
TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context
	
SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Wrong principal in request)
	
TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context
	
SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Wrong principal in request)

TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context
	
SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Wrong principal in request)
	
TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context
	
SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Wrong principal in request)
	
TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context

SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Wrong principal in request)

TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context
	
SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Wrong principal in request)
	
TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context
	
SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Wrong principal in request)
	
TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context
	
SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Wrong principal in request)
	
TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context
	
SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Wrong principal in request)

TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context
	
SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Wr
	
TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context
	
SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Wrong principal in request)

TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context

SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Wrong principal in request)
	
TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context

path : /var/kerberos/krb5kdc/kdc.conf

 

[kdcdefaults]
kdc_ports = 88
kdc_tcp_ports = 88

[realms]
THADOOP.COM = {
#master_key_type = aes256-cts
acl_file = /var/kerberos/krb5kdc/kadm5.acl
dict_file = /usr/share/dict/words
admin_keytab = /var/kerberos/krb5kdc/kadm5.keytab
supported_enctypes = des3-hmac-sha1:normal arcfour-hmac:normal des-hmac-sha1:normal des-cbc-md5:normal des-cbc-crc:normal des-cbc-crc:v4 des-cbc-crc:afs3
max_life = 30d
max_renewable_life = 30d
default_principal_flags = +renewable, +forwardable
}

 

path : /etc/krb5.conf

 

[logging]
default = FILE:/var/log/krb5libs.log
kdc = FILE:/var/log/krb5kdc.log
admin_server = FILE:/var/log/kadmind.log

[libdefaults]
default_realm = THADOOP
dns_lookup_realm = false
dns_lookup_kdc = false
ticket_lifetime = 32d
renew_lifetime = 32d
forwardable = true
renewable = true
udp_preference_limit = 1
default_tgs_enctypes = arcfour-hmac
default_tkt_enctypes = arcfour-hmac

[realms]
THADOOP = {
kdc = kerberos.thadoop
admin_server = kerberos.thadoop
}

[domain_realm]
.thadoop = THADOOP
thadoop = THADOOP

path : /var/kerberos/krb5kdc/kadm5.acl

 

*/admin@THADOOP *

 

and...

 

[root@master01 210-impala-STATESTORE]# klist -e
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: admin/admin@THADOOP

Valid starting Expires Service principal
01/06/15 10:08:42 01/07/15 10:08:42 krbtgt/THADOOP@THADOOP
renew until 01/06/15 10:08:42, Etype (skey, tkt): des3-cbc-sha1, aes256-cts-hmac-sha1-96

 

============================================================

 

[root@master01 210-impala-STATESTORE]# pwd
/var/run/cloudera-scm-agent/process/210-impala-STATESTORE
[root@master01 210-impala-STATESTORE]# klist -ket impala.keytab
Keytab name: FILE:impala.keytab
KVNO Timestamp Principal
---- ----------------- --------------------------------------------------------
2 01/06/15 10:13:43 impala/master01.thadoop@THADOOP (aes256-cts-hmac-sha1-96)
2 01/06/15 10:13:43 impala/master01.thadoop@THADOOP (aes128-cts-hmac-sha1-96)
2 01/06/15 10:13:44 impala/master01.thadoop@THADOOP (des3-cbc-sha1)
2 01/06/15 10:13:44 impala/master01.thadoop@THADOOP (arcfour-hmac)

 

(There is no HTTP.keytab this is normal?)

 

by the way...

 

Kerberos Encryption Types : des3-cbc-sha1 (default rc4-hmac) 

 

Anyone have any suggestions how to resolve this problem?

1 ACCEPTED SOLUTION

avatar
Expert Contributor

Hi Jason,

 

This might be due to the mismatch of encryption types between clients and the KDC server. Please follow the below steps and see if it helps.

1. Stop the cluster through CM
2. Go to CM --> Administration --> Kerberos --> 'Kerberos Encryption Types', then add the following encryption types:

des3-hmac-sha1
arcfour-hmac
des-hmac-sha1
des-cbc-md5
des-cbc-crc

3. Do redeploy krb5.conf through CM, and restart the cluster.

 

Daisuke

View solution in original post

23 REPLIES 23

avatar
Explorer

Hello, please help

 

After enabled Kerberos, impala daemon for all data nodes doesn't start, but at the same time Impala Catalog Server and Impala StateStore have been started successfully (with Kerberos).

 

 

log file /var/log/impalad :

 

Log file created at: 2018/06/01 12:03:58
Running on machine: hadoop-srv01.STAT.RO
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
I0601 12:03:58.609741  1344 logging.cc:125] stdout will be logged to this file.
E0601 12:03:58.610020  1344 logging.cc:126] stderr will be logged to this file.
I0601 12:03:58.610366  1344 minidump.cc:231] Setting minidump size limit to 20971520.
I0601 12:03:58.610431  1344 atomicops-internals-x86.cc:98] vendor GenuineIntel  family 6  model 15  sse2 1  cmpxchg16b 1
I0601 12:03:58.619076  1344 authentication.cc:722] Using internal kerberos principal "impala/hadoop-srv01.STAT.RO@MAIN.STAT.RO"
I0601 12:03:58.619089  1344 authentication.cc:1058] Internal communication is authenticated with Kerberos
I0601 12:03:58.619333  1344 authentication.cc:840] Waiting for Kerberos ticket for principal: impala/hadoop-srv01.STAT.RO@MAIN.STAT.RO
I0601 12:03:58.619376  1411 authentication.cc:498] Registering impala/hadoop-srv01.STAT.RO@MAIN.STAT.RO, keytab file /var/run/cloudera-scm-agent/process/288-impala-IMPALAD/impala.keytab
I0601 12:03:58.636188  1344 authentication.cc:842] Kerberos ticket granted to impala/hadoop-srv01.STAT.RO@MAIN.STAT.RO
I0601 12:03:58.636248  1344 authentication.cc:722] Using external kerberos principal "impala/hadoop-srv01.STAT.RO@MAIN.STAT.RO"
I0601 12:03:58.636255  1344 authentication.cc:1074] External communication is authenticated with Kerberos
I0601 12:03:58.636765  1344 init.cc:221] impalad version 2.10.0-cdh5.13.1 RELEASE (build 1e4b23c4eb52dac95c5be6316f49685c41783c51)
Built on Thu Nov  9 08:29:47 PST 2017
I0601 12:03:58.636775  1344 init.cc:222] Using hostname: hadoop-srv01.STAT.RO
I0601 12:03:58.637948  1344 logging.cc:161] Flags (see also /varz are on debug webserver):
--catalog_service_port=26000
--initial_hms_cnxn_timeout_s=120
--load_catalog_in_background=false
--num_metadata_loading_threads=16
--sentry_catalog_polling_frequency_s=60
--sentry_config=
--asm_module_dir=
--disable_optimization_passes=false
--dump_ir=false
--opt_module_dir=
--perf_map=false
--print_llvm_ir_instruction_count=false
--unopt_module_dir=
--abort_on_config_error=true
--be_port=22000
--be_principal=
--buffer_pool_clean_pages_limit=10%
--buffer_pool_limit=80%
--compact_catalog_topic=false
--disable_kudu=false
--disable_mem_pools=false
--enable_accept_queue_server=true
--enable_minidumps=true
--enable_process_lifetime_heap_profiling=false
--enable_stats_extrapolation=false
--heap_profile_dir=
--hostname=hadoop-srv01.STAT.RO
--inc_stats_size_limit_bytes=209715200
--keytab_file=/var/run/cloudera-scm-agent/process/288-impala-IMPALAD/impala.keytab
--krb5_conf=
--krb5_debug_file=
--kudu_operation_timeout_ms=180000
--load_auth_to_local_rules=false
--max_minidumps=9
--mem_limit=49893343232
--min_buffer_size=65536
--minidump_path=/var/log/impala-minidumps/impalad
--minidump_size_limit_hint_kb=20480
--principal=impala/hadoop-srv01.STAT.RO@MAIN.STAT.RO
--redaction_rules_file=
--local_library_dir=/var/lib/impala/udfs
--max_audit_event_log_files=0
--max_log_files=10
--memory_maintenance_sleep_time_ms=10000
--pause_monitor_sleep_time_ms=500
--pause_monitor_warn_threshold_ms=10000
--log_filename=impalad
--redirect_stdout_stderr=true
--data_source_batch_size=1024
--exchg_node_buffer_size_bytes=10485760
--enable_quadratic_probing=true
--skip_lzo_version_check=false
--parquet_min_filter_reject_ratio=0.10000000000000001
--runtime_filter_wait_time_ms=1000
--suppress_unknown_disk_id_warnings=false
--max_row_batches=0
--kudu_max_row_batches=0
--kudu_read_mode=READ_LATEST
--kudu_scanner_keep_alive_period_sec=15
--pick_only_leaders_for_tests=false
--kudu_mutation_buffer_size=10485760
--kudu_sink_mem_required=20971520
--convert_legacy_hive_parquet_utc_timestamps=false
--max_page_header_size=8388608
--accepted_cnxn_queue_depth=10000
--enable_ldap_auth=false
--internal_principals_whitelist=hdfs
--kerberos_reinit_interval=60
--ldap_allow_anonymous_binds=false
--ldap_baseDN=
--ldap_bind_pattern=
--ldap_ca_certificate=
--ldap_domain=
--ldap_manual_config=false
--ldap_passwords_in_clear_ok=false
--ldap_tls=false
--ldap_uri=
--sasl_path=
--concurrent_scratch_ios_per_device=2
--madvise_huge_pages=true
--mmap_buffers=false
--insert_inherit_permissions=false
--datastream_sender_timeout_ms=120000
--adls_read_chunk_size=131072
--use_hdfs_pread=false
--max_cached_file_handles=0
--max_free_io_buffers=128
--num_adls_io_threads=16
--num_disks=0
--num_io_threads_per_rotational_disk=0
--num_io_threads_per_solid_state_disk=0
--num_remote_hdfs_io_threads=8
--num_s3_io_threads=16
--num_threads_per_disk=0
--read_size=8388608
--unused_file_handle_timeout_sec=270
--backend_client_connection_num_retries=3
--backend_client_rpc_timeout_ms=300000
--catalog_client_connection_num_retries=3
--catalog_client_rpc_timeout_ms=0
--catalog_service_host=hadoop-mgmt01.STAT.RO
--coordinator_rpc_threads=12
--disable_admission_control=false
--enable_webserver=true
--num_hdfs_worker_threads=16
--state_store_host=hadoop-mgmt01.STAT.RO
--state_store_subscriber_port=23000
--status_report_interval=5
--s3a_access_key_cmd=
--s3a_secret_key_cmd=
--log_mem_usage_interval=0
--max_filter_error_rate=0.75
--num_threads_per_core=3
--use_local_tz_for_unix_timestamp_conversions=false
--allow_multiple_scratch_dirs_per_device=false
--disk_spill_encryption=false
--scratch_dirs=/data/data01/impala/impalad,/data/data02/impala/impalad,/data/data03/impala/impalad,/data/data04/impala/impalad,/data/data05/impala/impalad,/data/data06/impala/impalad,/data/data07/impala/impalad,/data/data08/impala/impalad,/data/data09/impala/impalad,/data/data10/impala/impalad,/data/data11/impala/impalad,/data/data12/impala/impalad,/data/data13/impala/impalad
--queue_wait_timeout_ms=60000
--default_pool_max_queued=200
--default_pool_max_requests=-1
--default_pool_mem_limit=
--disable_pool_max_requests=false
--disable_pool_mem_limits=false
--fair_scheduler_allocation_path=/var/run/cloudera-scm-agent/process/288-impala-IMPALAD/impala-conf/fair-scheduler.xml
--llama_site_path=/var/run/cloudera-scm-agent/process/288-impala-IMPALAD/impala-conf/llama-site.xml
--require_username=false
--authorization_policy_file=
--authorization_policy_provider_class=org.apache.sentry.provider.common.HadoopGroupResourceAuthorizationProvider
--authorized_proxy_user_config=
--authorized_proxy_user_config_delimiter=,
--kudu_master_hosts=
--server_name=
--abort_on_failed_audit_event=false
--abort_on_failed_lineage_event=true
--audit_event_log_dir=/var/log/impalad/audit
--beeswax_port=21000
--cancellation_thread_pool_size=5
--default_query_options=
--fe_service_threads=64
--hs2_port=21050
--idle_query_timeout=0
--idle_session_timeout=0
--is_coordinator=true
--is_executor=true
--lineage_event_log_dir=/var/log/impalad/lineage
--log_query_to_file=true
--max_audit_event_log_file_size=5000
--max_lineage_log_file_size=5000
--max_profile_log_file_size=5000
--max_profile_log_files=10
--max_result_cache_size=100000
--profile_log_dir=
--query_log_size=25
--ssl_cipher_list=
--ssl_client_ca_certificate=
--ssl_minimum_version=tlsv1
--ssl_private_key=
--ssl_private_key_password_cmd=
--ssl_server_certificate=
--statestore_subscriber_cnxn_attempts=10
--statestore_subscriber_cnxn_retry_interval_ms=3000
--statestore_subscriber_timeout_seconds=30
--state_store_port=24000
--statestore_heartbeat_frequency_ms=1000
--statestore_heartbeat_tcp_timeout_seconds=3
--statestore_max_missed_heartbeats=10
--statestore_num_heartbeat_threads=10
--statestore_num_update_threads=10
--statestore_update_frequency_ms=2000
--statestore_update_tcp_timeout_seconds=300
--force_lowercase_usernames=false
--num_cores=0
--web_log_bytes=1048576
--non_impala_java_vlog=0
--periodic_counter_update_period_ms=500
--enable_webserver_doc_root=true
--webserver_authentication_domain=
--webserver_certificate_file=
--webserver_doc_root=/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala
--webserver_interface=
--webserver_password_file=
--webserver_port=25000
--webserver_private_key_file=
--webserver_private_key_password_cmd=
--webserver_x_frame_options=DENY
--flagfile=/var/run/cloudera-scm-agent/process/288-impala-IMPALAD/impala-conf/impalad_flags
--fromenv=
--tryfromenv=
--undefok=
--tab_completion_columns=80
--tab_completion_word=
--help=false
--helpfull=false
--helpmatch=
--helpon=
--helppackage=false
--helpshort=false
--helpxml=false
--version=false
--alsologtoemail=
--alsologtostderr=false
--colorlogtostderr=false
--drop_log_memory=true
--log_backtrace_at=
--log_dir=/var/log/impalad
--log_link=
--log_prefix=true
--logbuflevel=0
--logbufsecs=30
--logemaillevel=999
--logmailer=/bin/mail
--logtostderr=false
--max_log_size=200
--minloglevel=0
--stderrthreshold=4
--stop_logging_if_full_disk=false
--symbolize_stacktrace=true
--v=1
--vmodule=
I0601 12:03:58.638094  1344 init.cc:227] Cpu Info:
  Model: Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
  Cores: 40
  Max Possible Cores: 40
  L1 Cache: 32.00 KB (Line: 64.00 B)
  L2 Cache: 256.00 KB (Line: 64.00 B)
  L3 Cache: 25.00 MB (Line: 64.00 B)
  Hardware Supports:
    ssse3
    sse4_1
    sse4_2
    popcnt
    avx
    avx2
  Numa Nodes: 2
  Numa Nodes of Cores: 0->0 | 1->0 | 2->0 | 3->0 | 4->0 | 5->0 | 6->0 | 7->0 | 8->0 | 9->0 | 10->1 | 11->1 | 12->1 | 13->1 | 14->1 | 15->1 | 16->1 | 17->1 | 18->1 | 19->1 | 20->0 | 21->0 | 22->0 | 23->0 | 24->0 | 25->0 | 26->0 | 27->0 | 28->0 | 29->0 | 30->1 | 31->1 | 32->1 | 33->1 | 34->1 | 35->1 | 36->1 | 37->1 | 38->1 | 39->1 |
I0601 12:03:58.638119  1344 init.cc:228] Disk Info:
  Num disks 15:
    sdc (rotational=true)
    sde (rotational=true)
    sdf (rotational=true)
    sdh (rotational=true)
    sdb (rotational=true)
    sdd (rotational=true)
    sdj (rotational=true)
    sdi (rotational=true)
    sdg (rotational=true)
    sda (rotational=false)
    sdk (rotational=true)
    sdm (rotational=true)
    sdn (rotational=true)
    sdl (rotational=true)
    dm- (rotational=true)

I0601 12:03:58.638208  1344 init.cc:229] Physical Memory: 126.04 GB
Transparent Huge Pages Config:
  enabled: always madvise [never]
  defrag: always madvise [never]
  khugepaged defrag: [yes] no
I0601 12:03:58.638216  1344 init.cc:230] OS version: Linux version 2.6.32-696.16.1.el6.x86_64 (mockbuild@c1bl.rdu2.centos.org) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-18) (GCC) ) #1 SMP Wed Nov 15 16:51:15 UTC 2017
Clock: clocksource: 'tsc', clockid_t: CLOCK_MONOTONIC
I0601 12:03:58.638221  1344 init.cc:231] Process ID: 1344
I0601 12:04:00.055625  1344 llvm-codegen.cc:136] CPU class for runtime code generation: broadwell
I0601 12:04:00.055665  1344 llvm-codegen.cc:139] CPU flags for runtime code generation: -sse4a,-avx512bw,+cx16,-tbm,+xsave,-fma4,-avx512vl,+prfchw,+bmi2,+adx,-xsavec,+fsgsbase,+avx,-avx512cd,-avx512pf,+rtm,+popcnt,+fma,+bmi,+aes,+rdrnd,-xsaves,+sse4.1,+sse4.2,+avx2,-avx512er,+sse,+lzcnt,+pclmul,-avx512f,+f16c,+ssse3,+mmx,-pku,+cmov,-xop,+rdseed,+movbe,+hle,+xsaveopt,-sha,+sse2,+sse3,-avx512dq
I0601 12:04:00.749959  1344 GlogAppender.java:137] Logging (re)initialized. Impala: VLOG, All other: INFO
I0601 12:04:00.752050  1344 JniFrontend.java:139] Authorization is 'DISABLED'.
I0601 12:04:00.752928  1344 JniFrontend.java:141] Java Input arguments:
-Djava.library.path=/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/bin/../lib/impala/lib
Java System properties:
java.vm.version:24.65-b04
awt.toolkit:sun.awt.X11.XToolkit
java.vendor.url:http://java.oracle.com/
sun.jnu.encoding:UTF-8
java.vm.info:mixed mode
sun.boot.library.path:/usr/java/jdk1.7.0_67-cloudera/jre/lib/amd64
java.home:/usr/java/jdk1.7.0_67-cloudera/jre
user.dir:/var/run/cloudera-scm-agent/process/288-impala-IMPALAD
java.vm.name:Java HotSpot(TM) 64-Bit Server VM
sun.cpu.isalist:
java.awt.graphicsenv:sun.awt.X11GraphicsEnvironment
sun.os.patch.level:unknown
java.endorsed.dirs:/usr/java/jdk1.7.0_67-cloudera/jre/lib/endorsed
sun.management.compiler:HotSpot 64-Bit Tiered Compilers
user.home:/var/lib/impala
java.runtime.name:Java(TM) SE Runtime Environment
java.io.tmpdir:/tmp
java.awt.printerjob:sun.print.PSPrinterJob
java.version:1.7.0_67
file.encoding.pkg:sun.io
java.library.path:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/bin/../lib/impala/lib
file.separator:/
java.vendor.url.bug:http://bugreport.sun.com/bugreport/
java.specification.vendor:Oracle Corporation
file.encoding:UTF-8
line.separator:

java.vm.specification.version:1.7
java.vm.specification.vendor:Oracle Corporation
user.timezone:Europe/Tiraspol
os.name:Linux
java.vm.vendor:Oracle Corporation
path.separator::
java.ext.dirs:/usr/java/jdk1.7.0_67-cloudera/jre/lib/ext:/usr/java/packages/lib/ext
java.class.path:/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar:/var/lib/impala/*.jar:/usr/share/java/mysql-connector-java.jar:/var/run/cloudera-scm-agent/process/288-impala-IMPALAD/impala-conf:/var/run/cloudera-scm-agent/process/288-impala-IMPALAD/hadoop-conf:/var/run/cloudera-scm-agent/process/288-impala-IMPALAD/hive-conf:/var/run/cloudera-scm-agent/process/288-impala-IMPALAD/hbase-conf:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/libthrift-0.9.0.jar::/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/ST4-4.0.4.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/activation-1.1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/ant-1.5.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/ant-1.9.1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/ant-contrib-1.0b3.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/ant-launcher-1.9.1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/antlr-2.7.7.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/antlr-runtime-3.3.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/aopalliance-1.0.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/apache-log4j-extras-1.2.17.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/apacheds-i18n-2.0.0-M15.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/apacheds-jdbm1-2.0.0-M2.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/api-asn1-api-1.0.0-M20.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/api-util-1.0.0-M20.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/asm-3.1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/asm-commons-3.1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/asm-tree-3.1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/async-1.3.1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/audience-annotations-0.4.0.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/avro.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/aws-java-sdk-bundle-1.11.134.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/azure-data-lake-store-sdk-2.2.3.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/bonecp-0.7.1.RELEASE.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/calcite-avatica-1.0.0-incubating.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/calcite-core-1.0.0-incubating.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/calcite-linq4j-1.0.0-incubating.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/commons-beanutils-1.8.3.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/commons-beanutils-core-1.8.0.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/commons-cli-1.2.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/commons-codec-1.9.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/commons-collections-3.2.2.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/commons-compiler-2.7.6.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/commons-compress-1.4.1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/commons-configuration-1.6.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/commons-daemon-1.0.13.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/commons-dbcp-1.4.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/commons-digester-1.8.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/commons-el-1.0.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/commons-httpclient-3.1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/commons-io-2.4.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/commons-lang-2.6.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/commons-lang3-3.1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/commons-logging-1.1.1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/commons-math-2.2.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/commons-math3-3.1.1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/commons-net-3.1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/commons-pool-1.5.4.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/commons-pool2-2.4.2.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/core-3.1.1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/curator-client-2.7.1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/curator-framework-2.7.1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/curator-recipes-2.7.1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/curator-test-2.11.1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/curator-x-discovery-2.11.1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/datanucleus-api-jdo-3.2.6.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/datanucleus-core-3.2.12.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/datanucleus-rdbms-3.2.12.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/derby-10.10.2.0.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/eigenbase-properties-1.1.4.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/findbugs-annotations-1.3.9-1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/flatbuffers-java-1.6.0.1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/geronimo-annotation_1.0_spec-1.1.1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/geronimo-jaspic_1.0_spec-1.0.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/geronimo-jta_1.1_spec-1.1.1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/groovy-all-2.4.4.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/gson-2.2.4.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/guava-11.0.2.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/guice-3.0.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/guice-servlet-3.0.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/hadoop-annotations.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/hadoop-archives.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/hadoop-auth.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/hadoop-aws.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/hadoop-azure-datalake.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/hadoop-common.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/hadoop-hdfs.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/hadoop-mapreduce-client-common.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/hadoop-mapreduce-client-core.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/hadoop-mapreduce-client-jobclient.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/hadoop-mapreduce-client-shuffle.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/hadoop-yarn-api.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/hadoop-yarn-client.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/hadoop-yarn-common.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/hadoop-yarn-server-applicationhistoryservice.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/hadoop-yarn-server-common.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/hadoop-yarn-server-nodemanager.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/hadoop-yarn-server-resourcemanager.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/hadoop-yarn-server-web-proxy.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/hbase-annotations.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/hbase-client.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/hbase-common.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/hbase-protocol.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/high-scale-lib-1.1.1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/hive-ant.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/hive-beeline.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/hive-cli.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/hive-common.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/hive-exec.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/hive-hbase-handler.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/hive-hcatalog-core.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/hive-hcatalog-server-extensions.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/hive-metastore.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/hive-serde.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/hive-service.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/hive-shims-common.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/hive-shims-scheduler.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/hive-shims.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/hsqldb-1.8.0.10.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/htrace-core-3.0.4.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/htrace-core-3.2.0-incubating.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/htrace-core4-4.0.1-incubating.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/httpclient-4.2.5.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/httpcore-4.2.5.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/impala-data-source-api-1.0-SNAPSHOT.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/impala-frontend-0.1-SNAPSHOT.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/ivy-2.4.0.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/jackson-annotations-2.2.3.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/jackson-core-2.2.3.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/jackson-core-asl-1.9.13.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/jackson-databind-2.2.3.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/jackson-jaxrs-1.8.3.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/jackson-mapper-asl-1.9.13.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/jackson-xc-1.8.3.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/jamon-runtime-2.3.1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/janino-2.7.6.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/jasper-compiler-5.5.23.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/jasper-runtime-5.5.23.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/java-cup-0.11-a-czt02-cdh.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/java-cup-runtime-0.11-a-czt01-cdh.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/java-xmlbuilder-0.4.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/javassist-3.18.1-GA.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/javax.inject-1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/javax.jdo-3.2.0-m3.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/javax.json-1.0.2.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/javax.servlet-2.5.0.v201103041518.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/jaxb-api-2.2.2.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/jaxb-impl-2.2.3-1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/jcodings-1.0.8.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/jdo-api-3.0.1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/jersey-client-1.9.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/jersey-core-1.9.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/jersey-guice-1.9.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/jersey-json-1.9.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/jersey-server-1.9.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/jersey-servlet-1.14.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/jets3t-0.9.0.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/jettison-1.1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/jetty-6.1.26.cloudera.4.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/jetty-all-7.6.0.v20120127.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/jetty-continuation-7.6.16.v20140903.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/jetty-http-7.6.16.v20140903.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/jetty-io-7.6.16.v20140903.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/jetty-security-7.6.16.v20140903.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/jetty-server-7.6.16.v20140903.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/jetty-servlet-7.6.16.v20140903.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/jetty-util-6.1.26.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/jetty-util-7.6.16.v20140903.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/jline-2.12.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/jms-1.1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/joda-time-2.8.1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/joni-2.1.2.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/jpam-1.1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/jsch-0.1.42.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/json-simple-1.1.1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/jsp-api-2.1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/jsr305-3.0.0.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/jta-1.1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/kudu-client-1.5.0-cdh5.13.1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/leveldbjni-all-1.8.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/libfb303-0.9.0.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/libgcc_s.so.1:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/libhadoop.so:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/libhadoop.so.1.0.0:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/libhdfs.so:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/libhdfs.so.0.0.0:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/libkudu_client.so.0:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/libkudu_client.so.0.1.0:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/libstdc++.so.6:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/libstdc++.so.6.0.20:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/libthrift-0.9.0.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/log4j-1.2.17.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/logredactor-1.0.3.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/mail-1.4.1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/metrics-core-2.2.0.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/metrics-core-3.0.2.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/metrics-healthchecks-3.0.2.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/metrics-json-3.0.2.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/metrics-jvm-3.0.2.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/metrics-servlets-3.0.2.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/mockito-all-1.9.5.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/netty-3.10.5.Final.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/netty-all-4.0.23.Final.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/okhttp-2.4.0.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/okio-1.4.0.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/opencsv-2.3.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/oro-2.0.8.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/paranamer-2.3.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/parquet-hadoop-bundle.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/pentaho-aggdesigner-algorithm-5.1.5-jhyde.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/postgresql-9.0-801.jdbc4.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/protobuf-java-2.5.0.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/sentry-binding-hive-conf.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/sentry-binding-hive-follower.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/sentry-binding-hive.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/sentry-core-common.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/sentry-core-model-db.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/sentry-core-model-kafka.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/sentry-core-model-search.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/sentry-hdfs-common.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/sentry-policy-common.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/sentry-policy-db.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/sentry-policy-kafka.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/sentry-policy-search.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/sentry-provider-cache.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/sentry-provider-common.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/sentry-provider-db-sh.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/sentry-provider-file.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/servlet-api-2.5.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/shiro-core-1.2.3.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/slf4j-api-1.7.5.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/slf4j-log4j12-1.7.5.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/snappy-java-1.0.4.1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/stax-api-1.0-2.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/stax-api-1.0.1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/stringtemplate-3.2.1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/super-csv-2.2.0.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/transaction-api-1.1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/velocity-1.5.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/xercesImpl-2.9.1.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/xml-apis-1.3.04.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/xmlenc-0.52.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/xz-1.0.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/zookeeper.jar:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/openssl::/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/libcrypto.so:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/libcrypto.so.1.0.0:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/libssl.so:/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/libssl.so.1.0.0::/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar
sun.arch.data.model:64
os.version:2.6.32-696.16.1.el6.x86_64
java.specification.name:Java Platform API Specification
sun.io.unicode.encoding:UnicodeLittle
user.language:en
user.name:impala
os.arch:amd64
java.runtime.version:1.7.0_67-b01
sun.boot.class.path:/usr/java/jdk1.7.0_67-cloudera/jre/lib/resources.jar:/usr/java/jdk1.7.0_67-cloudera/jre/lib/rt.jar:/usr/java/jdk1.7.0_67-cloudera/jre/lib/sunrsasign.jar:/usr/java/jdk1.7.0_67-cloudera/jre/lib/jsse.jar:/usr/java/jdk1.7.0_67-cloudera/jre/lib/jce.jar:/usr/java/jdk1.7.0_67-cloudera/jre/lib/charsets.jar:/usr/java/jdk1.7.0_67-cloudera/jre/lib/jfr.jar:/usr/java/jdk1.7.0_67-cloudera/jre/classes
user.country:US
java.class.version:51.0
java.vendor:Oracle Corporation
java.vm.specification.name:Java Virtual Machine Specification
sun.cpu.endian:little
java.specification.version:1.7
I0601 12:04:01.343031  1344 AllocationFileLoaderService.java:212] Loading allocation file /var/run/cloudera-scm-agent/process/288-impala-IMPALAD/impala-conf/fair-scheduler.xml
I0601 12:04:01.384876  1344 impala-server.cc:1215] Default query options:TQueryOptions {
  01: abort_on_error (bool) = false,
  02: max_errors (i32) = 100,
  03: disable_codegen (bool) = false,
  04: batch_size (i32) = 0,
  05: num_nodes (i32) = 0,
  06: max_scan_range_length (i64) = 0,
  07: num_scanner_threads (i32) = 0,
  08: max_io_buffers (i32) = 0,
  09: allow_unsupported_formats (bool) = false,
  10: default_order_by_limit (i64) = -1,
  11: debug_action (string) = "",
  12: mem_limit (i64) = 0,
  13: abort_on_default_limit_exceeded (bool) = false,
  15: hbase_caching (i32) = 0,
  16: hbase_cache_blocks (bool) = false,
  17: parquet_file_size (i64) = 0,
  18: explain_level (i32) = 1,
  19: sync_ddl (bool) = false,
  23: disable_cached_reads (bool) = false,
  24: disable_outermost_topn (bool) = false,
  25: rm_initial_mem (i64) = 0,
  26: query_timeout_s (i32) = 0,
  28: appx_count_distinct (bool) = false,
  29: disable_unsafe_spills (bool) = false,
  31: exec_single_node_rows_threshold (i32) = 100,
  32: optimize_partition_key_scans (bool) = false,
  33: replica_preference (i32) = 0,
  34: schedule_random_replica (bool) = false,
  35: scan_node_codegen_threshold (i64) = 1800000,
  36: disable_streaming_preaggregations (bool) = false,
  37: runtime_filter_mode (i32) = 2,
  38: runtime_bloom_filter_size (i32) = 1048576,
  39: runtime_filter_wait_time_ms (i32) = 0,
  40: disable_row_runtime_filtering (bool) = false,
  41: max_num_runtime_filters (i32) = 10,
  42: parquet_annotate_strings_utf8 (bool) = false,
  43: parquet_fallback_schema_resolution (i32) = 0,
  45: s3_skip_insert_staging (bool) = true,
  46: runtime_filter_min_size (i32) = 1048576,
  47: runtime_filter_max_size (i32) = 16777216,
  48: prefetch_mode (i32) = 1,
  49: strict_mode (bool) = false,
  50: scratch_limit (i64) = -1,
  51: enable_expr_rewrites (bool) = true,
  52: decimal_v2 (bool) = false,
  53: parquet_dictionary_filtering (bool) = true,
  54: parquet_array_resolution (i32) = 2,
  55: parquet_read_statistics (bool) = true,
  56: default_join_distribution_mode (i32) = 0,
  57: disable_codegen_rows_threshold (i32) = 50000,
  58: default_spillable_buffer_size (i64) = 2097152,
  59: min_spillable_buffer_size (i64) = 65536,
  60: max_row_size (i64) = 524288,
}
I0601 12:04:01.385177  1460 RequestPoolService.java:150] Loading Llama configuration: /var/run/cloudera-scm-agent/process/288-impala-IMPALAD/impala-conf/llama-site.xml
W0601 12:04:01.491346  1344 UserGroupInformation.java:1920] PriviledgedActionException as:impala (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
W0601 12:04:01.493019  1344 Client.java:713] Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
W0601 12:04:01.493314  1344 UserGroupInformation.java:1920] PriviledgedActionException as:impala (auth:KERBEROS) cause:java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
I0601 12:04:01.506008  1344 status.cc:122] Could not read the root directory at hdfs://hadoop-mgmt01.STAT.RO:8020. Error was:
Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "hadoop-srv01.STAT.RO/192.168.128.11"; destination host is: "hadoop-mgmt01.STAT.RO":8020;
    @           0x83e789  impala::Status::Status()
    @           0xa79206  impala::Frontend::ValidateSettings()
    @           0xaa0124  impala::ImpalaServer::ImpalaServer()
    @           0xaa18d6  impala::CreateImpalaServer()
    @           0xb186d7  ImpaladMain()
    @           0x7d9423  main
    @     0x7f3814795d1d  __libc_start_main
    @           0x80c2dd  (unknown)
E0601 12:04:01.506031  1344 impala-server.cc:282] Could not read the root directory at hdfs://hadoop-mgmt01.STAT.RO:8020. Error was:
Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "hadoop-srv01.STAT.RO/192.168.128.11"; destination host is: "hadoop-mgmt01.STAT.RO":8020;
E0601 12:04:01.506062  1344 impala-server.cc:285] Aborting Impala Server startup due to improper configuration. Impalad exiting.

avatar
Contributor

@vvinaga It looks like it cannot talk to the HDFS NameNode from the logs. Could you check if HDFS is configured correctly to use Kerberos?

avatar
New Contributor

HI,

 

we are now facing same kerberos authentication failure issue due to not using all below encryption types while generating the keytab.
Because, now IT infrastructure/AD team not allowed to use below weak encryption types while generating keytab as it's denied by corporate policy. And we are only allowed to use below string encryptions.

Could you please advise for the alternate solutions for this authentication issue?

 

Weak encryptions:
DES-CBC-CRC | DES-CBC-MD5 | RC4-HMAC-NT

Strong encryptions:
AES256-SHA1 | AES128-SHA1

 

Thanks in Adv,

Sanjay

avatar
New Contributor

HI,

we are now facing same kerberos authentication failure issue due to not using all below encryption types while generating the keytab.
Because, now IT infrastructure/AD team not allowed to use below weak encryption types while generating keytab as it's being denied by corporate policy. And we are only allowed to use below strong encryptions.

Could you please advise for the alternate solutions for this authentication issue?

Weak encryptions:
DES-CBC-CRC | DES-CBC-MD5 | RC4-HMAC-NT

Strong encryptions:
AES256-SHA1 | AES128-SHA1