kerberos authentication failure: GSSAPI Failure: gss_accept_sec_context

[#1]

Environment : CDH 5.3.0 Parcels + +kerberos security(MIT kerberos version 5)

Cloudera Manager -> enable Kerberos -> HDFS(ok) -> YARN (MR2 Included)(ok) -> Hive(ok) -> Impala (error)

	
Using internal kerberos principal "impala/master01.thadoop@THADOOP"
	
Internal communication is authenticated with Kerberos
	
Registering impala/master01.thadoop@THADOOP, keytab file /var/run/cloudera-scm-agent/process/210-impala-STATESTORE/impala.keytab
	
Waiting for Kerberos ticket for principal: impala/master01.thadoop@THADOOP

Kerberos ticket granted to impala/master01.thadoop@THADOOP

Using external kerberos principal "impala/master01.thadoop@THADOOP"
	
External communication is authenticated with Kerberos
	
statestored version 2.1.0-cdh5 RELEASE (build e48c2b48c53ea9601b8f47a39373aa83ff7ca6e2)
Built on Tue, 16 Dec 2014 19:25:34 PST
	
Using hostname: master01.thadoop
	
Flags (see also /varz are on debug webserver):
--catalog_service_port=26000
--load_catalog_in_background=true
--num_metadata_loading_threads=16
--sentry_config=
--disable_optimization_passes=false
--dump_ir=false
--opt_module=
--print_llvm_ir_instruction_count=false
--unopt_module=
--abort_on_config_error=true
--be_port=22000
--be_principal=
--compact_catalog_topic=false
--disable_mem_pools=false
--enable_process_lifetime_heap_profiling=false
--heap_profile_dir=
--hostname=master01.thadoop
--keytab_file=/var/run/cloudera-scm-agent/process/210-impala-STATESTORE/impala.keytab
--krb5_conf=
--krb5_debug_file=
--mem_limit=80%
--principal=impala/master01.thadoop@THADOOP
--log_filename=statestored
--redirect_stdout_stderr=true
--data_source_batch_size=1024
--exchg_node_buffer_size_bytes=10485760
--enable_partitioned_aggregation=true
--enable_partitioned_hash_join=true
--enable_probe_side_filtering=true
--skip_lzo_version_check=false
--max_row_batches=0
--debug_disable_streaming_gzip=false
--enable_phj_probe_side_filtering=true
--enable_ldap_auth=false
--kerberos_reinit_interval=60
--ldap_allow_anonymous_binds=false
--ldap_baseDN=
--ldap_bind_pattern=
--ldap_ca_certificate=
--ldap_domain=
--ldap_manual_config=false
--ldap_passwords_in_clear_ok=false
--ldap_tls=false
--ldap_uri=
--sasl_path=/usr/lib/sasl2:/usr/lib64/sasl2:/usr/local/lib/sasl2:/usr/lib/x86_64-linux-gnu/sasl2
--rpc_cnxn_attempts=10
--rpc_cnxn_retry_interval_ms=2000
--disk_spill_encryption=false
--insert_inherit_permissions=false
--max_free_io_buffers=128
--min_buffer_size=1024
--num_disks=0
--num_threads_per_disk=0
--read_size=8388608
--catalog_service_host=localhost
--cgroup_hierarchy_path=
--enable_rm=false
--enable_webserver=true
--llama_addresses=
--llama_callback_port=28000
--llama_host=
--llama_max_request_attempts=5
--llama_port=15000
--llama_registration_timeout_secs=30
--llama_registration_wait_secs=3
--num_hdfs_worker_threads=16
--resource_broker_cnxn_attempts=1
--resource_broker_cnxn_retry_interval_ms=3000
--resource_broker_recv_timeout=0
--resource_broker_send_timeout=0
--staging_cgroup=impala_staging
--state_store_host=localhost
--state_store_subscriber_port=23000
--use_statestore=true
--local_library_dir=/tmp
--serialize_batch=false
--status_report_interval=5
--num_threads_per_core=3
--scratch_dirs=/tmp
--queue_wait_timeout_ms=60000
--default_pool_max_queued=200
--default_pool_max_requests=200
--default_pool_mem_limit=
--disable_pool_max_requests=false
--disable_pool_mem_limits=false
--fair_scheduler_allocation_path=
--llama_site_path=
--log_mem_usage_interval=0
--authorization_policy_file=
--authorization_policy_provider_class=org.apache.sentry.provider.common.HadoopGroupResourceAuthorizationProvider
--authorized_proxy_user_config=
--load_catalog_at_startup=false
--server_name=
--abort_on_failed_audit_event=true
--audit_event_log_dir=
--be_service_threads=64
--beeswax_port=21000
--cancellation_thread_pool_size=5
--default_query_options=
--fe_service_threads=64
--hs2_port=21050
--idle_query_timeout=0
--idle_session_timeout=0
--local_nodemanager_url=
--log_query_to_file=true
--max_audit_event_log_file_size=5000
--max_profile_log_file_size=5000
--max_result_cache_size=100000
--profile_log_dir=
--query_log_size=25
--ssl_client_ca_certificate=
--ssl_private_key=
--ssl_server_certificate=
--max_vcore_oversubscription_ratio=2.5
--rm_always_use_defaults=false
--rm_default_cpu_vcores=2
--rm_default_memory=4G
--disable_admission_control=true
--require_username=false
--statestore_subscriber_cnxn_attempts=10
--statestore_subscriber_cnxn_retry_interval_ms=3000
--statestore_subscriber_timeout_seconds=30
--state_store_port=24000
--statestore_heartbeat_frequency_ms=1000
--statestore_max_missed_heartbeats=10
--statestore_num_heartbeat_threads=10
--statestore_num_update_threads=10
--statestore_update_frequency_ms=2000
--force_lowercase_usernames=false
--num_cores=0
--web_log_bytes=1048576
--non_impala_java_vlog=0
--periodic_counter_update_period_ms=500
--enable_webserver_doc_root=true
--webserver_authentication_domain=
--webserver_certificate_file=
--webserver_doc_root=/opt/cloudera/parcels/CDH-5.3.0-1.cdh5.3.0.p0.30/lib/impala
--webserver_interface=
--webserver_password_file=
--webserver_port=25010
--flagfile=/var/run/cloudera-scm-agent/process/210-impala-STATESTORE/impala-conf/state_store_flags
--fromenv=
--tryfromenv=
--undefok=
--tab_completion_columns=80
--tab_completion_word=
--help=false
--helpfull=false
--helpmatch=
--helpon=
--helppackage=false
--helpshort=false
--helpxml=false
--version=false
--alsologtoemail=
--alsologtostderr=false
--drop_log_memory=true
--log_backtrace_at=
--log_dir=/var/log/statestore
--log_link=
--log_prefix=true
--logbuflevel=0
--logbufsecs=30
--logbufvlevel=1
--logemaillevel=999
--logmailer=/bin/mail
--logtostderr=false
--max_log_size=200
--minloglevel=0
--stderrthreshold=4
--stop_logging_if_full_disk=false
--symbolize_stacktrace=true
--v=1
--vmodule=
	
Cpu Info:
  Model: QEMU Virtual CPU version 0.14.1
  Cores: 4
  L1 Cache: 32.00 KB
  L2 Cache: 2.00 MB
  L3 Cache: 0
  Hardware Supports:
    popcnt
	
Disk Info: 
  Num disks 1: 
    vda (rotational=true)

	
Physical Memory: 7.69 GB
	
OS version: Linux version 2.6.32-431.el6.x86_64 (mockbuild@c6b8.bsys.dev.centos.org) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-4) (GCC) ) #1 SMP Fri Nov 22 03:15:09 UTC 2013
	
Process ID: 22645
	
Starting webserver on 0.0.0.0:25010
	
Document root: /opt/cloudera/parcels/CDH-5.3.0-1.cdh5.3.0.p0.30/lib/impala
	
Webserver started

ThriftServer 'StatestoreService' started on port: 24000
	
SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Wrong principal in request)
	
TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context
	
SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Wrong principal in request)

TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context
	
SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Wrong principal in request)

TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context
	
Failed to extend Kerberos ticket. Error: Shell cmd: 'kinit -R' exited with an error: ''. Output was: ''. Failure count: 1
	
SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Wrong principal in request)
	
TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context
	
SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Wrong principal in request)
	
TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context
	
SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Wrong principal in request)

TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context
	
SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Wrong principal in request)
	
TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context
	
SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Wrong principal in request)
	
TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context

SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Wrong principal in request)

TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context
	
SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Wrong principal in request)
	
TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context
	
SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Wrong principal in request)
	
TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context
	
SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Wrong principal in request)
	
TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context
	
SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Wrong principal in request)

TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context
	
SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Wr
	
TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context
	
SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Wrong principal in request)

TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context

SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Wrong principal in request)
	
TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context

path : /var/kerberos/krb5kdc/kdc.conf

[kdcdefaults]
kdc_ports = 88
kdc_tcp_ports = 88

[realms]
THADOOP.COM = {
#master_key_type = aes256-cts
acl_file = /var/kerberos/krb5kdc/kadm5.acl
dict_file = /usr/share/dict/words
admin_keytab = /var/kerberos/krb5kdc/kadm5.keytab
supported_enctypes = des3-hmac-sha1:normal arcfour-hmac:normal des-hmac-sha1:normal des-cbc-md5:normal des-cbc-crc:normal des-cbc-crc:v4 des-cbc-crc:afs3
max_life = 30d
max_renewable_life = 30d
default_principal_flags = +renewable, +forwardable
}

path : /etc/krb5.conf

[logging]
default = FILE:/var/log/krb5libs.log
kdc = FILE:/var/log/krb5kdc.log
admin_server = FILE:/var/log/kadmind.log

[libdefaults]
default_realm = THADOOP
dns_lookup_realm = false
dns_lookup_kdc = false
ticket_lifetime = 32d
renew_lifetime = 32d
forwardable = true
renewable = true
udp_preference_limit = 1
default_tgs_enctypes = arcfour-hmac
default_tkt_enctypes = arcfour-hmac

[realms]
THADOOP = {
kdc = kerberos.thadoop
admin_server = kerberos.thadoop
}

[domain_realm]
.thadoop = THADOOP
thadoop = THADOOP

path : /var/kerberos/krb5kdc/kadm5.acl

*/admin@THADOOP *

and...

[root@master01 210-impala-STATESTORE]# klist -e
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: admin/admin@THADOOP

Valid starting Expires Service principal
01/06/15 10:08:42 01/07/15 10:08:42 krbtgt/THADOOP@THADOOP
renew until 01/06/15 10:08:42, Etype (skey, tkt): des3-cbc-sha1, aes256-cts-hmac-sha1-96

============================================================

[root@master01 210-impala-STATESTORE]# pwd
/var/run/cloudera-scm-agent/process/210-impala-STATESTORE
[root@master01 210-impala-STATESTORE]# klist -ket impala.keytab
Keytab name: FILE:impala.keytab
KVNO Timestamp Principal
---- ----------------- --------------------------------------------------------
2 01/06/15 10:13:43 impala/master01.thadoop@THADOOP (aes256-cts-hmac-sha1-96)
2 01/06/15 10:13:43 impala/master01.thadoop@THADOOP (aes128-cts-hmac-sha1-96)
2 01/06/15 10:13:44 impala/master01.thadoop@THADOOP (des3-cbc-sha1)
2 01/06/15 10:13:44 impala/master01.thadoop@THADOOP (arcfour-hmac)

(There is no HTTP.keytab this is normal?)

by the way...

Kerberos Encryption Types : des3-cbc-sha1 (default rc4-hmac) 

Anyone have any suggestions how to resolve this problem?

[#2]

Hi Cibi,

Sorry for the delay. I missed the notification. The kerberos knobs seem correct. Could you attach your catalogd, statestored and some impalad logs?

- Sailesh

[#3]

Hi Sailesh,

Please find the requested log files.

Thanks,

Cibi

Statestore log:

Log file created at: 2016/09/25 01:58:34
Running on machine: statestore_host
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
I0925 01:58:34.314796 2701924 logging.cc:119] stdout will be logged to this file.
E0925 01:58:34.314962 2701924 logging.cc:120] stderr will be logged to this file.
I0925 01:58:34.318999 2701924 authentication.cc:692] Using internal kerberos principal "impala/statestore_host_FQDN@SPARTA.COM"
I0925 01:58:34.319008 2701924 authentication.cc:1027] Internal communication is authenticated with Kerberos
I0925 01:58:34.319118 2701960 authentication.cc:491] Registering impala/statestore_host_FQDN@SPARTA.COM, keytab file /var/run/cloudera-scm-agent/process/1103-impala-STATESTORE/impala.keytab
I0925 01:58:34.319125 2701924 authentication.cc:812] Waiting for Kerberos ticket for principal: impala/statestore_host_FQDN@SPARTA.COM
I0925 01:58:34.537320 2701924 authentication.cc:814] Kerberos ticket granted to impala/statestore_host_FQDN@SPARTA.COM
I0925 01:58:34.537376 2701924 authentication.cc:692] Using external kerberos principal "impala/statestore_host_FQDN@SPARTA.COM"
I0925 01:58:34.537382 2701924 authentication.cc:1043] External communication is authenticated with Kerberos
I0925 01:58:34.537528 2701924 init.cc:158] statestored version 2.3.0-cdh5.5.4 RELEASE (build e65ded24350974ae6b4e475557b358b718fad29e)
Built on Mon, 25 Apr 2016 11:30:47 PST
I0925 01:58:34.537534 2701924 init.cc:159] Using hostname: statestore_host
I0925 01:58:34.538158 2701924 logging.cc:155] Flags (see also /varz are on debug webserver):
--catalog_service_port=26000
--load_catalog_in_background=false
--num_metadata_loading_threads=16
--sentry_config=
--disable_optimization_passes=false
--dump_ir=false
--opt_module_dir=
--print_llvm_ir_instruction_count=false
--unopt_module_dir=
--abort_on_config_error=true
--be_port=22000
--be_principal=
--compact_catalog_topic=false
--disable_mem_pools=false
--enable_process_lifetime_heap_profiling=false
--heap_profile_dir=
--hostname=statestore_host
--keytab_file=/var/run/cloudera-scm-agent/process/1103-impala-STATESTORE/impala.keytab
--krb5_conf=
--krb5_debug_file=
--mem_limit=80%
--principal=impala/statestore_host_FQDN@SPARTA.COM
--redaction_rules_file=
--max_log_files=10
--log_filename=statestored
--redirect_stdout_stderr=true
--data_source_batch_size=1024
--exchg_node_buffer_size_bytes=10485760
--enable_partitioned_aggregation=true
--enable_partitioned_hash_join=true
--enable_probe_side_filtering=true
--enable_quadratic_probing=true
--skip_lzo_version_check=false
--convert_legacy_hive_parquet_utc_timestamps=false
--max_page_header_size=8388608
--max_row_batches=0
--debug_disable_streaming_gzip=false
--enable_phj_probe_side_filtering=true
--enable_ldap_auth=false
--internal_principals_whitelist=hdfs
--kerberos_reinit_interval=60
--ldap_allow_anonymous_binds=false
--ldap_baseDN=
--ldap_bind_pattern=
--ldap_ca_certificate=
--ldap_domain=
--ldap_manual_config=false
--ldap_passwords_in_clear_ok=false
--ldap_tls=false
--ldap_uri=
--sasl_path=/usr/lib/sasl2:/usr/lib64/sasl2:/usr/local/lib/sasl2:/usr/lib/x86_64-linux-gnu/sasl2
--rpc_cnxn_attempts=10
--rpc_cnxn_retry_interval_ms=2000
--disk_spill_encryption=false
--insert_inherit_permissions=false
--max_cached_file_handles=0
--max_free_io_buffers=128
--min_buffer_size=1024
--num_disks=0
--num_remote_hdfs_io_threads=8
--num_s3_io_threads=16
--num_threads_per_disk=0
--read_size=8388608
--catalog_service_host=localhost
--cgroup_hierarchy_path=
--enable_rm=false
--enable_webserver=true
--llama_addresses=
--llama_callback_port=28000
--llama_host=
--llama_max_request_attempts=5
--llama_port=15000
--llama_registration_timeout_secs=30
--llama_registration_wait_secs=3
--num_hdfs_worker_threads=16
--resource_broker_cnxn_attempts=1
--resource_broker_cnxn_retry_interval_ms=3000
--resource_broker_recv_timeout=0
--resource_broker_send_timeout=0
--staging_cgroup=impala_staging
--state_store_host=localhost
--state_store_subscriber_port=23000
--use_statestore=true
--local_library_dir=/tmp
--serialize_batch=false
--status_report_interval=5
--num_threads_per_core=3
--use_local_tz_for_unix_timestamp_conversions=false
--scratch_dirs=/tmp
--queue_wait_timeout_ms=60000
--default_pool_max_queued=200
--default_pool_max_requests=200
--default_pool_mem_limit=
--disable_pool_max_requests=false
--disable_pool_mem_limits=false
--fair_scheduler_allocation_path=
--llama_site_path=
--log_mem_usage_interval=0
--authorization_policy_file=
--authorization_policy_provider_class=org.apache.sentry.provider.common.HadoopGroupResourceAuthorizationProvider
--authorized_proxy_user_config=
--authorized_proxy_user_config_delimiter=,
--load_catalog_at_startup=false
--server_name=
--abort_on_failed_audit_event=true
--abort_on_failed_lineage_event=true
--audit_event_log_dir=
--be_service_threads=64
--beeswax_port=21000
--cancellation_thread_pool_size=5
--default_query_options=
--fe_service_threads=64
--hs2_port=21050
--idle_query_timeout=0
--idle_session_timeout=0
--lineage_event_log_dir=
--local_nodemanager_url=
--log_query_to_file=true
--max_audit_event_log_file_size=5000
--max_lineage_log_file_size=5000
--max_profile_log_file_size=5000
--max_result_cache_size=100000
--profile_log_dir=
--query_log_size=25
--ssl_client_ca_certificate=
--ssl_private_key=
--ssl_private_key_password_cmd=
--ssl_server_certificate=
--max_vcore_oversubscription_ratio=2.5
--rm_always_use_defaults=false
--rm_default_cpu_vcores=2
--rm_default_memory=4G
--disable_admission_control=true
--require_username=false
--statestore_subscriber_cnxn_attempts=10
--statestore_subscriber_cnxn_retry_interval_ms=3000
--statestore_subscriber_timeout_seconds=30
--state_store_port=24000
--statestore_heartbeat_frequency_ms=1000
--statestore_heartbeat_tcp_timeout_seconds=3
--statestore_max_missed_heartbeats=10
--statestore_num_heartbeat_threads=10
--statestore_num_update_threads=10
--statestore_update_frequency_ms=2000
--statestore_update_tcp_timeout_seconds=300
--force_lowercase_usernames=false
--num_cores=0
--web_log_bytes=1048576
--non_impala_java_vlog=0
--periodic_counter_update_period_ms=500
--enable_webserver_doc_root=true
--webserver_authentication_domain=
--webserver_certificate_file=
--webserver_doc_root=/vol_01/cloudera/parcels/CDH-5.5.4-1.cdh5.5.4.p0.9/lib/impala
--webserver_interface=
--webserver_password_file=
--webserver_port=25010
--webserver_private_key_file=
--webserver_private_key_password_cmd=
--flagfile=/var/run/cloudera-scm-agent/process/1103-impala-STATESTORE/impala-conf/state_store_flags
--fromenv=
--tryfromenv=
--undefok=
--tab_completion_columns=80
--tab_completion_word=
--help=false
--helpfull=false
--helpmatch=
--helpon=
--helppackage=false
--helpshort=false
--helpxml=false
--version=false
--alsologtoemail=
--alsologtostderr=false
--drop_log_memory=true
--log_backtrace_at=
--log_dir=/vol_01/log/statestore
--log_link=
--log_prefix=true
--logbuflevel=0
--logbufsecs=30
--logbufvlevel=1
--logemaillevel=999
--logmailer=/bin/mail
--logtostderr=false
--max_log_size=200
--minloglevel=0
--stderrthreshold=4
--stop_logging_if_full_disk=false
--symbolize_stacktrace=true
--v=1
--vmodule=
I0925 01:58:34.538220 2701924 init.cc:164] Cpu Info:
Model: Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz
Cores: 4
L1 Cache: 32.00 KB
L2 Cache: 256.00 KB
L3 Cache: 25.00 MB
Hardware Supports:
ssse3
sse4_1
sse4_2
popcnt
I0925 01:58:34.538228 2701924 init.cc:165] Disk Info:
Num disks 3:
sda (rotational=true)
sdb (rotational=true)
dm- (rotational=true)

I0925 01:58:34.538235 2701924 init.cc:166] Physical Memory: 15.58 GB
I0925 01:58:34.538240 2701924 init.cc:167] OS version: Linux version 2.6.32-573.18.1.el6.x86_64 (mockbuild@x86-010.build.bos.redhat.com) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-16) (GCC) ) #1 SMP Wed Jan 6 11:20:49 EST 2016
I0925 01:58:34.538242 2701924 init.cc:168] Process ID: 2701924
I0925 01:58:34.538297 2701924 webserver.cc:210] Starting webserver on 0.0.0.0:25010
I0925 01:58:34.538313 2701924 webserver.cc:224] Document root: /vol_01/cloudera/parcels/CDH-5.5.4-1.cdh5.5.4.p0.9/lib/impala
I0925 01:58:34.538424 2701924 webserver.cc:309] Webserver started
I0925 01:58:34.559240 2701924 thrift-server.cc:431] ThriftServer 'StatestoreService' started on port: 24000
I0925 01:58:36.257709 2701960 authentication.cc:519] Successfully renewed Keberos ticket
E0925 01:58:37.180516 2701998 authentication.cc:158] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request)
I0925 01:58:37.181058 2701998 thrift-util.cc:109] TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context
E0925 01:58:37.646294 2701998 authentication.cc:158] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request)
I0925 01:58:37.646749 2701998 thrift-util.cc:109] TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context
E0925 01:58:39.232471 2701998 authentication.cc:158] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request)
I0925 01:58:39.232897 2701998 thrift-util.cc:109] TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context
E0925 01:58:39.568882 2701998 authentication.cc:158] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request)
I0925 01:58:39.569471 2701998 thrift-util.cc:109] TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context
E0925 01:58:40.184850 2701998 authentication.cc:158] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request)
I0925 01:58:40.185300 2701998 thrift-util.cc:109] TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context
E0925 01:58:40.650048 2701998 authentication.cc:158] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request)
I0925 01:58:40.650457 2701998 thrift-util.cc:109] TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context
E0925 01:58:42.236927 2701998 authentication.cc:158] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request)
I0925 01:58:42.237365 2701998 thrift-util.cc:109] TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context
E0925 01:58:42.573323 2701998 authentication.cc:158] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request)
I0925 01:58:42.575258 2701998 thrift-util.cc:109] TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context
E0925 01:58:47.326885 2701998 authentication.cc:158] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request)
I0925 01:58:47.327370 2701998 thrift-util.cc:109] TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context
E0925 01:58:48.607534 2701998 authentication.cc:158] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request)
I0925 01:58:48.608216 2701998 thrift-util.cc:109] TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context
E0925 01:58:50.330967 2701998 authentication.cc:158] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request)
I0925 01:58:50.331411 2701998 thrift-util.cc:109] TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context
E0925 01:58:50.694953 2701998 authentication.cc:158] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request)
I0925 01:58:50.695441 2701998 thrift-util.cc:109] TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context
E0925 01:58:51.611749 2701998 authentication.cc:158] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request)
I0925 01:58:51.612180 2701998 thrift-util.cc:109] TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context
E0925 01:58:52.205765 2701998 authentication.cc:158] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request)
I0925 01:58:52.206214 2701998 thrift-util.cc:109] TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context
E0925 01:58:53.698994 2701998 authentication.cc:158] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request)
I0925 01:58:53.699483 2701998 thrift-util.cc:109] TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context
E0925 01:58:55.210412 2701998 authentication.cc:158] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request)
I0925 01:58:55.210867 2701998 thrift-util.cc:109] TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context
E0925 01:58:58.699170 2701998 authentication.cc:158] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request)
I0925 01:58:58.699609 2701998 thrift-util.cc:109] TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context
E0925 01:59:00.513659 2701998 authentication.cc:158] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request)
I0925 01:59:00.514102 2701998 thrift-util.cc:109] TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context
E0925 01:59:01.703227 2701998 authentication.cc:158] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request)
I0925 01:59:01.703634 2701998 thrift-util.cc:109] TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context
E0925 01:59:03.486690 2701998 authentication.cc:158] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request)
I0925 01:59:03.487344 2701998 thrift-util.cc:109] TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context
E0925 01:59:03.517756 2701998 authentication.cc:158] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request)
I0925 01:59:03.518329 2701998 thrift-util.cc:109] TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context
E0925 01:59:05.393232 2701998 authentication.cc:158] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request)
I0925 01:59:05.393954 2701998 thrift-util.cc:109] TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context
E0925 01:59:06.490890 2701998 authentication.cc:158] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request)
I0925 01:59:06.491515 2701998 thrift-util.cc:109] TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context
E0925 01:59:08.398248 2701998 authentication.cc:158] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request)
I0925 01:59:08.398841 2701998 thrift-util.cc:109] TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context
E0925 01:59:11.103200 2701998 authentication.cc:158] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request)
I0925 01:59:11.103942 2701998 thrift-util.cc:109] TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context
E0925 01:59:13.687938 2701998 authentication.cc:158] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request)
I0925 01:59:13.688544 2701998 thrift-util.cc:109] TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context
E0925 01:59:14.109560 2701998 authentication.cc:158] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request)
I0925 01:59:14.110250 2701998 thrift-util.cc:109] TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context
E0925 01:59:16.692235 2701998 authentication.cc:158] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request)
I0925 01:59:16.692840 2701998 thrift-util.cc:109] TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context
E0925 01:59:17.763322 2701998 authentication.cc:158] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request)
I0925 01:59:17.763895 2701998 thrift-util.cc:109] TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context
E0925 01:59:19.819659 2701998 authentication.cc:158] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request)
I0925 01:59:19.820860 2701998 thrift-util.cc:109] TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context
E0925 01:59:20.768082 2701998 authentication.cc:158] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request)
I0925 01:59:20.768653 2701998 thrift-util.cc:109] TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context
E0925 01:59:22.824905 2701998 authentication.cc:158] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request)
I0925 01:59:22.825522 2701998 thrift-util.cc:109] TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context

[#4]

Adding Catalogd and Impalad logs:

Catalogd Log:

Log file created at: 2016/09/25 01:59:08
Running on machine: Catalod_host
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
I0925 01:59:08.209684 16058 logging.cc:119] stdout will be logged to this file.
E0925 01:59:08.209851 16058 logging.cc:120] stderr will be logged to this file.
I0925 01:59:08.214717 16058 authentication.cc:692] Using internal kerberos principal "impala/Catalod_host_FQDN@SPARTA.COM"
I0925 01:59:08.214726 16058 authentication.cc:1027] Internal communication is authenticated with Kerberos
I0925 01:59:08.214850 16107 authentication.cc:491] Registering impala/Catalod_host_FQDN@SPARTA.COM, keytab file /var/run/cloudera-scm-agent/process/1102-impala-CATALOGSERVER/impala.keytab
I0925 01:59:08.214891 16058 authentication.cc:812] Waiting for Kerberos ticket for principal: impala/Catalod_host_FQDN@SPARTA.COM
I0925 01:59:08.436725 16058 authentication.cc:814] Kerberos ticket granted to impala/Catalod_host_FQDN@SPARTA.COM
I0925 01:59:08.436791 16058 authentication.cc:692] Using external kerberos principal "impala/Catalod_host_FQDN@SPARTA.COM"
I0925 01:59:08.436797 16058 authentication.cc:1043] External communication is authenticated with Kerberos
I0925 01:59:08.437022 16058 init.cc:158] catalogd version 2.3.0-cdh5.5.4 RELEASE (build e65ded24350974ae6b4e475557b358b718fad29e)
Built on Mon, 25 Apr 2016 11:30:47 PST
I0925 01:59:08.437029 16058 init.cc:159] Using hostname: Catalod_host
I0925 01:59:08.437994 16058 logging.cc:155] Flags (see also /varz are on debug webserver):
--catalog_service_port=26000
--load_catalog_in_background=true
--num_metadata_loading_threads=16
--sentry_config=
--disable_optimization_passes=false
--dump_ir=false
--opt_module_dir=
--print_llvm_ir_instruction_count=false
--unopt_module_dir=
--abort_on_config_error=true
--be_port=22000
--be_principal=
--compact_catalog_topic=false
--disable_mem_pools=false
--enable_process_lifetime_heap_profiling=false
--heap_profile_dir=
--hostname=Catalod_host
--keytab_file=/var/run/cloudera-scm-agent/process/1102-impala-CATALOGSERVER/impala.keytab
--krb5_conf=
--krb5_debug_file=
--mem_limit=80%
--principal=impala/Catalod_host_FQDN@SPARTA.COM
--redaction_rules_file=
--max_log_files=10
--log_filename=catalogd
--redirect_stdout_stderr=true
--data_source_batch_size=1024
--exchg_node_buffer_size_bytes=10485760
--enable_partitioned_aggregation=true
--enable_partitioned_hash_join=true
--enable_probe_side_filtering=true
--enable_quadratic_probing=true
--skip_lzo_version_check=false
--convert_legacy_hive_parquet_utc_timestamps=false
--max_page_header_size=8388608
--max_row_batches=0
--debug_disable_streaming_gzip=false
--enable_phj_probe_side_filtering=true
--enable_ldap_auth=false
--internal_principals_whitelist=hdfs
--kerberos_reinit_interval=60
--ldap_allow_anonymous_binds=false
--ldap_baseDN=
--ldap_bind_pattern=
--ldap_ca_certificate=
--ldap_domain=
--ldap_manual_config=false
--ldap_passwords_in_clear_ok=false
--ldap_tls=false
--ldap_uri=
--sasl_path=/usr/lib/sasl2:/usr/lib64/sasl2:/usr/local/lib/sasl2:/usr/lib/x86_64-linux-gnu/sasl2
--rpc_cnxn_attempts=10
--rpc_cnxn_retry_interval_ms=2000
--disk_spill_encryption=false
--insert_inherit_permissions=false
--max_cached_file_handles=0
--max_free_io_buffers=128
--min_buffer_size=1024
--num_disks=0
--num_remote_hdfs_io_threads=8
--num_s3_io_threads=16
--num_threads_per_disk=0
--read_size=8388608
--catalog_service_host=localhost
--cgroup_hierarchy_path=
--enable_rm=false
--enable_webserver=true
--llama_addresses=
--llama_callback_port=28000
--llama_host=
--llama_max_request_attempts=5
--llama_port=15000
--llama_registration_timeout_secs=30
--llama_registration_wait_secs=3
--num_hdfs_worker_threads=16
--resource_broker_cnxn_attempts=1
--resource_broker_cnxn_retry_interval_ms=3000
--resource_broker_recv_timeout=0
--resource_broker_send_timeout=0
--staging_cgroup=impala_staging
--state_store_host=statestore_host_FQDN
--state_store_subscriber_port=23020
--use_statestore=true
--local_library_dir=/tmp
--serialize_batch=false
--status_report_interval=5
--num_threads_per_core=3
--use_local_tz_for_unix_timestamp_conversions=false
--scratch_dirs=/tmp
--queue_wait_timeout_ms=60000
--default_pool_max_queued=200
--default_pool_max_requests=200
--default_pool_mem_limit=
--disable_pool_max_requests=false
--disable_pool_mem_limits=false
--fair_scheduler_allocation_path=
--llama_site_path=
--log_mem_usage_interval=0
--authorization_policy_file=
--authorization_policy_provider_class=org.apache.sentry.provider.common.HadoopGroupResourceAuthorizationProvider
--authorized_proxy_user_config=
--authorized_proxy_user_config_delimiter=,
--load_catalog_at_startup=false
--server_name=
--abort_on_failed_audit_event=true
--abort_on_failed_lineage_event=true
--audit_event_log_dir=
--be_service_threads=64
--beeswax_port=21000
--cancellation_thread_pool_size=5
--default_query_options=
--fe_service_threads=64
--hs2_port=21050
--idle_query_timeout=0
--idle_session_timeout=0
--lineage_event_log_dir=
--local_nodemanager_url=
--log_query_to_file=true
--max_audit_event_log_file_size=5000
--max_lineage_log_file_size=5000
--max_profile_log_file_size=5000
--max_result_cache_size=100000
--profile_log_dir=
--query_log_size=25
--ssl_client_ca_certificate=
--ssl_private_key=
--ssl_private_key_password_cmd=
--ssl_server_certificate=
--max_vcore_oversubscription_ratio=2.5
--rm_always_use_defaults=false
--rm_default_cpu_vcores=2
--rm_default_memory=4G
--disable_admission_control=true
--require_username=false
--statestore_subscriber_cnxn_attempts=10
--statestore_subscriber_cnxn_retry_interval_ms=3000
--statestore_subscriber_timeout_seconds=30
--state_store_port=24000
--statestore_heartbeat_frequency_ms=1000
--statestore_heartbeat_tcp_timeout_seconds=3
--statestore_max_missed_heartbeats=10
--statestore_num_heartbeat_threads=10
--statestore_num_update_threads=10
--statestore_update_frequency_ms=2000
--statestore_update_tcp_timeout_seconds=300
--force_lowercase_usernames=false
--num_cores=0
--web_log_bytes=1048576
--non_impala_java_vlog=0
--periodic_counter_update_period_ms=500
--enable_webserver_doc_root=true
--webserver_authentication_domain=
--webserver_certificate_file=
--webserver_doc_root=/vol_01/cloudera/parcels/CDH-5.5.4-1.cdh5.5.4.p0.9/lib/impala
--webserver_interface=
--webserver_password_file=
--webserver_port=25020
--webserver_private_key_file=
--webserver_private_key_password_cmd=
--flagfile=/var/run/cloudera-scm-agent/process/1102-impala-CATALOGSERVER/impala-conf/catalogserver_flags
--fromenv=
--tryfromenv=
--undefok=
--tab_completion_columns=80
--tab_completion_word=
--help=false
--helpfull=false
--helpmatch=
--helpon=
--helppackage=false
--helpshort=false
--helpxml=false
--version=false
--alsologtoemail=
--alsologtostderr=false
--drop_log_memory=true
--log_backtrace_at=
--log_dir=/vol_01/log/catalogd
--log_link=
--log_prefix=true
--logbuflevel=0
--logbufsecs=30
--logbufvlevel=1
--logemaillevel=999
--logmailer=/bin/mail
--logtostderr=false
--max_log_size=200
--minloglevel=0
--stderrthreshold=4
--stop_logging_if_full_disk=false
--symbolize_stacktrace=true
--v=1
--vmodule=
I0925 01:59:08.438065 16058 init.cc:164] Cpu Info:
Model: Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz
Cores: 4
L1 Cache: 32.00 KB
L2 Cache: 256.00 KB
L3 Cache: 25.00 MB
Hardware Supports:
ssse3
sse4_1
sse4_2
popcnt
I0925 01:59:08.438073 16058 init.cc:165] Disk Info:
Num disks 3:
sda (rotational=true)
sdb (rotational=true)
dm- (rotational=true)

I0925 01:59:08.438082 16058 init.cc:166] Physical Memory: 15.58 GB
I0925 01:59:08.438087 16058 init.cc:167] OS version: Linux version 2.6.32-573.18.1.el6.x86_64 (mockbuild@x86-010.build.bos.redhat.com) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-16) (GCC) ) #1 SMP Wed Jan 6 11:20:49 EST 2016
I0925 01:59:08.438091 16058 init.cc:168] Process ID: 16058
I0925 01:59:09.604663 16058 webserver.cc:210] Starting webserver on 0.0.0.0:25020
I0925 01:59:09.604691 16058 webserver.cc:224] Document root: /vol_01/cloudera/parcels/CDH-5.5.4-1.cdh5.5.4.p0.9/lib/impala
I0925 01:59:09.604823 16058 webserver.cc:309] Webserver started
I0925 01:59:09.640769 16058 GlogAppender.java:123] Logging initialized. Impala: VLOG, All other: INFO
I0925 01:59:09.641646 16058 JniCatalog.java:89] Java Version Info: Java(TM) SE Runtime Environment (1.7.0_25-b15)
I0925 01:59:10.155648 16107 authentication.cc:519] Successfully renewed Keberos ticket
I0925 01:59:10.170464 16058 MetaStoreClientPool.java:55] Creating MetaStoreClient. Pool Size = 0
I0925 01:59:10.182169 16058 HiveMetaStoreClient.java:376] Trying to connect to metastore with URI thrift://Catalod_host_FQDN:9083
I0925 01:59:10.565619 16058 HiveMetaStoreClient.java:421] Opened a connection to metastore, current connections: 1
I0925 01:59:10.566120 16058 HiveMetaStoreClient.java:473] Connected to metastore.
I0925 01:59:10.566236 16058 MetaStoreClientPool.java:55] Creating MetaStoreClient. Pool Size = 1
I0925 01:59:10.566793 16058 HiveMetaStoreClient.java:376] Trying to connect to metastore with URI thrift://Catalod_host_FQDN:9083
I0925 01:59:10.578852 16058 HiveMetaStoreClient.java:421] Opened a connection to metastore, current connections: 2
I0925 01:59:10.579704 16058 HiveMetaStoreClient.java:473] Connected to metastore.
I0925 01:59:10.579993 16058 MetaStoreClientPool.java:55] Creating MetaStoreClient. Pool Size = 2
I0925 01:59:10.580477 16058 HiveMetaStoreClient.java:376] Trying to connect to metastore with URI thrift://Catalod_host_FQDN:9083
I0925 01:59:10.592774 16058 HiveMetaStoreClient.java:421] Opened a connection to metastore, current connections: 3
I0925 01:59:10.593262 16058 HiveMetaStoreClient.java:473] Connected to metastore.
I0925 01:59:10.593372 16058 MetaStoreClientPool.java:55] Creating MetaStoreClient. Pool Size = 3
I0925 01:59:10.595728 16058 HiveMetaStoreClient.java:376] Trying to connect to metastore with URI thrift://Catalod_host_FQDN:9083
I0925 01:59:10.606948 16058 HiveMetaStoreClient.java:421] Opened a connection to metastore, current connections: 4
I0925 01:59:10.607393 16058 HiveMetaStoreClient.java:473] Connected to metastore.
I0925 01:59:10.607458 16058 MetaStoreClientPool.java:55] Creating MetaStoreClient. Pool Size = 4
I0925 01:59:10.607743 16058 HiveMetaStoreClient.java:376] Trying to connect to metastore with URI thrift://Catalod_host_FQDN:9083
I0925 01:59:10.614889 16058 HiveMetaStoreClient.java:421] Opened a connection to metastore, current connections: 5
I0925 01:59:10.615327 16058 HiveMetaStoreClient.java:473] Connected to metastore.
I0925 01:59:10.834429 16058 MetaStoreClientPool.java:55] Creating MetaStoreClient. Pool Size = 4
I0925 01:59:10.834760 16058 HiveMetaStoreClient.java:376] Trying to connect to metastore with URI thrift://Catalod_host_FQDN:9083
I0925 01:59:10.842993 16058 HiveMetaStoreClient.java:421] Opened a connection to metastore, current connections: 6
I0925 01:59:10.843364 16058 HiveMetaStoreClient.java:473] Connected to metastore.
I0925 01:59:10.871893 16058 HiveMetaStoreClient.java:502] Closed a connection to metastore, current connections: 5
I0925 01:59:10.873522 16131 TableLoadingMgr.java:282] Loading next table. Remaining items in queue: 0
I0925 01:59:10.873736 16132 TableLoadingMgr.java:282] Loading next table. Remaining items in queue: 2
I0925 01:59:10.873814 16133 TableLoadingMgr.java:282] Loading next table. Remaining items in queue: 1
I0925 01:59:10.873906 16134 TableLoadingMgr.java:282] Loading next table. Remaining items in queue: 0
I0925 01:59:10.873977 16135 TableLoadingMgr.java:282] Loading next table. Remaining items in queue: 0
I0925 01:59:10.874081 16136 TableLoadingMgr.java:282] Loading next table. Remaining items in queue: 0
I0925 01:59:10.874209 16137 TableLoadingMgr.java:282] Loading next table. Remaining items in queue: 0
I0925 01:59:10.874346 16138 TableLoadingMgr.java:282] Loading next table. Remaining items in queue: 2
I0925 01:59:10.880722 16058 statestore-subscriber.cc:179] Starting statestore subscriber
I0925 01:59:10.882082 16149 TableLoader.java:60] Loading metadata for: core.product
I0925 01:59:10.882200 16149 MetaStoreClientPool.java:55] Creating MetaStoreClient. Pool Size = 3
I0925 01:59:10.882367 16152 TableLoader.java:60] Loading metadata for: core.student
I0925 01:59:10.882501 16149 HiveMetaStoreClient.java:376] Trying to connect to metastore with URI thrift://Catalod_host_FQDN:9083
I0925 01:59:10.882658 16153 TableLoader.java:60] Loading metadata for: core.product_bucketed
I0925 01:59:10.886121 16058 thrift-server.cc:431] ThriftServer 'StatestoreSubscriber' started on port: 23020
I0925 01:59:10.886129 16058 statestore-subscriber.cc:190] Registering with statestore
I0925 01:59:10.889982 16149 HiveMetaStoreClient.java:421] Opened a connection to metastore, current connections: 6
I0925 01:59:10.890362 16149 HiveMetaStoreClient.java:473] Connected to metastore.
I0925 01:59:10.890480 16153 MetaStoreClientPool.java:55] Creating MetaStoreClient. Pool Size = 1
I0925 01:59:10.890766 16153 HiveMetaStoreClient.java:376] Trying to connect to metastore with URI thrift://Catalod_host_FQDN:9083
I0925 01:59:10.891439 16154 TableLoader.java:60] Loading metadata for: core.c
I0925 01:59:10.891670 16155 TableLoader.java:60] Loading metadata for: core.b
I0925 01:59:10.891916 16139 TableLoadingMgr.java:282] Loading next table. Remaining items in queue: 0
I0925 01:59:10.892029 16156 TableLoader.java:60] Loading metadata for: core.product1
I0925 01:59:10.892103 16140 TableLoadingMgr.java:282] Loading next table. Remaining items in queue: 0
I0925 01:59:10.892426 16157 TableLoader.java:60] Loading metadata for: core.a
I0925 01:59:10.898161 16153 HiveMetaStoreClient.java:421] Opened a connection to metastore, current connections: 7
I0925 01:59:10.898551 16153 HiveMetaStoreClient.java:473] Connected to metastore.
I0925 01:59:10.898646 16152 MetaStoreClientPool.java:55] Creating MetaStoreClient. Pool Size = 0
I0925 01:59:10.898927 16152 HiveMetaStoreClient.java:376] Trying to connect to metastore with URI thrift://Catalod_host_FQDN:9083
I0925 01:59:10.900521 16158 TableLoader.java:60] Loading metadata for: core.test1
I0925 01:59:10.902216 16160 TableLoader.java:60] Loading metadata for: core.test
I0925 01:59:10.903240 16159 TableLoader.java:60] Loading metadata for: core.product2b
I0925 01:59:10.906101 16152 HiveMetaStoreClient.java:421] Opened a connection to metastore, current connections: 8
I0925 01:59:10.906488 16152 HiveMetaStoreClient.java:473] Connected to metastore.
I0925 01:59:10.906679 16159 MetaStoreClientPool.java:55] Creating MetaStoreClient. Pool Size = 0
I0925 01:59:10.906947 16159 HiveMetaStoreClient.java:376] Trying to connect to metastore with URI thrift://Catalod_host_FQDN:9083
I0925 01:59:10.914718 16159 HiveMetaStoreClient.java:421] Opened a connection to metastore, current connections: 9
I0925 01:59:10.915083 16159 HiveMetaStoreClient.java:473] Connected to metastore.
I0925 01:59:10.915179 16160 MetaStoreClientPool.java:55] Creating MetaStoreClient. Pool Size = 0
I0925 01:59:10.915460 16160 HiveMetaStoreClient.java:376] Trying to connect to metastore with URI thrift://Catalod_host_FQDN:9083
I0925 01:59:10.923107 16160 HiveMetaStoreClient.java:421] Opened a connection to metastore, current connections: 10
I0925 01:59:10.923576 16160 HiveMetaStoreClient.java:473] Connected to metastore.
I0925 01:59:10.923689 16158 MetaStoreClientPool.java:55] Creating MetaStoreClient. Pool Size = 0
I0925 01:59:10.923995 16158 HiveMetaStoreClient.java:376] Trying to connect to metastore with URI thrift://Catalod_host_FQDN:9083
I0925 01:59:10.931267 16158 HiveMetaStoreClient.java:421] Opened a connection to metastore, current connections: 11
I0925 01:59:10.931654 16158 HiveMetaStoreClient.java:473] Connected to metastore.
I0925 01:59:10.931756 16157 MetaStoreClientPool.java:55] Creating MetaStoreClient. Pool Size = 0
I0925 01:59:10.932042 16157 HiveMetaStoreClient.java:376] Trying to connect to metastore with URI thrift://Catalod_host_FQDN:9083
I0925 01:59:10.939201 16157 HiveMetaStoreClient.java:421] Opened a connection to metastore, current connections: 12
I0925 01:59:10.939621 16157 HiveMetaStoreClient.java:473] Connected to metastore.
I0925 01:59:10.939684 16150 catalog-server.cc:313] Publishing update: DATABASE:core@1
I0925 01:59:10.939714 16150 catalog-server.cc:313] Publishing update: TABLE:core.test@10
I0925 01:59:10.939726 16156 MetaStoreClientPool.java:55] Creating MetaStoreClient. Pool Size = 0
I0925 01:59:10.939764 16150 catalog-server.cc:313] Publishing update: TABLE:core.product1@6
I0925 01:59:10.939771 16150 catalog-server.cc:313] Publishing update: TABLE:core.a@2
I0925 01:59:10.939776 16150 catalog-server.cc:313] Publishing update: TABLE:core.b@3
I0925 01:59:10.939782 16150 catalog-server.cc:313] Publishing update: TABLE:core.product@5
I0925 01:59:10.939787 16150 catalog-server.cc:313] Publishing update: TABLE:core.c@4
I0925 01:59:10.939792 16150 catalog-server.cc:313] Publishing update: TABLE:core.product2b@7
I0925 01:59:10.939796 16150 catalog-server.cc:313] Publishing update: TABLE:core.test1@11
I0925 01:59:10.939802 16150 catalog-server.cc:313] Publishing update: TABLE:core.product_bucketed@8
I0925 01:59:10.939807 16150 catalog-server.cc:313] Publishing update: TABLE:core.student@9
I0925 01:59:10.939812 16150 catalog-server.cc:313] Publishing update: DATABASE:default@12
I0925 01:59:10.939820 16150 catalog-server.cc:313] Publishing update: CATALOG:d71b5b5808594fcd:afaa46f809705c83@12
I0925 01:59:10.940039 16156 HiveMetaStoreClient.java:376] Trying to connect to metastore with URI thrift://Catalod_host_FQDN:9083
I0925 01:59:10.946593 16156 HiveMetaStoreClient.java:421] Opened a connection to metastore, current connections: 13
I0925 01:59:10.946977 16156 HiveMetaStoreClient.java:473] Connected to metastore.
I0925 01:59:10.947360 16155 MetaStoreClientPool.java:55] Creating MetaStoreClient. Pool Size = 0
I0925 01:59:10.947641 16155 HiveMetaStoreClient.java:376] Trying to connect to metastore with URI thrift://Catalod_host_FQDN:9083
I0925 01:59:10.954646 16155 HiveMetaStoreClient.java:421] Opened a connection to metastore, current connections: 14
I0925 01:59:10.955044 16155 HiveMetaStoreClient.java:473] Connected to metastore.
I0925 01:59:10.955150 16154 MetaStoreClientPool.java:55] Creating MetaStoreClient. Pool Size = 0
I0925 01:59:10.955973 16154 HiveMetaStoreClient.java:376] Trying to connect to metastore with URI thrift://Catalod_host_FQDN:9083
I0925 01:59:10.964249 16154 HiveMetaStoreClient.java:421] Opened a connection to metastore, current connections: 15
I0925 01:59:10.964658 16154 HiveMetaStoreClient.java:473] Connected to metastore.
I0925 01:59:11.043257 16158 HdfsTable.java:1013] load table: core.test1
I0925 01:59:11.044705 16157 HdfsTable.java:1013] load table: core.a
I0925 01:59:11.044816 16159 HdfsTable.java:1013] load table: core.product2b
I0925 01:59:11.044934 16152 HdfsTable.java:1013] load table: core.student
I0925 01:59:11.045044 16153 HdfsTable.java:1013] load table: core.product_bucketed
I0925 01:59:11.045115 16154 HdfsTable.java:1013] load table: core.c
I0925 01:59:11.045204 16155 HdfsTable.java:1013] load table: core.b
I0925 01:59:11.045284 16156 HdfsTable.java:1013] load table: core.product1
I0925 01:59:11.045358 16149 HdfsTable.java:1013] load table: core.product
I0925 01:59:11.055538 16160 HdfsTable.java:1013] load table: core.test
I0925 01:59:11.106195 16058 status.cc:112] Couldn't open transport for statestore_host_FQDN:24000 (No more data to read.)
@ 0x7aeb93 (unknown)
@ 0xbc5cb4 (unknown)
@ 0xbc5d93 (unknown)
@ 0x9a4bb6 (unknown)
@ 0x9a5891 (unknown)
@ 0xa6ca83 (unknown)
@ 0xa6ea5c (unknown)
@ 0x782405 (unknown)
@ 0x77b8a6 (unknown)
@ 0x3192e1ed5d (unknown)
@ 0x77b1b9 (unknown)
I0925 01:59:11.106235 16058 thrift-client.cc:55] Unable to connect to statestore_host_FQDN:24000
I0925 01:59:11.106240 16058 thrift-client.cc:61] (Attempt 1 of 10)
I0925 01:59:11.111660 16157 Table.java:155] Loading column stats for table: a
I0925 01:59:11.114917 16155 Table.java:155] Loading column stats for table: b
I0925 01:59:11.116050 16160 Table.java:155] Loading column stats for table: test
I0925 01:59:11.116945 16154 Table.java:155] Loading column stats for table: c
I0925 01:59:11.118154 16158 Table.java:155] Loading column stats for table: test1
I0925 01:59:11.119752 16149 Table.java:155] Loading column stats for table: product
I0925 01:59:11.123488 16156 Table.java:155] Loading column stats for table: product1
I0925 01:59:11.124166 16153 Table.java:155] Loading column stats for table: product_bucketed
I0925 01:59:11.124459 16152 Table.java:155] Loading column stats for table: student
I0925 01:59:11.124534 16159 Table.java:155] Loading column stats for table: product2b
I0925 01:59:11.910172 16160 HiveMetaStoreClient.java:502] Closed a connection to metastore, current connections: 14
I0925 01:59:11.910339 16152 HiveMetaStoreClient.java:502] Closed a connection to metastore, current connections: 13
I0925 01:59:11.910528 16158 HiveMetaStoreClient.java:502] Closed a connection to metastore, current connections: 12
I0925 01:59:11.910676 16155 HiveMetaStoreClient.java:502] Closed a connection to metastore, current connections: 11
I0925 01:59:11.910926 16154 HiveMetaStoreClient.java:502] Closed a connection to metastore, current connections: 10
I0925 01:59:11.911077 16157 HiveMetaStoreClient.java:502] Closed a connection to metastore, current connections: 9
I0925 01:59:11.911568 16159 HiveMetaStoreClient.java:502] Closed a connection to metastore, current connections: 8
I0925 01:59:11.911725 16153 HiveMetaStoreClient.java:502] Closed a connection to metastore, current connections: 7
I0925 01:59:11.912005 16156 HiveMetaStoreClient.java:502] Closed a connection to metastore, current connections: 6
I0925 01:59:14.106725 16058 client-cache.cc:80] ReopenClient(): re-creating client for statestore_host_FQDN:24000
I0925 01:59:14.110769 16058 status.cc:112] Couldn't open transport for statestore_host_FQDN:24000 (No more data to read.)
@ 0x7aeb93 (unknown)
@ 0xbc5cb4 (unknown)
@ 0xbc5d93 (unknown)
@ 0x9a4bb6 (unknown)
@ 0x9a5254 (unknown)
@ 0xa6d5c3 (unknown)
@ 0xa6ea5c (unknown)
@ 0x782405 (unknown)
@ 0x77b8a6 (unknown)
@ 0x3192e1ed5d (unknown)
@ 0x77b1b9 (unknown)
I0925 01:59:14.110781 16058 thrift-client.cc:55] Unable to connect to statestore_host_FQDN:24000
I0925 01:59:14.110785 16058 thrift-client.cc:61] (Attempt 1 of 10)
I0925 01:59:16.646400 16149 HiveMetaStoreClient.java:502] Closed a connection to metastore, current connections: 5
I0925 01:59:17.111428 16058 status.cc:45] RPC Error: No more data to read.
@ 0x7ae3c6 (unknown)
@ 0xa6d724 (unknown)
@ 0xa6ea5c (unknown)
@ 0x782405 (unknown)
@ 0x77b8a6 (unknown)
@ 0x3192e1ed5d (unknown)
@ 0x77b1b9 (unknown)
I0925 01:59:17.111462 16058 statestore-subscriber.cc:196] statestore registration unsuccessful: RPC Error: No more data to read.
E0925 01:59:17.111650 16058 catalogd-main.cc:76] RPC Error: No more data to read.

Impalad Log:

Log file created at: 2016/09/25 01:59:12
Running on machine: Impalad_host
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
I0925 01:59:12.879591 2702404 logging.cc:119] stdout will be logged to this file.
E0925 01:59:12.879758 2702404 logging.cc:120] stderr will be logged to this file.
I0925 01:59:12.884112 2702404 authentication.cc:692] Using internal kerberos principal "impala/Impalad_host_FQDN@SPARTA.COM"
I0925 01:59:12.884125 2702404 authentication.cc:1027] Internal communication is authenticated with Kerberos
I0925 01:59:12.884238 2702458 authentication.cc:491] Registering impala/Impalad_host_FQDN@SPARTA.COM, keytab file /var/run/cloudera-scm-agent/process/1100-impala-IMPALAD/impala.keytab
I0925 01:59:12.884248 2702404 authentication.cc:812] Waiting for Kerberos ticket for principal: impala/Impalad_host_FQDN@SPARTA.COM
I0925 01:59:13.102747 2702404 authentication.cc:814] Kerberos ticket granted to impala/Impalad_host_FQDN@SPARTA.COM
I0925 01:59:13.102812 2702404 authentication.cc:692] Using external kerberos principal "impala/Impalad_host_FQDN@SPARTA.COM"
I0925 01:59:13.102818 2702404 authentication.cc:1043] External communication is authenticated with Kerberos
I0925 01:59:13.103026 2702404 init.cc:158] impalad version 2.3.0-cdh5.5.4 RELEASE (build e65ded24350974ae6b4e475557b358b718fad29e)
Built on Mon, 25 Apr 2016 11:30:47 PST
I0925 01:59:13.103032 2702404 init.cc:159] Using hostname: Impalad_host_FQDN
I0925 01:59:13.103698 2702404 logging.cc:155] Flags (see also /varz are on debug webserver):
--catalog_service_port=26000
--load_catalog_in_background=false
--num_metadata_loading_threads=16
--sentry_config=
--disable_optimization_passes=false
--dump_ir=false
--opt_module_dir=
--print_llvm_ir_instruction_count=false
--unopt_module_dir=
--abort_on_config_error=true
--be_port=22000
--be_principal=
--compact_catalog_topic=false
--disable_mem_pools=false
--enable_process_lifetime_heap_profiling=false
--heap_profile_dir=
--hostname=Impalad_host_FQDN
--keytab_file=/var/run/cloudera-scm-agent/process/1100-impala-IMPALAD/impala.keytab
--krb5_conf=
--krb5_debug_file=
--mem_limit=4328521728
--principal=impala/Impalad_host_FQDN@SPARTA.COM
--redaction_rules_file=
--max_log_files=10
--log_filename=impalad
--redirect_stdout_stderr=true
--data_source_batch_size=1024
--exchg_node_buffer_size_bytes=10485760
--enable_partitioned_aggregation=true
--enable_partitioned_hash_join=true
--enable_probe_side_filtering=true
--enable_quadratic_probing=true
--skip_lzo_version_check=false
--convert_legacy_hive_parquet_utc_timestamps=false
--max_page_header_size=8388608
--max_row_batches=0
--debug_disable_streaming_gzip=false
--enable_phj_probe_side_filtering=true
--enable_ldap_auth=false
--internal_principals_whitelist=hdfs
--kerberos_reinit_interval=60
--ldap_allow_anonymous_binds=false
--ldap_baseDN=
--ldap_bind_pattern=
--ldap_ca_certificate=
--ldap_domain=
--ldap_manual_config=false
--ldap_passwords_in_clear_ok=false
--ldap_tls=false
--ldap_uri=
--sasl_path=/usr/lib/sasl2:/usr/lib64/sasl2:/usr/local/lib/sasl2:/usr/lib/x86_64-linux-gnu/sasl2
--rpc_cnxn_attempts=10
--rpc_cnxn_retry_interval_ms=2000
--disk_spill_encryption=false
--insert_inherit_permissions=false
--max_cached_file_handles=0
--max_free_io_buffers=128
--min_buffer_size=1024
--num_disks=0
--num_remote_hdfs_io_threads=8
--num_s3_io_threads=16
--num_threads_per_disk=0
--read_size=8388608
--catalog_service_host=catalaogd_host_FQDN
--cgroup_hierarchy_path=
--enable_rm=false
--enable_webserver=true
--llama_addresses=
--llama_callback_port=28000
--llama_host=
--llama_max_request_attempts=5
--llama_port=15000
--llama_registration_timeout_secs=30
--llama_registration_wait_secs=3
--num_hdfs_worker_threads=16
--resource_broker_cnxn_attempts=1
--resource_broker_cnxn_retry_interval_ms=3000
--resource_broker_recv_timeout=0
--resource_broker_send_timeout=0
--staging_cgroup=impala_staging
--state_store_host=Impalad_host_FQDN
--state_store_subscriber_port=23000
--use_statestore=true
--local_library_dir=/vol_01/lib/impala/udfs
--serialize_batch=false
--status_report_interval=5
--num_threads_per_core=3
--use_local_tz_for_unix_timestamp_conversions=false
--scratch_dirs=/vol_01/impala/impalad
--queue_wait_timeout_ms=60000
--default_pool_max_queued=200
--default_pool_max_requests=200
--default_pool_mem_limit=
--disable_pool_max_requests=false
--disable_pool_mem_limits=false
--fair_scheduler_allocation_path=
--llama_site_path=
--log_mem_usage_interval=0
--authorization_policy_file=
--authorization_policy_provider_class=org.apache.sentry.provider.common.HadoopGroupResourceAuthorizationProvider
--authorized_proxy_user_config=
--authorized_proxy_user_config_delimiter=,
--load_catalog_at_startup=false
--server_name=
--abort_on_failed_audit_event=true
--abort_on_failed_lineage_event=true
--audit_event_log_dir=
--be_service_threads=64
--beeswax_port=21000
--cancellation_thread_pool_size=5
--default_query_options=
--fe_service_threads=64
--hs2_port=21050
--idle_query_timeout=0
--idle_session_timeout=0
--lineage_event_log_dir=/vol_01/log/impalad/lineage
--local_nodemanager_url=
--log_query_to_file=true
--max_audit_event_log_file_size=5000
--max_lineage_log_file_size=5000
--max_profile_log_file_size=5000
--max_result_cache_size=100000
--profile_log_dir=
--query_log_size=25
--ssl_client_ca_certificate=
--ssl_private_key=
--ssl_private_key_password_cmd=
--ssl_server_certificate=
--max_vcore_oversubscription_ratio=2.5
--rm_always_use_defaults=false
--rm_default_cpu_vcores=2
--rm_default_memory=4G
--disable_admission_control=true
--require_username=false
--statestore_subscriber_cnxn_attempts=10
--statestore_subscriber_cnxn_retry_interval_ms=3000
--statestore_subscriber_timeout_seconds=30
--state_store_port=24000
--statestore_heartbeat_frequency_ms=1000
--statestore_heartbeat_tcp_timeout_seconds=3
--statestore_max_missed_heartbeats=10
--statestore_num_heartbeat_threads=10
--statestore_num_update_threads=10
--statestore_update_frequency_ms=2000
--statestore_update_tcp_timeout_seconds=300
--force_lowercase_usernames=false
--num_cores=0
--web_log_bytes=1048576
--non_impala_java_vlog=0
--periodic_counter_update_period_ms=500
--enable_webserver_doc_root=true
--webserver_authentication_domain=
--webserver_certificate_file=
--webserver_doc_root=/vol_01/cloudera/parcels/CDH-5.5.4-1.cdh5.5.4.p0.9/lib/impala
--webserver_interface=
--webserver_password_file=
--webserver_port=25000
--webserver_private_key_file=
--webserver_private_key_password_cmd=
--flagfile=/var/run/cloudera-scm-agent/process/1100-impala-IMPALAD/impala-conf/impalad_flags
--fromenv=
--tryfromenv=
--undefok=
--tab_completion_columns=80
--tab_completion_word=
--help=false
--helpfull=false
--helpmatch=
--helpon=
--helppackage=false
--helpshort=false
--helpxml=false
--version=false
--alsologtoemail=
--alsologtostderr=false
--drop_log_memory=true
--log_backtrace_at=
--log_dir=/vol_01/log/impalad
--log_link=
--log_prefix=true
--logbuflevel=0
--logbufsecs=30
--logbufvlevel=1
--logemaillevel=999
--logmailer=/bin/mail
--logtostderr=false
--max_log_size=200
--minloglevel=0
--stderrthreshold=4
--stop_logging_if_full_disk=false
--symbolize_stacktrace=true
--v=1
--vmodule=
I0925 01:59:13.103770 2702404 init.cc:164] Cpu Info:
Model: Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz
Cores: 4
L1 Cache: 32.00 KB
L2 Cache: 256.00 KB
L3 Cache: 25.00 MB
Hardware Supports:
ssse3
sse4_1
sse4_2
popcnt
I0925 01:59:13.103778 2702404 init.cc:165] Disk Info:
Num disks 3:
sda (rotational=true)
sdb (rotational=true)
dm- (rotational=true)

I0925 01:59:13.103785 2702404 init.cc:166] Physical Memory: 15.58 GB
I0925 01:59:13.103790 2702404 init.cc:167] OS version: Linux version 2.6.32-573.18.1.el6.x86_64 (mockbuild@x86-010.build.bos.redhat.com) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-16) (GCC) ) #1 SMP Wed Jan 6 11:20:49 EST 2016
I0925 01:59:13.103793 2702404 init.cc:168] Process ID: 2702404
I0925 01:59:14.821723 2702458 authentication.cc:519] Successfully renewed Keberos ticket
I0925 01:59:15.032460 2702404 hbase-table-scanner.cc:157] Detected HBase version >= 0.95.2
I0925 01:59:15.142060 2702404 GlogAppender.java:123] Logging initialized. Impala: VLOG, All other: INFO
I0925 01:59:15.144978 2702404 JniFrontend.java:130] Authorization is 'DISABLED'.
I0925 01:59:15.145108 2702404 JniFrontend.java:132] Java Version Info: Java(TM) SE Runtime Environment (1.7.0_67-b01)
I0925 01:59:15.734531 2702404 simple-scheduler.cc:103] Admission control is disabled.
I0925 01:59:15.735497 2702404 impala-server.cc:1091] Default query options:TQueryOptions {
01: abort_on_error (bool) = false,
02: max_errors (i32) = 0,
03: disable_codegen (bool) = false,
04: batch_size (i32) = 0,
05: num_nodes (i32) = 0,
06: max_scan_range_length (i64) = 0,
07: num_scanner_threads (i32) = 0,
08: max_io_buffers (i32) = 0,
09: allow_unsupported_formats (bool) = false,
10: default_order_by_limit (i64) = -1,
11: debug_action (string) = "",
12: mem_limit (i64) = 0,
13: abort_on_default_limit_exceeded (bool) = false,
15: hbase_caching (i32) = 0,
16: hbase_cache_blocks (bool) = false,
17: parquet_file_size (i64) = 0,
18: explain_level (i32) = 1,
19: sync_ddl (bool) = false,
23: disable_cached_reads (bool) = false,
24: disable_outermost_topn (bool) = false,
25: rm_initial_mem (i64) = 0,
26: query_timeout_s (i32) = 0,
28: appx_count_distinct (bool) = false,
29: disable_unsafe_spills (bool) = false,
31: exec_single_node_rows_threshold (i32) = 100,
}
I0925 01:59:17.514686 2702404 tmp-file-mgr.cc:106] Using scratch directory /vol_01/impala/impalad/impala-scratch on disk 2
I0925 01:59:17.514786 2702404 simple-logger.cc:76] Logging to: /vol_01/log/impalad/profiles//impala_profile_log_1.1-1474786757514
I0925 01:59:17.514906 2702404 impala-server.cc:485] Event logging is disabled
I0925 01:59:17.514976 2702404 simple-logger.cc:76] Logging to: /vol_01/log/impalad/lineage/impala_lineage_log_1.0-1474786757514
I0925 01:59:17.530464 2702404 impala-server.cc:1722] Impala Beeswax Service listening on 21000
I0925 01:59:17.531194 2702404 impala-server.cc:1744] Impala HiveServer2 Service listening on 21050
I0925 01:59:17.531886 2702404 impala-server.cc:1764] ImpalaInternalService listening on 22000
I0925 01:59:17.532097 2702404 thrift-server.cc:431] ThriftServer 'backend' started on port: 22000
I0925 01:59:17.532107 2702404 exec-env.cc:296] Starting global services
I0925 01:59:17.543001 2702404 exec-env.cc:383] Using global memory limit: 4.03 GB
I0925 01:59:17.544988 2702404 webserver.cc:210] Starting webserver on 0.0.0.0:25000
I0925 01:59:17.544999 2702404 webserver.cc:224] Document root: /vol_01/cloudera/parcels/CDH-5.5.4-1.cdh5.5.4.p0.9/lib/impala
I0925 01:59:17.545121 2702404 webserver.cc:309] Webserver started
I0925 01:59:17.545142 2702404 simple-scheduler.cc:182] Starting simple scheduler
I0925 01:59:17.545276 2702404 simple-scheduler.cc:230] Simple-scheduler using Statestore_Host_IP as IP address
I0925 01:59:17.545287 2702404 statestore-subscriber.cc:179] Starting statestore subscriber
I0925 01:59:17.546177 2702404 thrift-server.cc:431] ThriftServer 'StatestoreSubscriber' started on port: 23000
I0925 01:59:17.546185 2702404 statestore-subscriber.cc:190] Registering with statestore
I0925 01:59:17.764664 2702404 status.cc:112] Couldn't open transport for Impalad_host_FQDN:24000 (No more data to read.)
@ 0x79f523 (unknown)
@ 0xbcdc24 (unknown)
@ 0xbcdd03 (unknown)
@ 0x9a00f6 (unknown)
@ 0x9a0dd1 (unknown)
@ 0xa6e6b3 (unknown)
@ 0xa7068c (unknown)
@ 0x963e66 (unknown)
@ 0x77b57c (unknown)
@ 0x313081ed5d (unknown)
@ 0x77b1b9 (unknown)
I0925 01:59:17.764691 2702404 thrift-client.cc:55] Unable to connect to Impalad_host_FQDN:24000
I0925 01:59:17.764695 2702404 thrift-client.cc:61] (Attempt 1 of 10)
I0925 01:59:20.764950 2702404 client-cache.cc:80] ReopenClient(): re-creating client for Impalad_host_FQDN:24000
I0925 01:59:20.769177 2702404 status.cc:112] Couldn't open transport for Impalad_host_FQDN:24000 (No more data to read.)
@ 0x79f523 (unknown)
@ 0xbcdc24 (unknown)
@ 0xbcdd03 (unknown)
@ 0x9a00f6 (unknown)
@ 0x9a0794 (unknown)
@ 0xa6f1f3 (unknown)
@ 0xa7068c (unknown)
@ 0x963e66 (unknown)
@ 0x77b57c (unknown)
@ 0x313081ed5d (unknown)
@ 0x77b1b9 (unknown)
I0925 01:59:20.769208 2702404 thrift-client.cc:55] Unable to connect to Impalad_host_FQDN:24000
I0925 01:59:20.769212 2702404 thrift-client.cc:61] (Attempt 1 of 10)
I0925 01:59:23.769857 2702404 status.cc:45] RPC Error: No more data to read.
@ 0x79ed56 (unknown)
@ 0xa6f354 (unknown)
@ 0xa7068c (unknown)
@ 0x963e66 (unknown)
@ 0x77b57c (unknown)
@ 0x313081ed5d (unknown)
@ 0x77b1b9 (unknown)
I0925 01:59:23.769894 2702404 statestore-subscriber.cc:196] statestore registration unsuccessful: RPC Error: No more data to read.
E0925 01:59:23.770056 2702404 impalad-main.cc:80] Impalad services did not start correctly, exiting. Error: RPC Error: No more data to read.
State Store Subscriber did not start up.

[#5]

Hi Sailesh,

Did you had time to look at the logs file details you requested?

Thanks,

Cibi

[#6]

Hi Cibi,

    Is your problem solved?

[#7]

Hi Miying,

No i'm still facing the problem.

Thanks,

Cibi

[#8]

Hi Cibi:

This seems the mismatch of encryption types between clients and the KDC server. her is main configuration steps:
1. Stop the cluster through CM
2. Go to CM --> Administration --> Kerberos --> 'Kerberos Encryption Types',
delete default encryption type:"rc4-hmac"(important!), then add the following encryption types:

arcfour-hmac
aes128-cts
aes256-cts
des-cbc-crc
des-cbc-md5

3. Do redeploy krb5.conf through CM, and restart the cluster.

[#9]

Hi Miying,

That resolved the issue. Thanks for the help! 🙂

Regards,

Cibi

[#10]

Its realted to krb5.conf file. you have to add few param in krb5.conf file to find KCM file. look below. its worked for me.

http://tekzak.com/forum/viewtopic.php?f=2&t=258743&sid=d2ba062d74fe403b7cbaea9c5786b6ac

[#11]

You have to add few param for krb5.conf related to KCM if you are using MAC.

http://tekzak.com/forum/viewtopic.php?f=2&t=258743&sid=d2ba062d74fe403b7cbaea9c5786b6ac