Impala service does not start after enabling kerberos on the cluster

[#1]

Hello ,

After kerberizing the cluster impala daemons fails to start . I see Impala state store is up but when looked at the Role logs it shows GSS execption . 

Tried to go through some of the community help as below, made sure the encryption types are fine but that did not help resolve the issue

One more observation below , the error doesnot give details as to what the error could be related to and just calls out saying Unspecified GSS faliure .

"TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information ()"

reference :

https://community.cloudera.com/t5/Interactive-Short-cycle-SQL/kerberos-authentication-failure-GSSAPI...

snap shot from CM:

===============

Error  

====

Apr 19, 9:45:16.664 AM INFO logging.cc:117
stdout will be logged to this file.
Apr 19, 9:45:16.664 AM ERROR logging.cc:118
stderr will be logged to this file.
Apr 19, 9:45:16.664 AM INFO minidump.cc:199
Setting minidump size limit to 20971520.
Apr 19, 9:45:16.664 AM INFO atomicops-internals-x86.cc:93
vendor GenuineIntel family 6 model 15 sse2 1 cmpxchg16b 1
Apr 19, 9:45:16.673 AM INFO authentication.cc:678
Using internal kerberos principal "impala/CM-PR-BGD-M02.telemovil.com.sv@TELEMOVIL.COM.SV"
Apr 19, 9:45:16.673 AM INFO authentication.cc:1013
Internal communication is authenticated with Kerberos
Apr 19, 9:45:16.673 AM INFO authentication.cc:798
Waiting for Kerberos ticket for principal: impala/CM-PR-BGD-M02.telemovil.com.sv@TELEMOVIL.COM.SV
Apr 19, 9:45:16.673 AM INFO authentication.cc:494
Registering impala/CM-PR-BGD-M02.telemovil.com.sv@TELEMOVIL.COM.SV, keytab file /var/run/cloudera-scm-agent/process/804-impala-STATESTORE/impala.keytab
Apr 19, 9:45:16.780 AM INFO authentication.cc:800
Kerberos ticket granted to impala/CM-PR-BGD-M02.telemovil.com.sv@TELEMOVIL.COM.SV
Apr 19, 9:45:16.780 AM INFO authentication.cc:678
Using external kerberos principal "impala/CM-PR-BGD-M02.telemovil.com.sv@TELEMOVIL.COM.SV"
Apr 19, 9:45:16.780 AM INFO authentication.cc:1029
External communication is authenticated with Kerberos
Apr 19, 9:45:16.780 AM INFO init.cc:201
statestored version 2.6.0-cdh5.8.4 RELEASE (build 207450616f75adbe082a4c2e1145a2384da83fa6)
Built on Mon, 06 Feb 2017 14:33:04 PST
Apr 19, 9:45:16.780 AM INFO init.cc:202
Using hostname: CM-PR-BGD-M02
Apr 19, 9:45:16.780 AM INFO logging.cc:153
Flags (see also /varz are on debug webserver):
--catalog_service_port=26000
--load_catalog_in_background=false
--num_metadata_loading_threads=16
--sentry_config=
--asm_module_dir=
--disable_optimization_passes=false
--dump_ir=false
--opt_module_dir=
--perf_map=false
--print_llvm_ir_instruction_count=false
--unopt_module_dir=
--abort_on_config_error=true
--be_port=22000
--be_principal=
--compact_catalog_topic=false
--disable_mem_pools=false
--enable_accept_queue_server=true
--enable_process_lifetime_heap_profiling=false
--heap_profile_dir=
--hostname=CM-PR-BGD-M02
--inc_stats_size_limit_bytes=209715200
--keytab_file=/var/run/cloudera-scm-agent/process/804-impala-STATESTORE/impala.keytab
--krb5_conf=
--krb5_debug_file=
--load_auth_to_local_rules=false
--max_minidumps=9
--mem_limit=80%
--minidump_path=/var/log/impala-minidumps/statestored
--minidump_size_limit_hint_kb=20480
--principal=impala/CM-PR-BGD-M02.telemovil.com.sv@TELEMOVIL.COM.SV
--redaction_rules_file=
--max_log_files=10
--pause_monitor_sleep_time_ms=500
--pause_monitor_warn_threshold_ms=10000
--log_filename=statestored
--redirect_stdout_stderr=true
--data_source_batch_size=1024
--exchg_node_buffer_size_bytes=10485760
--enable_partitioned_aggregation=true
--enable_partitioned_hash_join=true
--enable_probe_side_filtering=true
--enable_quadratic_probing=true
--skip_lzo_version_check=false
--convert_legacy_hive_parquet_utc_timestamps=false
--max_page_header_size=8388608
--parquet_min_filter_reject_ratio=0.10000000000000001
--max_row_batches=0
--runtime_filter_wait_time_ms=1000
--suppress_unknown_disk_id_warnings=false
--kudu_max_row_batches=0
--kudu_scanner_keep_alive_period_us=15000000
--kudu_scanner_keep_alive_period_sec=15
--kudu_scanner_timeout_sec=60
--pick_only_leaders_for_tests=false
--kudu_session_timeout_seconds=60
--enable_phj_probe_side_filtering=true
--accepted_cnxn_queue_depth=10000
--enable_ldap_auth=false
--internal_principals_whitelist=hdfs
--kerberos_reinit_interval=60
--ldap_allow_anonymous_binds=false
--ldap_baseDN=
--ldap_bind_pattern=
--ldap_ca_certificate=
--ldap_domain=
--ldap_manual_config=false
--ldap_passwords_in_clear_ok=false
--ldap_tls=false
--ldap_uri=
--sasl_path=
--rpc_cnxn_attempts=10
--rpc_cnxn_retry_interval_ms=2000
--disk_spill_encryption=false
--insert_inherit_permissions=false
--datastream_sender_timeout_ms=120000
--max_cached_file_handles=0
--max_free_io_buffers=128
--min_buffer_size=1024
--num_disks=0
--num_remote_hdfs_io_threads=8
--num_s3_io_threads=16
--num_threads_per_disk=0
--read_size=8388608
--backend_client_connection_num_retries=3
--backend_client_rpc_timeout_ms=300000
--catalog_client_connection_num_retries=3
--catalog_client_rpc_timeout_ms=0
--catalog_service_host=localhost
--cgroup_hierarchy_path=
--coordinator_rpc_threads=12
--enable_rm=false
--enable_webserver=true
--llama_addresses=
--llama_callback_port=28000
--llama_host=
--llama_max_request_attempts=5
--llama_port=15000
--llama_registration_timeout_secs=30
--llama_registration_wait_secs=3
--num_hdfs_worker_threads=16
--resource_broker_cnxn_attempts=1
--resource_broker_cnxn_retry_interval_ms=3000
--resource_broker_recv_timeout=0
--resource_broker_send_timeout=0
--staging_cgroup=impala_staging
--state_store_host=localhost
--state_store_subscriber_port=23000
--use_statestore=true
--s3a_access_key_cmd=
--s3a_secret_key_cmd=
--local_library_dir=/tmp
--serialize_batch=false
--status_report_interval=5
--max_filter_error_rate=0.75
--num_threads_per_core=3
--use_local_tz_for_unix_timestamp_conversions=false
--scratch_dirs=/tmp
--queue_wait_timeout_ms=60000
--max_vcore_oversubscription_ratio=2.5
--rm_mem_expansion_timeout_ms=5000
--rm_always_use_defaults=false
--rm_default_cpu_vcores=2
--rm_default_memory=4G
--default_pool_max_queued=200
--default_pool_max_requests=-1
--default_pool_mem_limit=
--disable_pool_max_requests=false
--disable_pool_mem_limits=false
--fair_scheduler_allocation_path=
--llama_site_path=
--require_username=false
--disable_admission_control=false
--log_mem_usage_interval=0
--authorization_policy_file=
--authorization_policy_provider_class=org.apache.sentry.provider.common.HadoopGroupResourceAuthorizationProvider
--authorized_proxy_user_config=
--authorized_proxy_user_config_delimiter=,
--server_name=
--abort_on_failed_audit_event=true
--abort_on_failed_lineage_event=true
--audit_event_log_dir=
--be_service_threads=64
--beeswax_port=21000
--cancellation_thread_pool_size=5
--default_query_options=
--fe_service_threads=64
--hs2_port=21050
--idle_query_timeout=0
--idle_session_timeout=0
--lineage_event_log_dir=
--local_nodemanager_url=
--log_query_to_file=true
--max_audit_event_log_file_size=5000
--max_lineage_log_file_size=5000
--max_profile_log_file_size=5000
--max_profile_log_files=10
--max_result_cache_size=100000
--profile_log_dir=
--query_log_size=25
--ssl_client_ca_certificate=
--ssl_private_key=
--ssl_private_key_password_cmd=
--ssl_server_certificate=
--statestore_subscriber_cnxn_attempts=10
--statestore_subscriber_cnxn_retry_interval_ms=3000
--statestore_subscriber_timeout_seconds=30
--state_store_port=24000
--statestore_heartbeat_frequency_ms=1000
--statestore_heartbeat_tcp_timeout_seconds=3
--statestore_max_missed_heartbeats=10
--statestore_num_heartbeat_threads=10
--statestore_num_update_threads=10
--statestore_update_frequency_ms=2000
--statestore_update_tcp_timeout_seconds=300
--force_lowercase_usernames=false
--num_cores=0
--web_log_bytes=1048576
--non_impala_java_vlog=0
--periodic_counter_update_period_ms=500
--enable_webserver_doc_root=true
--webserver_authentication_domain=
--webserver_certificate_file=
--webserver_doc_root=/opt/cloudera/parcels/CDH-5.8.4-1.cdh5.8.4.p0.5/lib/impala
--webserver_interface=
--webserver_password_file=
--webserver_port=25010
--webserver_private_key_file=
--webserver_private_key_password_cmd=
--webserver_x_frame_options=DENY
--flagfile=/var/run/cloudera-scm-agent/process/804-impala-STATESTORE/impala-conf/state_store_flags
--fromenv=
--tryfromenv=
--undefok=
--tab_completion_columns=80
--tab_completion_word=
--help=false
--helpfull=false
--helpmatch=
--helpon=
--helppackage=false
--helpshort=false
--helpxml=false
--version=false
--alsologtoemail=
--alsologtostderr=false
--drop_log_memory=true
--log_backtrace_at=
--log_dir=/var/log/statestore
--log_link=
--log_prefix=true
--logbuflevel=0
--logbufsecs=30
--logemaillevel=999
--logmailer=/bin/mail
--logtostderr=false
--max_log_size=200
--minloglevel=0
--stderrthreshold=4
--stop_logging_if_full_disk=false
--symbolize_stacktrace=true
--v=1
--vmodule=
Apr 19, 9:45:16.780 AM INFO init.cc:207
Cpu Info:
Model: Intel(R) Xeon(R) CPU E5-2640 v3 @ 2.60GHz
Cores: 32
L1 Cache: 32.00 KB (Line: 64.00 B)
L2 Cache: 256.00 KB (Line: 64.00 B)
L3 Cache: 20.00 MB (Line: 64.00 B)
Hardware Supports:
ssse3
sse4_1
sse4_2
popcnt
Apr 19, 9:45:16.780 AM INFO init.cc:208
Disk Info:
Num disks 5:
sda (rotational=true)
sdc (rotational=true)
sdb (rotational=true)
sdd (rotational=true)
dm- (rotational=true)

Apr 19, 9:45:16.780 AM INFO init.cc:209
Physical Memory: 251.89 GB
Apr 19, 9:45:16.780 AM INFO init.cc:210
OS version: Linux version 2.6.32-573.el6.x86_64 (mockbuild@c6b9.bsys.dev.centos.org) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-16) (GCC) ) #1 SMP Thu Jul 23 15:44:03 UTC 2015
Clock: clocksource: 'tsc', clockid_t: CLOCK_MONOTONIC
Apr 19, 9:45:16.780 AM INFO init.cc:211
Process ID: 3489
Apr 19, 9:45:16.780 AM INFO webserver.cc:219
Starting webserver on 0.0.0.0:25010
Apr 19, 9:45:16.780 AM INFO webserver.cc:233
Document root: /opt/cloudera/parcels/CDH-5.8.4-1.cdh5.8.4.p0.5/lib/impala
Apr 19, 9:45:16.781 AM INFO webserver.cc:317
Webserver started
Apr 19, 9:45:16.784 AM INFO thrift-server.cc:445
ThriftServer 'StatestoreService' started on port: 24000
Apr 19, 9:45:19.259 AM ERROR authentication.cc:155
SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()
Apr 19, 9:45:19.259 AM INFO thrift-util.cc:109
TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()
Apr 19, 9:45:19.344 AM ERROR authentication.cc:155
SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()
Apr 19, 9:45:19.344 AM INFO thrift-util.cc:109
TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()
Apr 19, 9:45:19.364 AM ERROR authentication.cc:155
SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()
Apr 19, 9:45:19.364 AM INFO thrift-util.cc:109
TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()
Apr 19, 9:45:19.396 AM ERROR authentication.cc:155
SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()
Apr 19, 9:45:19.396 AM INFO thrift-util.cc:109
TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()
Apr 19, 9:45:19.404 AM ERROR authentication.cc:155
SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()
Apr 19, 9:45:19.404 AM INFO thrift-util.cc:109
TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()
Apr 19, 9:45:22.263 AM ERROR authentication.cc:155
SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()
Apr 19, 9:45:22.263 AM INFO thrift-util.cc:109
TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()
Apr 19, 9:45:22.347 AM ERROR authentication.cc:155
SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()
Apr 19, 9:45:22.347 AM INFO thrift-util.cc:109
TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()
Apr 19, 9:45:22.366 AM ERROR authentication.cc:155
SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()
Apr 19, 9:45:22.367 AM INFO thrift-util.cc:109
TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()
Apr 19, 9:45:22.401 AM ERROR authentication.cc:155
SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()
Apr 19, 9:45:22.401 AM INFO thrift-util.cc:109
TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()
Apr 19, 9:45:22.407 AM ERROR authentication.cc:155
SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()
Apr 19, 9:45:22.407 AM INFO thrift-util.cc:109
TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()
Apr 19, 9:45:28.539 AM ERROR authentication.cc:155
SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()
Apr 19, 9:45:28.539 AM INFO thrift-util.cc:109
TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()
Apr 19, 9:45:29.153 AM ERROR authentication.cc:155
SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()
Apr 19, 9:45:29.153 AM INFO thrift-util.cc:109
TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()
Apr 19, 9:45:29.198 AM ERROR authentication.cc:155
SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()
Apr 19, 9:45:29.198 AM INFO thrift-util.cc:109
TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()
Apr 19, 9:45:29.203 AM ERROR authentication.cc:155
SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()
Apr 19, 9:45:29.203 AM INFO thrift-util.cc:109
TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()
Apr 19, 9:45:29.207 AM ERROR authentication.cc:155
SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()
Apr 19, 9:45:29.207 AM INFO thrift-util.cc:109
TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()
Apr 19, 9:45:31.543 AM ERROR authentication.cc:155
SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()
Apr 19, 9:45:31.543 AM INFO thrift-util.cc:109
TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()
Apr 19, 9:45:32.155 AM ERROR authentication.cc:155
SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()
Apr 19, 9:45:32.156 AM INFO thrift-util.cc:109
TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()
Apr 19, 9:45:32.200 AM ERROR authentication.cc:155
SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()
Apr 19, 9:45:32.201 AM INFO thrift-util.cc:109
TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()
Apr 19, 9:45:32.206 AM ERROR authentication.cc:155
SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()
Apr 19, 9:45:32.206 AM INFO thrift-util.cc:109
TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()
Apr 19, 9:45:32.209 AM ERROR authentication.cc:155
SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()
Apr 19, 9:45:32.209 AM INFO thrift-util.cc:109
TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()
Apr 19, 9:45:38.821 AM ERROR authentication.cc:155
SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()
Apr 19, 9:45:38.821 AM INFO thrift-util.cc:109
TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()
Apr 19, 9:45:39.986 AM ERROR authentication.cc:155
SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()
Apr 19, 9:45:39.987 AM INFO thrift-util.cc:109
TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()
Apr 19, 9:45:39.987 AM ERROR authentication.cc:155
SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()
Apr 19, 9:45:39.988 AM INFO thrift-util.cc:109
TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()
Apr 19, 9:45:39.993 AM ERROR authentication.cc:155
SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()
Apr 19, 9:45:39.993 AM INFO thrift-util.cc:109
TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()
Apr 19, 9:45:40.006 AM ERROR authentication.cc:155
SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()
Apr 19, 9:45:40.007 AM INFO thrift-util.cc:109
TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()
Apr 19, 9:45:41.824 AM ERROR authentication.cc:155
SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()
Apr 19, 9:45:41.824 AM INFO thrift-util.cc:109
TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()
Apr 19, 9:45:42.989 AM ERROR authentication.cc:155
SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()
Apr 19, 9:45:42.990 AM INFO thrift-util.cc:109
TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()
Apr 19, 9:45:42.990 AM ERROR authentication.cc:155
SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()
Apr 19, 9:45:42.991 AM INFO thrift-util.cc:109
TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()
Apr 19, 9:45:42.996 AM ERROR authentication.cc:155
SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()
Apr 19, 9:45:42.996 AM INFO thrift-util.cc:109
TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()
Apr 19, 9:45:43.009 AM ERROR authentication.cc:155
SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()
Apr 19, 9:45:43.009 AM INFO thrift-util.cc:109
TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()
Apr 19, 9:45:46.780 AM INFO logging-support.cc:136
Old log file deleted during log rotation: /var/log/statestore/statestored.CM-PR-BGD-M02.impala.log.INFO.20170412-102422.26964
Apr 19, 9:45:46.780 AM INFO logging-support.cc:136
Old log file deleted during log rotation: /var/log/statestore/statestored.CM-PR-BGD-M02.impala.log.WARNING.20170412-102422.26964
Apr 19, 9:45:46.780 AM INFO logging-support.cc:136
Old log file deleted during log rotation: /var/log/statestore/statestored.CM-PR-BGD-M02.impala.log.ERROR.20170412-102422.26964
Apr 19, 9:45:50.107 AM ERROR authentication.cc:155
SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()
Apr 19, 9:45:50.107 AM INFO thrift-util.cc:109
TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()
Apr 19, 9:45:51.879 AM ERROR authentication.cc:155
SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()
Apr 19, 9:45:51.880 AM INFO thrift-util.cc:109
TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()
Apr 19, 9:45:51.886 AM ERROR authentication.cc:155
SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()
Apr 19, 9:45:51.887 AM INFO thrift-util.cc:109
TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()
Apr 19, 9:45:51.905 AM ERROR authentication.cc:155
SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()
Apr 19, 9:45:51.905 AM INFO thrift-util.cc:109
TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()
Apr 19, 9:45:51.964 AM ERROR authentication.cc:155
SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()
Apr 19, 9:45:51.964 AM INFO thrift-util.cc:109
TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()
Apr 19, 9:45:53.111 AM ERROR authentication.cc:155
SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()
Apr 19, 9:45:53.111 AM INFO thrift-util.cc:109
TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()
Apr 19, 9:45:54.882 AM ERROR authentication.cc:155
SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()
Apr 19, 9:45:54.883 AM INFO thrift-util.cc:109
TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()
Apr 19, 9:45:54.889 AM ERROR authentication.cc:155
SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()
Apr 19, 9:45:54.890 AM INFO thrift-util.cc:109
TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()
Apr 19, 9:45:54.908 AM ERROR authentication.cc:155
SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()
Apr 19, 9:45:54.908 AM INFO thrift-util.cc:109
TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()
Apr 19, 9:45:54.967 AM ERROR authentication.cc:155
SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()
Apr 19, 9:45:54.967 AM INFO thrift-util.cc:109
TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information ()

[#2]

Those are probably messages when the other daemons try to contact it. The issue may still be with the statestore but check the catalogd and impalad logs to see if there is more information as this error is very generic. It just means that there was an authentication error between the client and the statestore. Also, if possible post the krb5.conf and principals for the impala roles.

[#3]

Below is the log from Impala Catalog Server

KRB5.CONF

==========

> cat /etc/krb5.conf
[libdefaults]
default_realm = <DOMAIN-NAME>
dns_lookup_kdc = true
dns_lookup_realm = false
ticket_lifetime = 86400
renew_lifetime = 604800
forwardable = true
default_tgs_enctypes = rc4-hmac aes128-cts aes256-cts des-cbc-crc des-cbc-md5
default_tkt_enctypes = rc4-hmac aes128-cts aes256-cts des-cbc-crc des-cbc-md5
permitted_enctypes = rc4-hmac aes128-cts aes256-cts des-cbc-crc des-cbc-md5
udp_preference_limit = 1
kdc_timeout = 3000

[realms]
<DOMAIN-NAME> = {
kdc = <AD SERVER FQDN>
admin_server = <AD SERVER FQDN>
}

KEYTAB 

=======

> klist -ket /var/run/cloudera-scm-agent/process/805-impala-CATALOGSERVER/impala.keytab
Keytab name: FILE:/var/run/cloudera-scm-agent/process/805-impala-CATALOGSERVER/impala.keytab
KVNO Timestamp Principal
---- ----------------- --------------------------------------------------------
1 04/14/17 03:17:07 impala/<HOSTNAME>@<DOMAIN> (arcfour-hmac)
1 04/14/17 03:17:08 impala/<HOSTNAME>@<DOMAIN> (aes128-cts-hmac-sha1-96)
1 04/14/17 03:17:08 impala/<HOSTNAME>@<DOMAIN> (aes256-cts-hmac-sha1-96)
1 04/14/17 03:17:08 impala/<HOSTNAME>@<DOMAIN> (des-cbc-crc)
1 04/14/17 03:17:08 impala/<HOSTNAME>@<DOMAIN> (des-cbc-md5)

ROLE LOG

=========

Time Log Level Source Log Message
Apr 19, 10:06:47.589 AM INFO logging.cc:117
stdout will be logged to this file.
Apr 19, 10:06:47.589 AM ERROR logging.cc:118
stderr will be logged to this file.
Apr 19, 10:06:47.589 AM INFO minidump.cc:199
Setting minidump size limit to 20971520.
Apr 19, 10:06:47.590 AM INFO atomicops-internals-x86.cc:93
vendor GenuineIntel family 6 model 15 sse2 1 cmpxchg16b 1
Apr 19, 10:06:47.599 AM INFO authentication.cc:678
Using internal kerberos principal "impala/<FQDN>@<DOMAIN-NAME>"
Apr 19, 10:06:47.599 AM INFO authentication.cc:1013
Internal communication is authenticated with Kerberos
Apr 19, 10:06:47.599 AM INFO authentication.cc:798
Waiting for Kerberos ticket for principal: impala/<FQDN>@<DOMAIN-NAME>
Apr 19, 10:06:47.599 AM INFO authentication.cc:494
Registering impala/<FQDN>@<DOMAIN-NAME>, keytab file /var/run/cloudera-scm-agent/process/811-impala-CATALOGSERVER/impala.keytab
Apr 19, 10:06:47.709 AM INFO authentication.cc:800
Kerberos ticket granted to impala/<FQDN>@<DOMAIN-NAME>
Apr 19, 10:06:47.709 AM INFO authentication.cc:678
Using external kerberos principal "impala/<FQDN>@<DOMAIN-NAME>"
Apr 19, 10:06:47.709 AM INFO authentication.cc:1029
External communication is authenticated with Kerberos
Apr 19, 10:06:47.709 AM INFO init.cc:201
catalogd version 2.6.0-cdh5.8.4 RELEASE (build 207450616f75adbe082a4c2e1145a2384da83fa6)
Built on Mon, 06 Feb 2017 14:33:04 PST
Apr 19, 10:06:47.709 AM INFO init.cc:202
Using hostname: <hostname>
Apr 19, 10:06:47.710 AM INFO logging.cc:153
Flags (see also /varz are on debug webserver):
--catalog_service_port=26000
--load_catalog_in_background=true
--num_metadata_loading_threads=16
--sentry_config=/var/run/cloudera-scm-agent/process/811-impala-CATALOGSERVER/impala-conf/sentry-site.xml
--asm_module_dir=
--disable_optimization_passes=false
--dump_ir=false
--opt_module_dir=
--perf_map=false
--print_llvm_ir_instruction_count=false
--unopt_module_dir=
--abort_on_config_error=true
--be_port=22000
--be_principal=
--compact_catalog_topic=false
--disable_mem_pools=false
--enable_accept_queue_server=true
--enable_process_lifetime_heap_profiling=false
--heap_profile_dir=
--hostname=<hostname>
--inc_stats_size_limit_bytes=209715200
--keytab_file=/var/run/cloudera-scm-agent/process/811-impala-CATALOGSERVER/impala.keytab
--krb5_conf=
--krb5_debug_file=
--load_auth_to_local_rules=false
--max_minidumps=9
--mem_limit=80%
--minidump_path=/var/log/impala-minidumps/catalogd
--minidump_size_limit_hint_kb=20480
--principal=impala/<FQDN>@<DOMAIN-NAME>
--redaction_rules_file=
--max_log_files=10
--pause_monitor_sleep_time_ms=500
--pause_monitor_warn_threshold_ms=10000
--log_filename=catalogd
--redirect_stdout_stderr=true
--data_source_batch_size=1024
--exchg_node_buffer_size_bytes=10485760
--enable_partitioned_aggregation=true
--enable_partitioned_hash_join=true
--enable_probe_side_filtering=true
--enable_quadratic_probing=true
--skip_lzo_version_check=false
--convert_legacy_hive_parquet_utc_timestamps=false
--max_page_header_size=8388608
--parquet_min_filter_reject_ratio=0.10000000000000001
--max_row_batches=0
--runtime_filter_wait_time_ms=1000
--suppress_unknown_disk_id_warnings=false
--kudu_max_row_batches=0
--kudu_scanner_keep_alive_period_us=15000000
--kudu_scanner_keep_alive_period_sec=15
--kudu_scanner_timeout_sec=60
--pick_only_leaders_for_tests=false
--kudu_session_timeout_seconds=60
--enable_phj_probe_side_filtering=true
--accepted_cnxn_queue_depth=10000
--enable_ldap_auth=false
--internal_principals_whitelist=hdfs
--kerberos_reinit_interval=60
--ldap_allow_anonymous_binds=false
--ldap_baseDN=
--ldap_bind_pattern=
--ldap_ca_certificate=
--ldap_domain=
--ldap_manual_config=false
--ldap_passwords_in_clear_ok=false
--ldap_tls=false
--ldap_uri=
--sasl_path=
--rpc_cnxn_attempts=10
--rpc_cnxn_retry_interval_ms=2000
--disk_spill_encryption=false
--insert_inherit_permissions=false
--datastream_sender_timeout_ms=120000
--max_cached_file_handles=0
--max_free_io_buffers=128
--min_buffer_size=1024
--num_disks=0
--num_remote_hdfs_io_threads=8
--num_s3_io_threads=16
--num_threads_per_disk=0
--read_size=8388608
--backend_client_connection_num_retries=3
--backend_client_rpc_timeout_ms=300000
--catalog_client_connection_num_retries=3
--catalog_client_rpc_timeout_ms=0
--catalog_service_host=localhost
--cgroup_hierarchy_path=
--coordinator_rpc_threads=12
--enable_rm=false
--enable_webserver=true
--llama_addresses=
--llama_callback_port=28000
--llama_host=
--llama_max_request_attempts=5
--llama_port=15000
--llama_registration_timeout_secs=30
--llama_registration_wait_secs=3
--num_hdfs_worker_threads=16
--resource_broker_cnxn_attempts=1
--resource_broker_cnxn_retry_interval_ms=3000
--resource_broker_recv_timeout=0
--resource_broker_send_timeout=0
--staging_cgroup=impala_staging
--state_store_host=<FQDN>
--state_store_subscriber_port=23020
--use_statestore=true
--s3a_access_key_cmd=
--s3a_secret_key_cmd=
--local_library_dir=/tmp
--serialize_batch=false
--status_report_interval=5
--max_filter_error_rate=0.75
--num_threads_per_core=3
--use_local_tz_for_unix_timestamp_conversions=false
--scratch_dirs=/tmp
--queue_wait_timeout_ms=60000
--max_vcore_oversubscription_ratio=2.5
--rm_mem_expansion_timeout_ms=5000
--rm_always_use_defaults=false
--rm_default_cpu_vcores=2
--rm_default_memory=4G
--default_pool_max_queued=200
--default_pool_max_requests=-1
--default_pool_mem_limit=
--disable_pool_max_requests=false
--disable_pool_mem_limits=false
--fair_scheduler_allocation_path=
--llama_site_path=
--require_username=false
--disable_admission_control=false
--log_mem_usage_interval=0
--authorization_policy_file=
--authorization_policy_provider_class=org.apache.sentry.provider.common.HadoopGroupResourceAuthorizationProvider
--authorized_proxy_user_config=
--authorized_proxy_user_config_delimiter=,
--server_name=
--abort_on_failed_audit_event=true
--abort_on_failed_lineage_event=true
--audit_event_log_dir=
--be_service_threads=64
--beeswax_port=21000
--cancellation_thread_pool_size=5
--default_query_options=
--fe_service_threads=64
--hs2_port=21050
--idle_query_timeout=0
--idle_session_timeout=0
--lineage_event_log_dir=
--local_nodemanager_url=
--log_query_to_file=true
--max_audit_event_log_file_size=5000
--max_lineage_log_file_size=5000
--max_profile_log_file_size=5000
--max_profile_log_files=10
--max_result_cache_size=100000
--profile_log_dir=
--query_log_size=25
--ssl_client_ca_certificate=
--ssl_private_key=
--ssl_private_key_password_cmd=
--ssl_server_certificate=
--statestore_subscriber_cnxn_attempts=10
--statestore_subscriber_cnxn_retry_interval_ms=3000
--statestore_subscriber_timeout_seconds=30
--state_store_port=24000
--statestore_heartbeat_frequency_ms=1000
--statestore_heartbeat_tcp_timeout_seconds=3
--statestore_max_missed_heartbeats=10
--statestore_num_heartbeat_threads=10
--statestore_num_update_threads=10
--statestore_update_frequency_ms=2000
--statestore_update_tcp_timeout_seconds=300
--force_lowercase_usernames=false
--num_cores=0
--web_log_bytes=1048576
--non_impala_java_vlog=0
--periodic_counter_update_period_ms=500
--enable_webserver_doc_root=true
--webserver_authentication_domain=
--webserver_certificate_file=
--webserver_doc_root=/opt/cloudera/parcels/CDH-5.8.4-1.cdh5.8.4.p0.5/lib/impala
--webserver_interface=
--webserver_password_file=
--webserver_port=25020
--webserver_private_key_file=
--webserver_private_key_password_cmd=
--webserver_x_frame_options=DENY
--flagfile=/var/run/cloudera-scm-agent/process/811-impala-CATALOGSERVER/impala-conf/catalogserver_flags
--fromenv=
--tryfromenv=
--undefok=
--tab_completion_columns=80
--tab_completion_word=
--help=false
--helpfull=false
--helpmatch=
--helpon=
--helppackage=false
--helpshort=false
--helpxml=false
--version=false
--alsologtoemail=
--alsologtostderr=false
--drop_log_memory=true
--log_backtrace_at=
--log_dir=/var/log/catalogd
--log_link=
--log_prefix=true
--logbuflevel=0
--logbufsecs=30
--logemaillevel=999
--logmailer=/bin/mail
--logtostderr=false
--max_log_size=200
--minloglevel=0
--stderrthreshold=4
--stop_logging_if_full_disk=false
--symbolize_stacktrace=true
--v=1
--vmodule=
Apr 19, 10:06:47.710 AM INFO init.cc:207
Cpu Info:
Model: Intel(R) Xeon(R) CPU E5-2640 v3 @ 2.60GHz
Cores: 32
L1 Cache: 32.00 KB (Line: 64.00 B)
L2 Cache: 256.00 KB (Line: 64.00 B)
L3 Cache: 20.00 MB (Line: 64.00 B)
Hardware Supports:
ssse3
sse4_1
sse4_2
popcnt
Apr 19, 10:06:47.710 AM INFO init.cc:208
Disk Info:
Num disks 5:
sda (rotational=true)
sdc (rotational=true)
sdb (rotational=true)
sdd (rotational=true)
dm- (rotational=true)

Apr 19, 10:06:47.710 AM INFO init.cc:209
Physical Memory: 251.89 GB
Apr 19, 10:06:47.710 AM INFO init.cc:210
OS version: Linux version 2.6.32-573.el6.x86_64 (mockbuild@c6b9.bsys.dev.centos.org) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-16) (GCC) ) #1 SMP Thu Jul 23 15:44:03 UTC 2015
Clock: clocksource: 'tsc', clockid_t: CLOCK_MONOTONIC
Apr 19, 10:06:47.710 AM INFO init.cc:211
Process ID: 8626
Apr 19, 10:06:48.686 AM INFO webserver.cc:219
Starting webserver on 0.0.0.0:25020
Apr 19, 10:06:48.686 AM INFO webserver.cc:233
Document root: /opt/cloudera/parcels/CDH-5.8.4-1.cdh5.8.4.p0.5/lib/impala
Apr 19, 10:06:48.686 AM INFO webserver.cc:317
Webserver started
Apr 19, 10:06:48.728 AM INFO GlogAppender.java:123
Logging initialized. Impala: VLOG, All other: INFO
Apr 19, 10:06:48.731 AM INFO JniCatalog.java:99
Java Version Info: Java(TM) SE Runtime Environment (1.7.0_67-b01)
Apr 19, 10:06:49.196 AM INFO MetaStoreClientPool.java:56
Creating MetaStoreClient. Pool Size = 0
Apr 19, 10:06:49.206 AM INFO HiveMetaStoreClient.java:386
Trying to connect to metastore with URI thrift://<hostname-1>:9083
Apr 19, 10:06:49.324 AM INFO HiveMetaStoreClient.java:431
Opened a connection to metastore, current connections: 1
Apr 19, 10:06:49.324 AM INFO HiveMetaStoreClient.java:483
Connected to metastore.
Apr 19, 10:06:49.324 AM INFO MetaStoreClientPool.java:56
Creating MetaStoreClient. Pool Size = 1
Apr 19, 10:06:49.325 AM INFO HiveMetaStoreClient.java:386
Trying to connect to metastore with URI thrift://<hostname-1>:9083
Apr 19, 10:06:49.333 AM INFO HiveMetaStoreClient.java:431
Opened a connection to metastore, current connections: 2
Apr 19, 10:06:49.334 AM INFO HiveMetaStoreClient.java:483
Connected to metastore.
Apr 19, 10:06:49.334 AM INFO MetaStoreClientPool.java:56
Creating MetaStoreClient. Pool Size = 2
Apr 19, 10:06:49.334 AM INFO HiveMetaStoreClient.java:386
Trying to connect to metastore with URI thrift://<hostname-1>:9083
Apr 19, 10:06:49.341 AM INFO HiveMetaStoreClient.java:431
Opened a connection to metastore, current connections: 3
Apr 19, 10:06:49.342 AM INFO HiveMetaStoreClient.java:483
Connected to metastore.
Apr 19, 10:06:49.342 AM INFO MetaStoreClientPool.java:56
Creating MetaStoreClient. Pool Size = 3
Apr 19, 10:06:49.343 AM INFO HiveMetaStoreClient.java:386
Trying to connect to metastore with URI thrift://<hostname-1>:9083
Apr 19, 10:06:49.351 AM INFO HiveMetaStoreClient.java:431
Opened a connection to metastore, current connections: 4
Apr 19, 10:06:49.351 AM INFO HiveMetaStoreClient.java:483
Connected to metastore.
Apr 19, 10:06:49.351 AM INFO MetaStoreClientPool.java:56
Creating MetaStoreClient. Pool Size = 4
Apr 19, 10:06:49.352 AM INFO HiveMetaStoreClient.java:386
Trying to connect to metastore with URI thrift://<hostname-1>:9083
Apr 19, 10:06:49.359 AM INFO HiveMetaStoreClient.java:431
Opened a connection to metastore, current connections: 5
Apr 19, 10:06:49.360 AM INFO HiveMetaStoreClient.java:483
Connected to metastore.
Apr 19, 10:06:49.656 AM INFO MetaStoreClientPool.java:56
Creating MetaStoreClient. Pool Size = 4
Apr 19, 10:06:49.656 AM INFO HiveMetaStoreClient.java:386
Trying to connect to metastore with URI thrift://<hostname-1>:9083
Apr 19, 10:06:49.663 AM INFO HiveMetaStoreClient.java:431
Opened a connection to metastore, current connections: 6
Apr 19, 10:06:49.663 AM INFO HiveMetaStoreClient.java:483
Connected to metastore.
Apr 19, 10:06:49.690 AM INFO CatalogServiceCatalog.java:549
Loading native functions for database: bedrock
Apr 19, 10:06:49.690 AM INFO CatalogServiceCatalog.java:575
Loading Java functions for database: bedrock
Apr 19, 10:06:49.709 AM INFO CatalogServiceCatalog.java:549
Loading native functions for database: default
Apr 19, 10:06:49.709 AM INFO CatalogServiceCatalog.java:575
Loading Java functions for database: default
Apr 19, 10:06:49.714 AM INFO HiveMetaStoreClient.java:512
Closed a connection to metastore, current connections: 5
Apr 19, 10:06:49.715 AM INFO TableLoadingMgr.java:278
Loading next table. Remaining items in queue: 0
Apr 19, 10:06:49.716 AM INFO TableLoadingMgr.java:278
Loading next table. Remaining items in queue: 0
Apr 19, 10:06:49.717 AM INFO TableLoader.java:56
Loading metadata for: default.test
Apr 19, 10:06:49.717 AM INFO MetaStoreClientPool.java:56
Creating MetaStoreClient. Pool Size = 3
Apr 19, 10:06:49.717 AM INFO HiveMetaStoreClient.java:386
Trying to connect to metastore with URI thrift://<hostname-1>:9083
Apr 19, 10:06:49.717 AM INFO TableLoader.java:56
Loading metadata for: bedrock.student_testing_1
Apr 19, 10:06:49.723 AM INFO statestore-subscriber.cc:178
Starting statestore subscriber
Apr 19, 10:06:49.725 AM INFO thrift-server.cc:445
ThriftServer 'StatestoreSubscriber' started on port: 23020
Apr 19, 10:06:49.725 AM INFO statestore-subscriber.cc:189
Registering with statestore
Apr 19, 10:06:49.725 AM INFO HiveMetaStoreClient.java:431
Opened a connection to metastore, current connections: 6
Apr 19, 10:06:49.725 AM INFO HiveMetaStoreClient.java:483
Connected to metastore.
Apr 19, 10:06:49.725 AM INFO MetaStoreClientPool.java:56
Creating MetaStoreClient. Pool Size = 2
Apr 19, 10:06:49.726 AM INFO HiveMetaStoreClient.java:386
Trying to connect to metastore with URI thrift://<hostname-1>:9083
Apr 19, 10:06:49.731 AM INFO status.cc:111
Couldn't open transport for <FQDN>:24000 (No more data to read.)
@ 0x8133b9 (unknown)
@ 0xd5052f (unknown)
@ 0xd50752 (unknown)
@ 0xa0462b (unknown)
@ 0xa04c50 (unknown)
@ 0xb09647 (unknown)
@ 0xb0b5b0 (unknown)
@ 0x7e62c3 (unknown)
@ 0x7df2ff (unknown)
@ 0x7de816 (unknown)
@ 0x7f532f738d1d __libc_start_main
@ 0x7de4dd (unknown)
Apr 19, 10:06:49.731 AM INFO thrift-client.cc:58
Unable to connect to <FQDN>:24000
Apr 19, 10:06:49.731 AM INFO thrift-client.cc:64
(Attempt 1 of 10)
Apr 19, 10:06:49.733 AM INFO HiveMetaStoreClient.java:431
Opened a connection to metastore, current connections: 7
Apr 19, 10:06:49.733 AM INFO HiveMetaStoreClient.java:483
Connected to metastore.
Apr 19, 10:06:49.751 AM INFO catalog-server.cc:313
Publishing update: DATABASE:bedrock@1
Apr 19, 10:06:49.751 AM INFO catalog-server.cc:313
Publishing update: TABLE:bedrock.student_testing_1@2
Apr 19, 10:06:49.751 AM INFO catalog-server.cc:313
Publishing update: DATABASE:default@3
Apr 19, 10:06:49.751 AM INFO catalog-server.cc:313
Publishing update: TABLE:default.test@4
Apr 19, 10:06:49.751 AM INFO catalog-server.cc:313
Publishing update: CATALOG:ad685743df72412a:9aa412bb03b25186@4
Apr 19, 10:06:49.879 AM INFO Table.java:158
Loading column stats for table: test
Apr 19, 10:06:49.879 AM INFO Table.java:158
Loading column stats for table: student_testing_1
Apr 19, 10:06:49.899 AM INFO HdfsTable.java:1021
load table from Hive Metastore: default.test
Apr 19, 10:06:49.899 AM INFO HdfsTable.java:1021
load table from Hive Metastore: bedrock.student_testing_1
Apr 19, 10:06:50.155 AM INFO HiveMetaStoreClient.java:512
Closed a connection to metastore, current connections: 6
Apr 19, 10:06:50.156 AM INFO HiveMetaStoreClient.java:512
Closed a connection to metastore, current connections: 5
Apr 19, 10:06:52.731 AM INFO thrift-util.cc:109
TSocket::write_partial() send() <Host: <FQDN> Port: 24000>Broken pipe
Apr 19, 10:06:52.732 AM INFO client-cache.h:259
client 0x9f0b040 unexpected exception: write() send(): Broken pipe, type=N6apache6thrift9transport19TTransportExceptionE
Apr 19, 10:06:52.732 AM INFO client-cache.cc:78
ReopenClient(): re-creating client for <FQDN>:24000
Apr 19, 10:06:52.735 AM INFO status.cc:111
Couldn't open transport for <FQDN>:24000 (No more data to read.)
@ 0x8133b9 (unknown)
@ 0xd5052f (unknown)
@ 0xd50752 (unknown)
@ 0xa0462b (unknown)
@ 0xa05079 (unknown)
@ 0xb0ee78 (unknown)
@ 0xb0f0ee (unknown)
@ 0xb099bf (unknown)
@ 0xb0b5b0 (unknown)
@ 0x7e62c3 (unknown)
@ 0x7df2ff (unknown)
@ 0x7de816 (unknown)
@ 0x7f532f738d1d __libc_start_main
@ 0x7de4dd (unknown)
Apr 19, 10:06:52.735 AM INFO thrift-client.cc:58
Unable to connect to <FQDN>:24000
Apr 19, 10:06:52.735 AM INFO thrift-client.cc:64
(Attempt 1 of 10)
Apr 19, 10:06:55.735 AM INFO thrift-util.cc:109
TSocket::write_partial() send() <Host: <FQDN> Port: 24000>Broken pipe
Apr 19, 10:06:55.736 AM INFO status.cc:44
RPC Error: write() send(): Broken pipe
@ 0x81228a (unknown)
@ 0xb0f40b (unknown)
@ 0xb099bf (unknown)
@ 0xb0b5b0 (unknown)
@ 0x7e62c3 (unknown)
@ 0x7df2ff (unknown)
@ 0x7de816 (unknown)
@ 0x7f532f738d1d __libc_start_main
@ 0x7de4dd (unknown)
Apr 19, 10:06:55.736 AM INFO client-cache.cc:163
Broken Connection, destroy client for <FQDN>:24000
Apr 19, 10:06:55.736 AM INFO statestore-subscriber.cc:195
statestore registration unsuccessful: RPC Error: write() send(): Broken pipe
Apr 19, 10:06:55.736 AM FATAL catalogd-main.cc:76
RPC Error: write() send(): Broken pipe
. Impalad exiting.

[#4 · Accepted solution]

Got this issue fixed , looks like impala doesnot accest upper case hostnames once cluster is kerberized .  changing the hostname from uppercase to lower case resolved this issue .

Thanks...

[#5]

Hi Friend, 

Where do we need to change this hostname from upper case to lower case? in which configuration file. Please advice. 

[#6]

@Srinivs,

In CDH, currently your hosts need to have lowercase letters in their fully-qualified hostnames.

The change would be to your host's hostname (i.e. /etc/sysconfig/network, etc.)

Depends on your OS and config

[#7]

Hi All,

I am also facing the same issue. 

after enabling the kerberos, impala services are not started. Tried to start manually, but failed. in UI, i am able to see all the 3 instances in started state, but when i am running impala-shell command i am getting the error. the java processes for impala are also not present.

Please help me in troubleshooting the issue.

my hostname is also in small letters....

Thanks in advance...