Hello ,   After kerberizing the cluster impala daemons fails to start . I see Impala state store is up but when looked at the Role logs it shows GSS execption .    Tried to go through some of the community help as below, made sure the encryption types are fine but that did not help resolve the issue   One more observation below , the error doesnot give details as to what the error could be related to and just calls out saying Unspecified GSS faliure . "TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information ()"   reference : https://community.cloudera.com/t5/Interactive-Short-cycle-SQL/kerberos-authentication-failure-GSSAPI-Failure-gss-accept-sec/m-p/24707#M742   snap shot from CM: =============== Error   ==== Apr 19, 9:45:16.664 AM INFO logging.cc:117 stdout will be logged to this file. Apr 19, 9:45:16.664 AM ERROR logging.cc:118 stderr will be logged to this file. Apr 19, 9:45:16.664 AM INFO minidump.cc:199 Setting minidump size limit to 20971520. Apr 19, 9:45:16.664 AM INFO atomicops-internals-x86.cc:93 vendor GenuineIntel family 6 model 15 sse2 1 cmpxchg16b 1 Apr 19, 9:45:16.673 AM INFO authentication.cc:678 Using internal kerberos principal "impala/CM-PR-BGD-M02.telemovil.com.sv@TELEMOVIL.COM.SV" Apr 19, 9:45:16.673 AM INFO authentication.cc:1013 Internal communication is authenticated with Kerberos Apr 19, 9:45:16.673 AM INFO authentication.cc:798 Waiting for Kerberos ticket for principal: impala/CM-PR-BGD-M02.telemovil.com.sv@TELEMOVIL.COM.SV Apr 19, 9:45:16.673 AM INFO authentication.cc:494 Registering impala/CM-PR-BGD-M02.telemovil.com.sv@TELEMOVIL.COM.SV, keytab file /var/run/cloudera-scm-agent/process/804-impala-STATESTORE/impala.keytab Apr 19, 9:45:16.780 AM INFO authentication.cc:800 Kerberos ticket granted to impala/CM-PR-BGD-M02.telemovil.com.sv@TELEMOVIL.COM.SV Apr 19, 9:45:16.780 AM INFO authentication.cc:678 Using external kerberos principal "impala/CM-PR-BGD-M02.telemovil.com.sv@TELEMOVIL.COM.SV" Apr 19, 9:45:16.780 AM INFO authentication.cc:1029 External communication is authenticated with Kerberos Apr 19, 9:45:16.780 AM INFO init.cc:201 statestored version 2.6.0-cdh5.8.4 RELEASE (build 207450616f75adbe082a4c2e1145a2384da83fa6) Built on Mon, 06 Feb 2017 14:33:04 PST Apr 19, 9:45:16.780 AM INFO init.cc:202 Using hostname: CM-PR-BGD-M02 Apr 19, 9:45:16.780 AM INFO logging.cc:153 Flags (see also /varz are on debug webserver): --catalog_service_port=26000 --load_catalog_in_background=false --num_metadata_loading_threads=16 --sentry_config= --asm_module_dir= --disable_optimization_passes=false --dump_ir=false --opt_module_dir= --perf_map=false --print_llvm_ir_instruction_count=false --unopt_module_dir= --abort_on_config_error=true --be_port=22000 --be_principal= --compact_catalog_topic=false --disable_mem_pools=false --enable_accept_queue_server=true --enable_process_lifetime_heap_profiling=false --heap_profile_dir= --hostname=CM-PR-BGD-M02 --inc_stats_size_limit_bytes=209715200 --keytab_file=/var/run/cloudera-scm-agent/process/804-impala-STATESTORE/impala.keytab --krb5_conf= --krb5_debug_file= --load_auth_to_local_rules=false --max_minidumps=9 --mem_limit=80% --minidump_path=/var/log/impala-minidumps/statestored --minidump_size_limit_hint_kb=20480 --principal=impala/CM-PR-BGD-M02.telemovil.com.sv@TELEMOVIL.COM.SV --redaction_rules_file= --max_log_files=10 --pause_monitor_sleep_time_ms=500 --pause_monitor_warn_threshold_ms=10000 --log_filename=statestored --redirect_stdout_stderr=true --data_source_batch_size=1024 --exchg_node_buffer_size_bytes=10485760 --enable_partitioned_aggregation=true --enable_partitioned_hash_join=true --enable_probe_side_filtering=true --enable_quadratic_probing=true --skip_lzo_version_check=false --convert_legacy_hive_parquet_utc_timestamps=false --max_page_header_size=8388608 --parquet_min_filter_reject_ratio=0.10000000000000001 --max_row_batches=0 --runtime_filter_wait_time_ms=1000 --suppress_unknown_disk_id_warnings=false --kudu_max_row_batches=0 --kudu_scanner_keep_alive_period_us=15000000 --kudu_scanner_keep_alive_period_sec=15 --kudu_scanner_timeout_sec=60 --pick_only_leaders_for_tests=false --kudu_session_timeout_seconds=60 --enable_phj_probe_side_filtering=true --accepted_cnxn_queue_depth=10000 --enable_ldap_auth=false --internal_principals_whitelist=hdfs --kerberos_reinit_interval=60 --ldap_allow_anonymous_binds=false --ldap_baseDN= --ldap_bind_pattern= --ldap_ca_certificate= --ldap_domain= --ldap_manual_config=false --ldap_passwords_in_clear_ok=false --ldap_tls=false --ldap_uri= --sasl_path= --rpc_cnxn_attempts=10 --rpc_cnxn_retry_interval_ms=2000 --disk_spill_encryption=false --insert_inherit_permissions=false --datastream_sender_timeout_ms=120000 --max_cached_file_handles=0 --max_free_io_buffers=128 --min_buffer_size=1024 --num_disks=0 --num_remote_hdfs_io_threads=8 --num_s3_io_threads=16 --num_threads_per_disk=0 --read_size=8388608 --backend_client_connection_num_retries=3 --backend_client_rpc_timeout_ms=300000 --catalog_client_connection_num_retries=3 --catalog_client_rpc_timeout_ms=0 --catalog_service_host=localhost --cgroup_hierarchy_path= --coordinator_rpc_threads=12 --enable_rm=false --enable_webserver=true --llama_addresses= --llama_callback_port=28000 --llama_host= --llama_max_request_attempts=5 --llama_port=15000 --llama_registration_timeout_secs=30 --llama_registration_wait_secs=3 --num_hdfs_worker_threads=16 --resource_broker_cnxn_attempts=1 --resource_broker_cnxn_retry_interval_ms=3000 --resource_broker_recv_timeout=0 --resource_broker_send_timeout=0 --staging_cgroup=impala_staging --state_store_host=localhost --state_store_subscriber_port=23000 --use_statestore=true --s3a_access_key_cmd= --s3a_secret_key_cmd= --local_library_dir=/tmp --serialize_batch=false --status_report_interval=5 --max_filter_error_rate=0.75 --num_threads_per_core=3 --use_local_tz_for_unix_timestamp_conversions=false --scratch_dirs=/tmp --queue_wait_timeout_ms=60000 --max_vcore_oversubscription_ratio=2.5 --rm_mem_expansion_timeout_ms=5000 --rm_always_use_defaults=false --rm_default_cpu_vcores=2 --rm_default_memory=4G --default_pool_max_queued=200 --default_pool_max_requests=-1 --default_pool_mem_limit= --disable_pool_max_requests=false --disable_pool_mem_limits=false --fair_scheduler_allocation_path= --llama_site_path= --require_username=false --disable_admission_control=false --log_mem_usage_interval=0 --authorization_policy_file= --authorization_policy_provider_class=org.apache.sentry.provider.common.HadoopGroupResourceAuthorizationProvider --authorized_proxy_user_config= --authorized_proxy_user_config_delimiter=, --server_name= --abort_on_failed_audit_event=true --abort_on_failed_lineage_event=true --audit_event_log_dir= --be_service_threads=64 --beeswax_port=21000 --cancellation_thread_pool_size=5 --default_query_options= --fe_service_threads=64 --hs2_port=21050 --idle_query_timeout=0 --idle_session_timeout=0 --lineage_event_log_dir= --local_nodemanager_url= --log_query_to_file=true --max_audit_event_log_file_size=5000 --max_lineage_log_file_size=5000 --max_profile_log_file_size=5000 --max_profile_log_files=10 --max_result_cache_size=100000 --profile_log_dir= --query_log_size=25 --ssl_client_ca_certificate= --ssl_private_key= --ssl_private_key_password_cmd= --ssl_server_certificate= --statestore_subscriber_cnxn_attempts=10 --statestore_subscriber_cnxn_retry_interval_ms=3000 --statestore_subscriber_timeout_seconds=30 --state_store_port=24000 --statestore_heartbeat_frequency_ms=1000 --statestore_heartbeat_tcp_timeout_seconds=3 --statestore_max_missed_heartbeats=10 --statestore_num_heartbeat_threads=10 --statestore_num_update_threads=10 --statestore_update_frequency_ms=2000 --statestore_update_tcp_timeout_seconds=300 --force_lowercase_usernames=false --num_cores=0 --web_log_bytes=1048576 --non_impala_java_vlog=0 --periodic_counter_update_period_ms=500 --enable_webserver_doc_root=true --webserver_authentication_domain= --webserver_certificate_file= --webserver_doc_root=/opt/cloudera/parcels/CDH-5.8.4-1.cdh5.8.4.p0.5/lib/impala --webserver_interface= --webserver_password_file= --webserver_port=25010 --webserver_private_key_file= --webserver_private_key_password_cmd= --webserver_x_frame_options=DENY --flagfile=/var/run/cloudera-scm-agent/process/804-impala-STATESTORE/impala-conf/state_store_flags --fromenv= --tryfromenv= --undefok= --tab_completion_columns=80 --tab_completion_word= --help=false --helpfull=false --helpmatch= --helpon= --helppackage=false --helpshort=false --helpxml=false --version=false --alsologtoemail= --alsologtostderr=false --drop_log_memory=true --log_backtrace_at= --log_dir=/var/log/statestore --log_link= --log_prefix=true --logbuflevel=0 --logbufsecs=30 --logemaillevel=999 --logmailer=/bin/mail --logtostderr=false --max_log_size=200 --minloglevel=0 --stderrthreshold=4 --stop_logging_if_full_disk=false --symbolize_stacktrace=true --v=1 --vmodule= Apr 19, 9:45:16.780 AM INFO init.cc:207 Cpu Info: Model: Intel(R) Xeon(R) CPU E5-2640 v3 @ 2.60GHz Cores: 32 L1 Cache: 32.00 KB (Line: 64.00 B) L2 Cache: 256.00 KB (Line: 64.00 B) L3 Cache: 20.00 MB (Line: 64.00 B) Hardware Supports: ssse3 sse4_1 sse4_2 popcnt Apr 19, 9:45:16.780 AM INFO init.cc:208 Disk Info: Num disks 5: sda (rotational=true) sdc (rotational=true) sdb (rotational=true) sdd (rotational=true) dm- (rotational=true) Apr 19, 9:45:16.780 AM INFO init.cc:209 Physical Memory: 251.89 GB Apr 19, 9:45:16.780 AM INFO init.cc:210 OS version: Linux version 2.6.32-573.el6.x86_64 (mockbuild@c6b9.bsys.dev.centos.org) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-16) (GCC) ) #1 SMP Thu Jul 23 15:44:03 UTC 2015 Clock: clocksource: 'tsc', clockid_t: CLOCK_MONOTONIC Apr 19, 9:45:16.780 AM INFO init.cc:211 Process ID: 3489 Apr 19, 9:45:16.780 AM INFO webserver.cc:219 Starting webserver on 0.0.0.0:25010 Apr 19, 9:45:16.780 AM INFO webserver.cc:233 Document root: /opt/cloudera/parcels/CDH-5.8.4-1.cdh5.8.4.p0.5/lib/impala Apr 19, 9:45:16.781 AM INFO webserver.cc:317 Webserver started Apr 19, 9:45:16.784 AM INFO thrift-server.cc:445 ThriftServer 'StatestoreService' started on port: 24000 Apr 19, 9:45:19.259 AM ERROR authentication.cc:155 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:19.259 AM INFO thrift-util.cc:109 TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:19.344 AM ERROR authentication.cc:155 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:19.344 AM INFO thrift-util.cc:109 TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:19.364 AM ERROR authentication.cc:155 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:19.364 AM INFO thrift-util.cc:109 TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:19.396 AM ERROR authentication.cc:155 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:19.396 AM INFO thrift-util.cc:109 TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:19.404 AM ERROR authentication.cc:155 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:19.404 AM INFO thrift-util.cc:109 TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:22.263 AM ERROR authentication.cc:155 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:22.263 AM INFO thrift-util.cc:109 TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:22.347 AM ERROR authentication.cc:155 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:22.347 AM INFO thrift-util.cc:109 TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:22.366 AM ERROR authentication.cc:155 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:22.367 AM INFO thrift-util.cc:109 TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:22.401 AM ERROR authentication.cc:155 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:22.401 AM INFO thrift-util.cc:109 TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:22.407 AM ERROR authentication.cc:155 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:22.407 AM INFO thrift-util.cc:109 TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:28.539 AM ERROR authentication.cc:155 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:28.539 AM INFO thrift-util.cc:109 TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:29.153 AM ERROR authentication.cc:155 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:29.153 AM INFO thrift-util.cc:109 TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:29.198 AM ERROR authentication.cc:155 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:29.198 AM INFO thrift-util.cc:109 TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:29.203 AM ERROR authentication.cc:155 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:29.203 AM INFO thrift-util.cc:109 TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:29.207 AM ERROR authentication.cc:155 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:29.207 AM INFO thrift-util.cc:109 TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:31.543 AM ERROR authentication.cc:155 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:31.543 AM INFO thrift-util.cc:109 TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:32.155 AM ERROR authentication.cc:155 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:32.156 AM INFO thrift-util.cc:109 TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:32.200 AM ERROR authentication.cc:155 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:32.201 AM INFO thrift-util.cc:109 TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:32.206 AM ERROR authentication.cc:155 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:32.206 AM INFO thrift-util.cc:109 TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:32.209 AM ERROR authentication.cc:155 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:32.209 AM INFO thrift-util.cc:109 TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:38.821 AM ERROR authentication.cc:155 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:38.821 AM INFO thrift-util.cc:109 TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:39.986 AM ERROR authentication.cc:155 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:39.987 AM INFO thrift-util.cc:109 TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:39.987 AM ERROR authentication.cc:155 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:39.988 AM INFO thrift-util.cc:109 TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:39.993 AM ERROR authentication.cc:155 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:39.993 AM INFO thrift-util.cc:109 TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:40.006 AM ERROR authentication.cc:155 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:40.007 AM INFO thrift-util.cc:109 TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:41.824 AM ERROR authentication.cc:155 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:41.824 AM INFO thrift-util.cc:109 TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:42.989 AM ERROR authentication.cc:155 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:42.990 AM INFO thrift-util.cc:109 TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:42.990 AM ERROR authentication.cc:155 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:42.991 AM INFO thrift-util.cc:109 TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:42.996 AM ERROR authentication.cc:155 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:42.996 AM INFO thrift-util.cc:109 TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:43.009 AM ERROR authentication.cc:155 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:43.009 AM INFO thrift-util.cc:109 TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:46.780 AM INFO logging-support.cc:136 Old log file deleted during log rotation: /var/log/statestore/statestored.CM-PR-BGD-M02.impala.log.INFO.20170412-102422.26964 Apr 19, 9:45:46.780 AM INFO logging-support.cc:136 Old log file deleted during log rotation: /var/log/statestore/statestored.CM-PR-BGD-M02.impala.log.WARNING.20170412-102422.26964 Apr 19, 9:45:46.780 AM INFO logging-support.cc:136 Old log file deleted during log rotation: /var/log/statestore/statestored.CM-PR-BGD-M02.impala.log.ERROR.20170412-102422.26964 Apr 19, 9:45:50.107 AM ERROR authentication.cc:155 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:50.107 AM INFO thrift-util.cc:109 TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:51.879 AM ERROR authentication.cc:155 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:51.880 AM INFO thrift-util.cc:109 TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:51.886 AM ERROR authentication.cc:155 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:51.887 AM INFO thrift-util.cc:109 TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:51.905 AM ERROR authentication.cc:155 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:51.905 AM INFO thrift-util.cc:109 TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:51.964 AM ERROR authentication.cc:155 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:51.964 AM INFO thrift-util.cc:109 TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:53.111 AM ERROR authentication.cc:155 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:53.111 AM INFO thrift-util.cc:109 TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:54.882 AM ERROR authentication.cc:155 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:54.883 AM INFO thrift-util.cc:109 TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:54.889 AM ERROR authentication.cc:155 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:54.890 AM INFO thrift-util.cc:109 TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:54.908 AM ERROR authentication.cc:155 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:54.908 AM INFO thrift-util.cc:109 TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:54.967 AM ERROR authentication.cc:155 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:54.967 AM INFO thrift-util.cc:109 TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () ... View more

ERROR from falcon application log ================= After installing falcon I see SERVICE UNAVAILABLE ERROR on falcon UI . On restart of falcon service see the below error which points hdfs:hdfs is user & group ownership and falcon cannot execute . 017-02-03 20:10:46,874 INFO ipc.Server (Server.java:logException(2394)) - IPC Server handler 105 on 8020, call org.apache.hadoop.hdfs.protocol.ClientProtocol.getFileInfo from 192. 168.1.95:43132 Call#0 Retry#0: org.apache.hadoop.security.AccessControlException: Permission denied: user=falcon, access=EXECUTE, inode="/apps/falcon/extensions":hdfs:hdfs:drwx----- Below is the output of hadoop fs -ls ... [root@hdpkdc ~]# hadoop fs -ls /apps/falcon Found 1 items drwxr-xr-x - falcon users 0 2017-02-02 20:53 /apps/falcon/extensions [root@hdpkdc ~]# hadoop fs -ls /apps/falcon/extensions Found 4 items drwxr-xr-x - falcon users 0 2017-02-02 20:53 /apps/falcon/extensions/hdfs-mirroring drwxr-xr-x - falcon users 0 2017-02-02 20:53 /apps/falcon/extensions/hdfs-snapshot-mirroring drwxr-xr-x - falcon users 0 2017-02-02 20:53 /apps/falcon/extensions/hive-mirroring drwxrwx--- - falcon users 0 2017-02-02 20:53 /apps/falcon/extensions/mirroring Thanks, Jagdish ... View more

Installing Ranger from using Ambari Wizard gets stuck at stage where masters are assigned . After hitting Next the wizard does not move to the next screen. Ambari Version : 2.4.2 1. Which logs would provide details about the progress , couldn't find much details under ambari-server logs . ... View more

user tickets are valid and have been verified , but when the below commands are executed below is the error that comes up . ================ [<username@<hostname> ~]$ export HBASE_CONF_PATH=/etc/hbase/conf:/etc/hadoop/conf [<username@<hostname> ~]$ /usr/hdp/current/phoenix-client/bin/sqlline.py <zookeeper-node>:2181:/hbase-secure:<user-principal>:<user keytab> =============== Mon Aug 29 13:44:54 CDT 2016, RpcRetryingCaller{globalStartTime=1472495354480, pause=100, retries=35}, org.apache.hadoop.hbase.MasterNotRunningException: com.google.protobuf.ServiceException: java.io.IOException: Couldn't setup connection for <USERNAME>@<REALM> to hbase/<FQDN>@<REALM> at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:147) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3917) at org.apache.hadoop.hbase.client.HBaseAdmin.getTableDescriptor(HBaseAdmin.java:441) at org.apache.hadoop.hbase.client.HBaseAdmin.getTableDescriptor(HBaseAdmin.java:463) at org.apache.phoenix.query.ConnectionQueryServicesImpl.ensureTableCreated(ConnectionQueryServicesImpl.java:815) ... 31 more Caused by: org.apache.hadoop.hbase.MasterNotRunningException: com.google.protobuf.ServiceException: java.io.IOException: Couldn't setup connection for <USERNAME>@<REALM> to hbase/<FQDN>@<REALM> at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1533) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1553) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1704) at org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:38) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:124) ... 35 more Caused by: com.google.protobuf.ServiceException: java.io.IOException: Couldn't setup connection for <USERNAME>@<REALM> to hbase/<FQDN>@<REALM> at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:223) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java:50918) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.isMasterRunning(ConnectionManager.java:1564) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStubNoRetries(ConnectionManager.java:1502) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1524) ... 39 more Caused by: java.io.IOException: Couldn't setup connection for <USERNAME>@<REALM> to hbase/<FQDN>@<REALM> at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$1.run(RpcClientImpl.java:665) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.handleSaslConnectionFailure(RpcClientImpl.java:637) at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:745) at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.writeRequest(RpcClientImpl.java:887) at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.tracedWriteRequest(RpcClientImpl.java:856) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1200) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:213) ... 44 more Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)] at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212) at org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:179) at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupSaslConnection(RpcClientImpl.java:611) at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.access$600(RpcClientImpl.java:156) at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:737) at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:734) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:734) ... 48 more Caused by: GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds) at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:710) at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248) at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193) ... 57 more Caused by: KrbException: Fail to create credential. (63) - No service creds at sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:282) at sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:456) at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:641) ... 60 more sqlline version 1.1.8 ... View more