Hello ,   After kerberizing the cluster impala daemons fails to start . I see Impala state store is up but when looked at the Role logs it shows GSS execption .    Tried to go through some of the community help as below, made sure the encryption types are fine but that did not help resolve the issue   One more observation below , the error doesnot give details as to what the error could be related to and just calls out saying Unspecified GSS faliure . "TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information ()"   reference : https://community.cloudera.com/t5/Interactive-Short-cycle-SQL/kerberos-authentication-failure-GSSAPI-Failure-gss-accept-sec/m-p/24707#M742   snap shot from CM: =============== Error   ==== Apr 19, 9:45:16.664 AM INFO logging.cc:117 stdout will be logged to this file. Apr 19, 9:45:16.664 AM ERROR logging.cc:118 stderr will be logged to this file. Apr 19, 9:45:16.664 AM INFO minidump.cc:199 Setting minidump size limit to 20971520. Apr 19, 9:45:16.664 AM INFO atomicops-internals-x86.cc:93 vendor GenuineIntel family 6 model 15 sse2 1 cmpxchg16b 1 Apr 19, 9:45:16.673 AM INFO authentication.cc:678 Using internal kerberos principal "impala/CM-PR-BGD-M02.telemovil.com.sv@TELEMOVIL.COM.SV" Apr 19, 9:45:16.673 AM INFO authentication.cc:1013 Internal communication is authenticated with Kerberos Apr 19, 9:45:16.673 AM INFO authentication.cc:798 Waiting for Kerberos ticket for principal: impala/CM-PR-BGD-M02.telemovil.com.sv@TELEMOVIL.COM.SV Apr 19, 9:45:16.673 AM INFO authentication.cc:494 Registering impala/CM-PR-BGD-M02.telemovil.com.sv@TELEMOVIL.COM.SV, keytab file /var/run/cloudera-scm-agent/process/804-impala-STATESTORE/impala.keytab Apr 19, 9:45:16.780 AM INFO authentication.cc:800 Kerberos ticket granted to impala/CM-PR-BGD-M02.telemovil.com.sv@TELEMOVIL.COM.SV Apr 19, 9:45:16.780 AM INFO authentication.cc:678 Using external kerberos principal "impala/CM-PR-BGD-M02.telemovil.com.sv@TELEMOVIL.COM.SV" Apr 19, 9:45:16.780 AM INFO authentication.cc:1029 External communication is authenticated with Kerberos Apr 19, 9:45:16.780 AM INFO init.cc:201 statestored version 2.6.0-cdh5.8.4 RELEASE (build 207450616f75adbe082a4c2e1145a2384da83fa6) Built on Mon, 06 Feb 2017 14:33:04 PST Apr 19, 9:45:16.780 AM INFO init.cc:202 Using hostname: CM-PR-BGD-M02 Apr 19, 9:45:16.780 AM INFO logging.cc:153 Flags (see also /varz are on debug webserver): --catalog_service_port=26000 --load_catalog_in_background=false --num_metadata_loading_threads=16 --sentry_config= --asm_module_dir= --disable_optimization_passes=false --dump_ir=false --opt_module_dir= --perf_map=false --print_llvm_ir_instruction_count=false --unopt_module_dir= --abort_on_config_error=true --be_port=22000 --be_principal= --compact_catalog_topic=false --disable_mem_pools=false --enable_accept_queue_server=true --enable_process_lifetime_heap_profiling=false --heap_profile_dir= --hostname=CM-PR-BGD-M02 --inc_stats_size_limit_bytes=209715200 --keytab_file=/var/run/cloudera-scm-agent/process/804-impala-STATESTORE/impala.keytab --krb5_conf= --krb5_debug_file= --load_auth_to_local_rules=false --max_minidumps=9 --mem_limit=80% --minidump_path=/var/log/impala-minidumps/statestored --minidump_size_limit_hint_kb=20480 --principal=impala/CM-PR-BGD-M02.telemovil.com.sv@TELEMOVIL.COM.SV --redaction_rules_file= --max_log_files=10 --pause_monitor_sleep_time_ms=500 --pause_monitor_warn_threshold_ms=10000 --log_filename=statestored --redirect_stdout_stderr=true --data_source_batch_size=1024 --exchg_node_buffer_size_bytes=10485760 --enable_partitioned_aggregation=true --enable_partitioned_hash_join=true --enable_probe_side_filtering=true --enable_quadratic_probing=true --skip_lzo_version_check=false --convert_legacy_hive_parquet_utc_timestamps=false --max_page_header_size=8388608 --parquet_min_filter_reject_ratio=0.10000000000000001 --max_row_batches=0 --runtime_filter_wait_time_ms=1000 --suppress_unknown_disk_id_warnings=false --kudu_max_row_batches=0 --kudu_scanner_keep_alive_period_us=15000000 --kudu_scanner_keep_alive_period_sec=15 --kudu_scanner_timeout_sec=60 --pick_only_leaders_for_tests=false --kudu_session_timeout_seconds=60 --enable_phj_probe_side_filtering=true --accepted_cnxn_queue_depth=10000 --enable_ldap_auth=false --internal_principals_whitelist=hdfs --kerberos_reinit_interval=60 --ldap_allow_anonymous_binds=false --ldap_baseDN= --ldap_bind_pattern= --ldap_ca_certificate= --ldap_domain= --ldap_manual_config=false --ldap_passwords_in_clear_ok=false --ldap_tls=false --ldap_uri= --sasl_path= --rpc_cnxn_attempts=10 --rpc_cnxn_retry_interval_ms=2000 --disk_spill_encryption=false --insert_inherit_permissions=false --datastream_sender_timeout_ms=120000 --max_cached_file_handles=0 --max_free_io_buffers=128 --min_buffer_size=1024 --num_disks=0 --num_remote_hdfs_io_threads=8 --num_s3_io_threads=16 --num_threads_per_disk=0 --read_size=8388608 --backend_client_connection_num_retries=3 --backend_client_rpc_timeout_ms=300000 --catalog_client_connection_num_retries=3 --catalog_client_rpc_timeout_ms=0 --catalog_service_host=localhost --cgroup_hierarchy_path= --coordinator_rpc_threads=12 --enable_rm=false --enable_webserver=true --llama_addresses= --llama_callback_port=28000 --llama_host= --llama_max_request_attempts=5 --llama_port=15000 --llama_registration_timeout_secs=30 --llama_registration_wait_secs=3 --num_hdfs_worker_threads=16 --resource_broker_cnxn_attempts=1 --resource_broker_cnxn_retry_interval_ms=3000 --resource_broker_recv_timeout=0 --resource_broker_send_timeout=0 --staging_cgroup=impala_staging --state_store_host=localhost --state_store_subscriber_port=23000 --use_statestore=true --s3a_access_key_cmd= --s3a_secret_key_cmd= --local_library_dir=/tmp --serialize_batch=false --status_report_interval=5 --max_filter_error_rate=0.75 --num_threads_per_core=3 --use_local_tz_for_unix_timestamp_conversions=false --scratch_dirs=/tmp --queue_wait_timeout_ms=60000 --max_vcore_oversubscription_ratio=2.5 --rm_mem_expansion_timeout_ms=5000 --rm_always_use_defaults=false --rm_default_cpu_vcores=2 --rm_default_memory=4G --default_pool_max_queued=200 --default_pool_max_requests=-1 --default_pool_mem_limit= --disable_pool_max_requests=false --disable_pool_mem_limits=false --fair_scheduler_allocation_path= --llama_site_path= --require_username=false --disable_admission_control=false --log_mem_usage_interval=0 --authorization_policy_file= --authorization_policy_provider_class=org.apache.sentry.provider.common.HadoopGroupResourceAuthorizationProvider --authorized_proxy_user_config= --authorized_proxy_user_config_delimiter=, --server_name= --abort_on_failed_audit_event=true --abort_on_failed_lineage_event=true --audit_event_log_dir= --be_service_threads=64 --beeswax_port=21000 --cancellation_thread_pool_size=5 --default_query_options= --fe_service_threads=64 --hs2_port=21050 --idle_query_timeout=0 --idle_session_timeout=0 --lineage_event_log_dir= --local_nodemanager_url= --log_query_to_file=true --max_audit_event_log_file_size=5000 --max_lineage_log_file_size=5000 --max_profile_log_file_size=5000 --max_profile_log_files=10 --max_result_cache_size=100000 --profile_log_dir= --query_log_size=25 --ssl_client_ca_certificate= --ssl_private_key= --ssl_private_key_password_cmd= --ssl_server_certificate= --statestore_subscriber_cnxn_attempts=10 --statestore_subscriber_cnxn_retry_interval_ms=3000 --statestore_subscriber_timeout_seconds=30 --state_store_port=24000 --statestore_heartbeat_frequency_ms=1000 --statestore_heartbeat_tcp_timeout_seconds=3 --statestore_max_missed_heartbeats=10 --statestore_num_heartbeat_threads=10 --statestore_num_update_threads=10 --statestore_update_frequency_ms=2000 --statestore_update_tcp_timeout_seconds=300 --force_lowercase_usernames=false --num_cores=0 --web_log_bytes=1048576 --non_impala_java_vlog=0 --periodic_counter_update_period_ms=500 --enable_webserver_doc_root=true --webserver_authentication_domain= --webserver_certificate_file= --webserver_doc_root=/opt/cloudera/parcels/CDH-5.8.4-1.cdh5.8.4.p0.5/lib/impala --webserver_interface= --webserver_password_file= --webserver_port=25010 --webserver_private_key_file= --webserver_private_key_password_cmd= --webserver_x_frame_options=DENY --flagfile=/var/run/cloudera-scm-agent/process/804-impala-STATESTORE/impala-conf/state_store_flags --fromenv= --tryfromenv= --undefok= --tab_completion_columns=80 --tab_completion_word= --help=false --helpfull=false --helpmatch= --helpon= --helppackage=false --helpshort=false --helpxml=false --version=false --alsologtoemail= --alsologtostderr=false --drop_log_memory=true --log_backtrace_at= --log_dir=/var/log/statestore --log_link= --log_prefix=true --logbuflevel=0 --logbufsecs=30 --logemaillevel=999 --logmailer=/bin/mail --logtostderr=false --max_log_size=200 --minloglevel=0 --stderrthreshold=4 --stop_logging_if_full_disk=false --symbolize_stacktrace=true --v=1 --vmodule= Apr 19, 9:45:16.780 AM INFO init.cc:207 Cpu Info: Model: Intel(R) Xeon(R) CPU E5-2640 v3 @ 2.60GHz Cores: 32 L1 Cache: 32.00 KB (Line: 64.00 B) L2 Cache: 256.00 KB (Line: 64.00 B) L3 Cache: 20.00 MB (Line: 64.00 B) Hardware Supports: ssse3 sse4_1 sse4_2 popcnt Apr 19, 9:45:16.780 AM INFO init.cc:208 Disk Info: Num disks 5: sda (rotational=true) sdc (rotational=true) sdb (rotational=true) sdd (rotational=true) dm- (rotational=true) Apr 19, 9:45:16.780 AM INFO init.cc:209 Physical Memory: 251.89 GB Apr 19, 9:45:16.780 AM INFO init.cc:210 OS version: Linux version 2.6.32-573.el6.x86_64 (mockbuild@c6b9.bsys.dev.centos.org) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-16) (GCC) ) #1 SMP Thu Jul 23 15:44:03 UTC 2015 Clock: clocksource: 'tsc', clockid_t: CLOCK_MONOTONIC Apr 19, 9:45:16.780 AM INFO init.cc:211 Process ID: 3489 Apr 19, 9:45:16.780 AM INFO webserver.cc:219 Starting webserver on 0.0.0.0:25010 Apr 19, 9:45:16.780 AM INFO webserver.cc:233 Document root: /opt/cloudera/parcels/CDH-5.8.4-1.cdh5.8.4.p0.5/lib/impala Apr 19, 9:45:16.781 AM INFO webserver.cc:317 Webserver started Apr 19, 9:45:16.784 AM INFO thrift-server.cc:445 ThriftServer 'StatestoreService' started on port: 24000 Apr 19, 9:45:19.259 AM ERROR authentication.cc:155 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:19.259 AM INFO thrift-util.cc:109 TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:19.344 AM ERROR authentication.cc:155 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:19.344 AM INFO thrift-util.cc:109 TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:19.364 AM ERROR authentication.cc:155 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:19.364 AM INFO thrift-util.cc:109 TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:19.396 AM ERROR authentication.cc:155 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:19.396 AM INFO thrift-util.cc:109 TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:19.404 AM ERROR authentication.cc:155 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:19.404 AM INFO thrift-util.cc:109 TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:22.263 AM ERROR authentication.cc:155 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:22.263 AM INFO thrift-util.cc:109 TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:22.347 AM ERROR authentication.cc:155 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:22.347 AM INFO thrift-util.cc:109 TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:22.366 AM ERROR authentication.cc:155 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:22.367 AM INFO thrift-util.cc:109 TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:22.401 AM ERROR authentication.cc:155 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:22.401 AM INFO thrift-util.cc:109 TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:22.407 AM ERROR authentication.cc:155 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:22.407 AM INFO thrift-util.cc:109 TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:28.539 AM ERROR authentication.cc:155 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:28.539 AM INFO thrift-util.cc:109 TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:29.153 AM ERROR authentication.cc:155 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:29.153 AM INFO thrift-util.cc:109 TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:29.198 AM ERROR authentication.cc:155 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:29.198 AM INFO thrift-util.cc:109 TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:29.203 AM ERROR authentication.cc:155 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:29.203 AM INFO thrift-util.cc:109 TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:29.207 AM ERROR authentication.cc:155 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:29.207 AM INFO thrift-util.cc:109 TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:31.543 AM ERROR authentication.cc:155 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:31.543 AM INFO thrift-util.cc:109 TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:32.155 AM ERROR authentication.cc:155 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:32.156 AM INFO thrift-util.cc:109 TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:32.200 AM ERROR authentication.cc:155 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:32.201 AM INFO thrift-util.cc:109 TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:32.206 AM ERROR authentication.cc:155 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:32.206 AM INFO thrift-util.cc:109 TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:32.209 AM ERROR authentication.cc:155 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:32.209 AM INFO thrift-util.cc:109 TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:38.821 AM ERROR authentication.cc:155 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:38.821 AM INFO thrift-util.cc:109 TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:39.986 AM ERROR authentication.cc:155 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:39.987 AM INFO thrift-util.cc:109 TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:39.987 AM ERROR authentication.cc:155 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:39.988 AM INFO thrift-util.cc:109 TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:39.993 AM ERROR authentication.cc:155 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:39.993 AM INFO thrift-util.cc:109 TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:40.006 AM ERROR authentication.cc:155 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:40.007 AM INFO thrift-util.cc:109 TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:41.824 AM ERROR authentication.cc:155 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:41.824 AM INFO thrift-util.cc:109 TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:42.989 AM ERROR authentication.cc:155 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:42.990 AM INFO thrift-util.cc:109 TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:42.990 AM ERROR authentication.cc:155 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:42.991 AM INFO thrift-util.cc:109 TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:42.996 AM ERROR authentication.cc:155 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:42.996 AM INFO thrift-util.cc:109 TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:43.009 AM ERROR authentication.cc:155 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:43.009 AM INFO thrift-util.cc:109 TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:46.780 AM INFO logging-support.cc:136 Old log file deleted during log rotation: /var/log/statestore/statestored.CM-PR-BGD-M02.impala.log.INFO.20170412-102422.26964 Apr 19, 9:45:46.780 AM INFO logging-support.cc:136 Old log file deleted during log rotation: /var/log/statestore/statestored.CM-PR-BGD-M02.impala.log.WARNING.20170412-102422.26964 Apr 19, 9:45:46.780 AM INFO logging-support.cc:136 Old log file deleted during log rotation: /var/log/statestore/statestored.CM-PR-BGD-M02.impala.log.ERROR.20170412-102422.26964 Apr 19, 9:45:50.107 AM ERROR authentication.cc:155 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:50.107 AM INFO thrift-util.cc:109 TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:51.879 AM ERROR authentication.cc:155 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:51.880 AM INFO thrift-util.cc:109 TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:51.886 AM ERROR authentication.cc:155 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:51.887 AM INFO thrift-util.cc:109 TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:51.905 AM ERROR authentication.cc:155 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:51.905 AM INFO thrift-util.cc:109 TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:51.964 AM ERROR authentication.cc:155 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:51.964 AM INFO thrift-util.cc:109 TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:53.111 AM ERROR authentication.cc:155 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:53.111 AM INFO thrift-util.cc:109 TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:54.882 AM ERROR authentication.cc:155 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:54.883 AM INFO thrift-util.cc:109 TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:54.889 AM ERROR authentication.cc:155 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:54.890 AM INFO thrift-util.cc:109 TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:54.908 AM ERROR authentication.cc:155 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:54.908 AM INFO thrift-util.cc:109 TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:54.967 AM ERROR authentication.cc:155 SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () Apr 19, 9:45:54.967 AM INFO thrift-util.cc:109 TAcceptQueueServer: Caught TException: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information () ... View more

ERROR from falcon application log ================= After installing falcon I see SERVICE UNAVAILABLE ERROR on falcon UI . On restart of falcon service see the below error which points hdfs:hdfs is user & group ownership and falcon cannot execute . 017-02-03 20:10:46,874 INFO ipc.Server (Server.java:logException(2394)) - IPC Server handler 105 on 8020, call org.apache.hadoop.hdfs.protocol.ClientProtocol.getFileInfo from 192. 168.1.95:43132 Call#0 Retry#0: org.apache.hadoop.security.AccessControlException: Permission denied: user=falcon, access=EXECUTE, inode="/apps/falcon/extensions":hdfs:hdfs:drwx----- Below is the output of hadoop fs -ls ... [root@hdpkdc ~]# hadoop fs -ls /apps/falcon Found 1 items drwxr-xr-x - falcon users 0 2017-02-02 20:53 /apps/falcon/extensions [root@hdpkdc ~]# hadoop fs -ls /apps/falcon/extensions Found 4 items drwxr-xr-x - falcon users 0 2017-02-02 20:53 /apps/falcon/extensions/hdfs-mirroring drwxr-xr-x - falcon users 0 2017-02-02 20:53 /apps/falcon/extensions/hdfs-snapshot-mirroring drwxr-xr-x - falcon users 0 2017-02-02 20:53 /apps/falcon/extensions/hive-mirroring drwxrwx--- - falcon users 0 2017-02-02 20:53 /apps/falcon/extensions/mirroring Thanks, Jagdish ... View more

Installing Ranger from using Ambari Wizard gets stuck at stage where masters are assigned . After hitting Next the wizard does not move to the next screen. Ambari Version : 2.4.2 1. Which logs would provide details about the progress , couldn't find much details under ambari-server logs . ... View more

user tickets are valid and have been verified , but when the below commands are executed below is the error that comes up . ================ [<username@<hostname> ~]$ export HBASE_CONF_PATH=/etc/hbase/conf:/etc/hadoop/conf [<username@<hostname> ~]$ /usr/hdp/current/phoenix-client/bin/sqlline.py <zookeeper-node>:2181:/hbase-secure:<user-principal>:<user keytab> =============== Mon Aug 29 13:44:54 CDT 2016, RpcRetryingCaller{globalStartTime=1472495354480, pause=100, retries=35}, org.apache.hadoop.hbase.MasterNotRunningException: com.google.protobuf.ServiceException: java.io.IOException: Couldn't setup connection for <USERNAME>@<REALM> to hbase/<FQDN>@<REALM> at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:147) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3917) at org.apache.hadoop.hbase.client.HBaseAdmin.getTableDescriptor(HBaseAdmin.java:441) at org.apache.hadoop.hbase.client.HBaseAdmin.getTableDescriptor(HBaseAdmin.java:463) at org.apache.phoenix.query.ConnectionQueryServicesImpl.ensureTableCreated(ConnectionQueryServicesImpl.java:815) ... 31 more Caused by: org.apache.hadoop.hbase.MasterNotRunningException: com.google.protobuf.ServiceException: java.io.IOException: Couldn't setup connection for <USERNAME>@<REALM> to hbase/<FQDN>@<REALM> at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1533) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1553) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1704) at org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:38) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:124) ... 35 more Caused by: com.google.protobuf.ServiceException: java.io.IOException: Couldn't setup connection for <USERNAME>@<REALM> to hbase/<FQDN>@<REALM> at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:223) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java:50918) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.isMasterRunning(ConnectionManager.java:1564) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStubNoRetries(ConnectionManager.java:1502) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1524) ... 39 more Caused by: java.io.IOException: Couldn't setup connection for <USERNAME>@<REALM> to hbase/<FQDN>@<REALM> at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$1.run(RpcClientImpl.java:665) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.handleSaslConnectionFailure(RpcClientImpl.java:637) at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:745) at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.writeRequest(RpcClientImpl.java:887) at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.tracedWriteRequest(RpcClientImpl.java:856) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1200) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:213) ... 44 more Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)] at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212) at org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:179) at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupSaslConnection(RpcClientImpl.java:611) at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.access$600(RpcClientImpl.java:156) at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:737) at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:734) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:734) ... 48 more Caused by: GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds) at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:710) at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248) at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193) ... 57 more Caused by: KrbException: Fail to create credential. (63) - No service creds at sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:282) at sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:456) at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:641) ... 60 more sqlline version 1.1.8 ... View more

Hello, We are testing SOLR 5.2.1 , below error pops up when creating a collection with more than 1 shard & more than 1 replica. The process fails after it creates 1 shard as it is unable to remove the write lock that is created under /solr/index on hdfs. 745698 [OverseerThreadFactory-4-thread-1-processing-{node_name=<hostname>:8983_solr}] ERROR org.apache.solr.cloud.OverseerCollectionProcessor [Test ] – Error from shard: http://<hostname>:8983/solr org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://<hostname>:8983/solr: Error CREATEing SolrCore 'Test_shard1_replica1': Unable to create core [Test_shard1_replica1] Caused by: Index locked for write for core Test_shard1_replica1 at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:560) at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:235) at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:227) at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1220) at org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:218) at org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:183) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) Thanks, Jagdish ... View more

Hello I have ATLAS 0.5.0 version deployed . I want to enable user login to the ATLAS web portal . Is there a way I can enable this ? ... View more

I have two questions both seem similar to me . 1. I have Ambari 2.1.2 installed on 2 environments . The same version of ambari is missing "Atlas" software under "ADD SERVICE" option . I am not sure why this would happen . 2. On a different environment we have Ambari 2.1.0 installed which show "SOLR" package, but on higher version 2.1.2 "SOLR" is missing . Thanks Jagdish ... View more

I am looking to delete tags from existing Atlas instance . Apache or Hortonworks doesn't have a comprehensive documentation on how to delete tags using REST API (https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.4/bk_data_governance/content/ch_app_metadata_store_ref.html) . Can you please help. ... View more

using a pig script to upload data . Below is the yarn app log : 2016-01-26 10:57:59,797 INFO [main-SendThread(am2rlccmrhdn04.r1-core.r1.aig.net:2181)] org.apache.zookeeper.ClientCnxn: Session establishment complete on server am2rlccmrhdn04.r1-core.r1.aig.net/10.175.68.14:2181, sessionid = 0x251c236ef7b0093, negotiated timeout = 30000 2016-01-26 10:57:59,924 INFO [main] org.apache.hadoop.hbase.mapreduce.TableOutputFormat: Created table instance for test 2016-01-26 10:57:59,951 INFO [main] org.apache.hadoop.mapred.Task: Using ResourceCalculatorProcessTree : [ ] 2016-01-26 10:58:00,413 INFO [main] org.apache.hadoop.mapred.MapTask: Processing split: Number of splits :1 Total Length = 739 Input split[0]: Length = 739 ClassName: org.apache.hadoop.mapreduce.lib.input.FileSplit Locations: ----------------------- 2016-01-26 10:58:00,443 INFO [main] org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader: Current split being processed hdfs://dr-gfat/user/u1448739/hbase_text.txt:0+739 2016-01-26 10:58:00,570 INFO [main] org.apache.pig.data.SchemaTupleBackend: Key [pig.schematuple] was not set... will not generate code. 2016-01-26 10:58:00,657 INFO [main] org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapOnly$Map: Aliases being processed per job phase (AliasName[line,offset]): M: raw_data[1,11],raw_data[-1,-1] C: R: 2016-01-26 10:58:00,713 WARN [main] org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigHadoopLogger: org.apache.pig.builtin.Utf8StorageConverter(FIELD_DISCARDED_TYPE_CONVERSION_FAILED): Unable to interpret value [32, 97, 103, 101] in field being converted to int, caught NumberFormatException <For input string: "age"> field discarded 2016-01-26 10:58:00,730 INFO [main] org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x251c236ef7b0093 2016-01-26 10:58:00,733 INFO [main] org.apache.zookeeper.ZooKeeper: Session: 0x251c236ef7b0093 closed 2016-01-26 10:58:00,733 INFO [main-EventThread] org.apache.zookeeper.ClientCnxn: EventThread shut down 2016-01-26 10:58:00,735 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.lang.IndexOutOfBoundsException: Index: 1, Size: 1 at java.util.ArrayList.rangeCheck(ArrayList.java:635) at java.util.ArrayList.get(ArrayList.java:411) at org.apache.pig.backend.hadoop.hbase.HBaseStorage.putNext(HBaseStorage.java:947) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:136) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:95) at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:655) at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89) at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapOnly$Map.collect(PigMapOnly.java:48) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.runPipeline(PigGenericMapBase.java:285) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:278) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:64) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) 2016-01-26 10:58:00,747 INFO [main] org.apache.hadoop.mapred.Task: Runnning cleanup for the task ... View more