Hi Miying,   That resolved the issue. Thanks for the help! 🙂   Regards, Cibi ... View more

Hi Miying,   No i'm still facing the problem.   Thanks, Cibi ... View more

Hi Sailesh,   Did you had time to look at the logs file details you requested?   Thanks, Cibi ... View more

Adding Catalogd and Impalad logs:   Catalogd Log:   Log file created at: 2016/09/25 01:59:08 Running on machine: Catalod_host Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg I0925 01:59:08.209684 16058 logging.cc:119] stdout will be logged to this file. E0925 01:59:08.209851 16058 logging.cc:120] stderr will be logged to this file. I0925 01:59:08.214717 16058 authentication.cc:692] Using internal kerberos principal "impala/Catalod_host_FQDN@SPARTA.COM" I0925 01:59:08.214726 16058 authentication.cc:1027] Internal communication is authenticated with Kerberos I0925 01:59:08.214850 16107 authentication.cc:491] Registering impala/Catalod_host_FQDN@SPARTA.COM, keytab file /var/run/cloudera-scm-agent/process/1102-impala-CATALOGSERVER/impala.keytab I0925 01:59:08.214891 16058 authentication.cc:812] Waiting for Kerberos ticket for principal: impala/Catalod_host_FQDN@SPARTA.COM I0925 01:59:08.436725 16058 authentication.cc:814] Kerberos ticket granted to impala/Catalod_host_FQDN@SPARTA.COM I0925 01:59:08.436791 16058 authentication.cc:692] Using external kerberos principal "impala/Catalod_host_FQDN@SPARTA.COM" I0925 01:59:08.436797 16058 authentication.cc:1043] External communication is authenticated with Kerberos I0925 01:59:08.437022 16058 init.cc:158] catalogd version 2.3.0-cdh5.5.4 RELEASE (build e65ded24350974ae6b4e475557b358b718fad29e) Built on Mon, 25 Apr 2016 11:30:47 PST I0925 01:59:08.437029 16058 init.cc:159] Using hostname: Catalod_host I0925 01:59:08.437994 16058 logging.cc:155] Flags (see also /varz are on debug webserver): --catalog_service_port=26000 --load_catalog_in_background=true --num_metadata_loading_threads=16 --sentry_config= --disable_optimization_passes=false --dump_ir=false --opt_module_dir= --print_llvm_ir_instruction_count=false --unopt_module_dir= --abort_on_config_error=true --be_port=22000 --be_principal= --compact_catalog_topic=false --disable_mem_pools=false --enable_process_lifetime_heap_profiling=false --heap_profile_dir= --hostname=Catalod_host --keytab_file=/var/run/cloudera-scm-agent/process/1102-impala-CATALOGSERVER/impala.keytab --krb5_conf= --krb5_debug_file= --mem_limit=80% --principal=impala/Catalod_host_FQDN@SPARTA.COM --redaction_rules_file= --max_log_files=10 --log_filename=catalogd --redirect_stdout_stderr=true --data_source_batch_size=1024 --exchg_node_buffer_size_bytes=10485760 --enable_partitioned_aggregation=true --enable_partitioned_hash_join=true --enable_probe_side_filtering=true --enable_quadratic_probing=true --skip_lzo_version_check=false --convert_legacy_hive_parquet_utc_timestamps=false --max_page_header_size=8388608 --max_row_batches=0 --debug_disable_streaming_gzip=false --enable_phj_probe_side_filtering=true --enable_ldap_auth=false --internal_principals_whitelist=hdfs --kerberos_reinit_interval=60 --ldap_allow_anonymous_binds=false --ldap_baseDN= --ldap_bind_pattern= --ldap_ca_certificate= --ldap_domain= --ldap_manual_config=false --ldap_passwords_in_clear_ok=false --ldap_tls=false --ldap_uri= --sasl_path=/usr/lib/sasl2:/usr/lib64/sasl2:/usr/local/lib/sasl2:/usr/lib/x86_64-linux-gnu/sasl2 --rpc_cnxn_attempts=10 --rpc_cnxn_retry_interval_ms=2000 --disk_spill_encryption=false --insert_inherit_permissions=false --max_cached_file_handles=0 --max_free_io_buffers=128 --min_buffer_size=1024 --num_disks=0 --num_remote_hdfs_io_threads=8 --num_s3_io_threads=16 --num_threads_per_disk=0 --read_size=8388608 --catalog_service_host=localhost --cgroup_hierarchy_path= --enable_rm=false --enable_webserver=true --llama_addresses= --llama_callback_port=28000 --llama_host= --llama_max_request_attempts=5 --llama_port=15000 --llama_registration_timeout_secs=30 --llama_registration_wait_secs=3 --num_hdfs_worker_threads=16 --resource_broker_cnxn_attempts=1 --resource_broker_cnxn_retry_interval_ms=3000 --resource_broker_recv_timeout=0 --resource_broker_send_timeout=0 --staging_cgroup=impala_staging --state_store_host=statestore_host_FQDN --state_store_subscriber_port=23020 --use_statestore=true --local_library_dir=/tmp --serialize_batch=false --status_report_interval=5 --num_threads_per_core=3 --use_local_tz_for_unix_timestamp_conversions=false --scratch_dirs=/tmp --queue_wait_timeout_ms=60000 --default_pool_max_queued=200 --default_pool_max_requests=200 --default_pool_mem_limit= --disable_pool_max_requests=false --disable_pool_mem_limits=false --fair_scheduler_allocation_path= --llama_site_path= --log_mem_usage_interval=0 --authorization_policy_file= --authorization_policy_provider_class=org.apache.sentry.provider.common.HadoopGroupResourceAuthorizationProvider --authorized_proxy_user_config= --authorized_proxy_user_config_delimiter=, --load_catalog_at_startup=false --server_name= --abort_on_failed_audit_event=true --abort_on_failed_lineage_event=true --audit_event_log_dir= --be_service_threads=64 --beeswax_port=21000 --cancellation_thread_pool_size=5 --default_query_options= --fe_service_threads=64 --hs2_port=21050 --idle_query_timeout=0 --idle_session_timeout=0 --lineage_event_log_dir= --local_nodemanager_url= --log_query_to_file=true --max_audit_event_log_file_size=5000 --max_lineage_log_file_size=5000 --max_profile_log_file_size=5000 --max_result_cache_size=100000 --profile_log_dir= --query_log_size=25 --ssl_client_ca_certificate= --ssl_private_key= --ssl_private_key_password_cmd= --ssl_server_certificate= --max_vcore_oversubscription_ratio=2.5 --rm_always_use_defaults=false --rm_default_cpu_vcores=2 --rm_default_memory=4G --disable_admission_control=true --require_username=false --statestore_subscriber_cnxn_attempts=10 --statestore_subscriber_cnxn_retry_interval_ms=3000 --statestore_subscriber_timeout_seconds=30 --state_store_port=24000 --statestore_heartbeat_frequency_ms=1000 --statestore_heartbeat_tcp_timeout_seconds=3 --statestore_max_missed_heartbeats=10 --statestore_num_heartbeat_threads=10 --statestore_num_update_threads=10 --statestore_update_frequency_ms=2000 --statestore_update_tcp_timeout_seconds=300 --force_lowercase_usernames=false --num_cores=0 --web_log_bytes=1048576 --non_impala_java_vlog=0 --periodic_counter_update_period_ms=500 --enable_webserver_doc_root=true --webserver_authentication_domain= --webserver_certificate_file= --webserver_doc_root=/vol_01/cloudera/parcels/CDH-5.5.4-1.cdh5.5.4.p0.9/lib/impala --webserver_interface= --webserver_password_file= --webserver_port=25020 --webserver_private_key_file= --webserver_private_key_password_cmd= --flagfile=/var/run/cloudera-scm-agent/process/1102-impala-CATALOGSERVER/impala-conf/catalogserver_flags --fromenv= --tryfromenv= --undefok= --tab_completion_columns=80 --tab_completion_word= --help=false --helpfull=false --helpmatch= --helpon= --helppackage=false --helpshort=false --helpxml=false --version=false --alsologtoemail= --alsologtostderr=false --drop_log_memory=true --log_backtrace_at= --log_dir=/vol_01/log/catalogd --log_link= --log_prefix=true --logbuflevel=0 --logbufsecs=30 --logbufvlevel=1 --logemaillevel=999 --logmailer=/bin/mail --logtostderr=false --max_log_size=200 --minloglevel=0 --stderrthreshold=4 --stop_logging_if_full_disk=false --symbolize_stacktrace=true --v=1 --vmodule= I0925 01:59:08.438065 16058 init.cc:164] Cpu Info: Model: Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz Cores: 4 L1 Cache: 32.00 KB L2 Cache: 256.00 KB L3 Cache: 25.00 MB Hardware Supports: ssse3 sse4_1 sse4_2 popcnt I0925 01:59:08.438073 16058 init.cc:165] Disk Info: Num disks 3: sda (rotational=true) sdb (rotational=true) dm- (rotational=true) I0925 01:59:08.438082 16058 init.cc:166] Physical Memory: 15.58 GB I0925 01:59:08.438087 16058 init.cc:167] OS version: Linux version 2.6.32-573.18.1.el6.x86_64 (mockbuild@x86-010.build.bos.redhat.com) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-16) (GCC) ) #1 SMP Wed Jan 6 11:20:49 EST 2016 I0925 01:59:08.438091 16058 init.cc:168] Process ID: 16058 I0925 01:59:09.604663 16058 webserver.cc:210] Starting webserver on 0.0.0.0:25020 I0925 01:59:09.604691 16058 webserver.cc:224] Document root: /vol_01/cloudera/parcels/CDH-5.5.4-1.cdh5.5.4.p0.9/lib/impala I0925 01:59:09.604823 16058 webserver.cc:309] Webserver started I0925 01:59:09.640769 16058 GlogAppender.java:123] Logging initialized. Impala: VLOG, All other: INFO I0925 01:59:09.641646 16058 JniCatalog.java:89] Java Version Info: Java(TM) SE Runtime Environment (1.7.0_25-b15) I0925 01:59:10.155648 16107 authentication.cc:519] Successfully renewed Keberos ticket I0925 01:59:10.170464 16058 MetaStoreClientPool.java:55] Creating MetaStoreClient. Pool Size = 0 I0925 01:59:10.182169 16058 HiveMetaStoreClient.java:376] Trying to connect to metastore with URI thrift://Catalod_host_FQDN:9083 I0925 01:59:10.565619 16058 HiveMetaStoreClient.java:421] Opened a connection to metastore, current connections: 1 I0925 01:59:10.566120 16058 HiveMetaStoreClient.java:473] Connected to metastore. I0925 01:59:10.566236 16058 MetaStoreClientPool.java:55] Creating MetaStoreClient. Pool Size = 1 I0925 01:59:10.566793 16058 HiveMetaStoreClient.java:376] Trying to connect to metastore with URI thrift://Catalod_host_FQDN:9083 I0925 01:59:10.578852 16058 HiveMetaStoreClient.java:421] Opened a connection to metastore, current connections: 2 I0925 01:59:10.579704 16058 HiveMetaStoreClient.java:473] Connected to metastore. I0925 01:59:10.579993 16058 MetaStoreClientPool.java:55] Creating MetaStoreClient. Pool Size = 2 I0925 01:59:10.580477 16058 HiveMetaStoreClient.java:376] Trying to connect to metastore with URI thrift://Catalod_host_FQDN:9083 I0925 01:59:10.592774 16058 HiveMetaStoreClient.java:421] Opened a connection to metastore, current connections: 3 I0925 01:59:10.593262 16058 HiveMetaStoreClient.java:473] Connected to metastore. I0925 01:59:10.593372 16058 MetaStoreClientPool.java:55] Creating MetaStoreClient. Pool Size = 3 I0925 01:59:10.595728 16058 HiveMetaStoreClient.java:376] Trying to connect to metastore with URI thrift://Catalod_host_FQDN:9083 I0925 01:59:10.606948 16058 HiveMetaStoreClient.java:421] Opened a connection to metastore, current connections: 4 I0925 01:59:10.607393 16058 HiveMetaStoreClient.java:473] Connected to metastore. I0925 01:59:10.607458 16058 MetaStoreClientPool.java:55] Creating MetaStoreClient. Pool Size = 4 I0925 01:59:10.607743 16058 HiveMetaStoreClient.java:376] Trying to connect to metastore with URI thrift://Catalod_host_FQDN:9083 I0925 01:59:10.614889 16058 HiveMetaStoreClient.java:421] Opened a connection to metastore, current connections: 5 I0925 01:59:10.615327 16058 HiveMetaStoreClient.java:473] Connected to metastore. I0925 01:59:10.834429 16058 MetaStoreClientPool.java:55] Creating MetaStoreClient. Pool Size = 4 I0925 01:59:10.834760 16058 HiveMetaStoreClient.java:376] Trying to connect to metastore with URI thrift://Catalod_host_FQDN:9083 I0925 01:59:10.842993 16058 HiveMetaStoreClient.java:421] Opened a connection to metastore, current connections: 6 I0925 01:59:10.843364 16058 HiveMetaStoreClient.java:473] Connected to metastore. I0925 01:59:10.871893 16058 HiveMetaStoreClient.java:502] Closed a connection to metastore, current connections: 5 I0925 01:59:10.873522 16131 TableLoadingMgr.java:282] Loading next table. Remaining items in queue: 0 I0925 01:59:10.873736 16132 TableLoadingMgr.java:282] Loading next table. Remaining items in queue: 2 I0925 01:59:10.873814 16133 TableLoadingMgr.java:282] Loading next table. Remaining items in queue: 1 I0925 01:59:10.873906 16134 TableLoadingMgr.java:282] Loading next table. Remaining items in queue: 0 I0925 01:59:10.873977 16135 TableLoadingMgr.java:282] Loading next table. Remaining items in queue: 0 I0925 01:59:10.874081 16136 TableLoadingMgr.java:282] Loading next table. Remaining items in queue: 0 I0925 01:59:10.874209 16137 TableLoadingMgr.java:282] Loading next table. Remaining items in queue: 0 I0925 01:59:10.874346 16138 TableLoadingMgr.java:282] Loading next table. Remaining items in queue: 2 I0925 01:59:10.880722 16058 statestore-subscriber.cc:179] Starting statestore subscriber I0925 01:59:10.882082 16149 TableLoader.java:60] Loading metadata for: core.product I0925 01:59:10.882200 16149 MetaStoreClientPool.java:55] Creating MetaStoreClient. Pool Size = 3 I0925 01:59:10.882367 16152 TableLoader.java:60] Loading metadata for: core.student I0925 01:59:10.882501 16149 HiveMetaStoreClient.java:376] Trying to connect to metastore with URI thrift://Catalod_host_FQDN:9083 I0925 01:59:10.882658 16153 TableLoader.java:60] Loading metadata for: core.product_bucketed I0925 01:59:10.886121 16058 thrift-server.cc:431] ThriftServer 'StatestoreSubscriber' started on port: 23020 I0925 01:59:10.886129 16058 statestore-subscriber.cc:190] Registering with statestore I0925 01:59:10.889982 16149 HiveMetaStoreClient.java:421] Opened a connection to metastore, current connections: 6 I0925 01:59:10.890362 16149 HiveMetaStoreClient.java:473] Connected to metastore. I0925 01:59:10.890480 16153 MetaStoreClientPool.java:55] Creating MetaStoreClient. Pool Size = 1 I0925 01:59:10.890766 16153 HiveMetaStoreClient.java:376] Trying to connect to metastore with URI thrift://Catalod_host_FQDN:9083 I0925 01:59:10.891439 16154 TableLoader.java:60] Loading metadata for: core.c I0925 01:59:10.891670 16155 TableLoader.java:60] Loading metadata for: core.b I0925 01:59:10.891916 16139 TableLoadingMgr.java:282] Loading next table. Remaining items in queue: 0 I0925 01:59:10.892029 16156 TableLoader.java:60] Loading metadata for: core.product1 I0925 01:59:10.892103 16140 TableLoadingMgr.java:282] Loading next table. Remaining items in queue: 0 I0925 01:59:10.892426 16157 TableLoader.java:60] Loading metadata for: core.a I0925 01:59:10.898161 16153 HiveMetaStoreClient.java:421] Opened a connection to metastore, current connections: 7 I0925 01:59:10.898551 16153 HiveMetaStoreClient.java:473] Connected to metastore. I0925 01:59:10.898646 16152 MetaStoreClientPool.java:55] Creating MetaStoreClient. Pool Size = 0 I0925 01:59:10.898927 16152 HiveMetaStoreClient.java:376] Trying to connect to metastore with URI thrift://Catalod_host_FQDN:9083 I0925 01:59:10.900521 16158 TableLoader.java:60] Loading metadata for: core.test1 I0925 01:59:10.902216 16160 TableLoader.java:60] Loading metadata for: core.test I0925 01:59:10.903240 16159 TableLoader.java:60] Loading metadata for: core.product2b I0925 01:59:10.906101 16152 HiveMetaStoreClient.java:421] Opened a connection to metastore, current connections: 8 I0925 01:59:10.906488 16152 HiveMetaStoreClient.java:473] Connected to metastore. I0925 01:59:10.906679 16159 MetaStoreClientPool.java:55] Creating MetaStoreClient. Pool Size = 0 I0925 01:59:10.906947 16159 HiveMetaStoreClient.java:376] Trying to connect to metastore with URI thrift://Catalod_host_FQDN:9083 I0925 01:59:10.914718 16159 HiveMetaStoreClient.java:421] Opened a connection to metastore, current connections: 9 I0925 01:59:10.915083 16159 HiveMetaStoreClient.java:473] Connected to metastore. I0925 01:59:10.915179 16160 MetaStoreClientPool.java:55] Creating MetaStoreClient. Pool Size = 0 I0925 01:59:10.915460 16160 HiveMetaStoreClient.java:376] Trying to connect to metastore with URI thrift://Catalod_host_FQDN:9083 I0925 01:59:10.923107 16160 HiveMetaStoreClient.java:421] Opened a connection to metastore, current connections: 10 I0925 01:59:10.923576 16160 HiveMetaStoreClient.java:473] Connected to metastore. I0925 01:59:10.923689 16158 MetaStoreClientPool.java:55] Creating MetaStoreClient. Pool Size = 0 I0925 01:59:10.923995 16158 HiveMetaStoreClient.java:376] Trying to connect to metastore with URI thrift://Catalod_host_FQDN:9083 I0925 01:59:10.931267 16158 HiveMetaStoreClient.java:421] Opened a connection to metastore, current connections: 11 I0925 01:59:10.931654 16158 HiveMetaStoreClient.java:473] Connected to metastore. I0925 01:59:10.931756 16157 MetaStoreClientPool.java:55] Creating MetaStoreClient. Pool Size = 0 I0925 01:59:10.932042 16157 HiveMetaStoreClient.java:376] Trying to connect to metastore with URI thrift://Catalod_host_FQDN:9083 I0925 01:59:10.939201 16157 HiveMetaStoreClient.java:421] Opened a connection to metastore, current connections: 12 I0925 01:59:10.939621 16157 HiveMetaStoreClient.java:473] Connected to metastore. I0925 01:59:10.939684 16150 catalog-server.cc:313] Publishing update: DATABASE:core@1 I0925 01:59:10.939714 16150 catalog-server.cc:313] Publishing update: TABLE:core.test@10 I0925 01:59:10.939726 16156 MetaStoreClientPool.java:55] Creating MetaStoreClient. Pool Size = 0 I0925 01:59:10.939764 16150 catalog-server.cc:313] Publishing update: TABLE:core.product1@6 I0925 01:59:10.939771 16150 catalog-server.cc:313] Publishing update: TABLE:core.a@2 I0925 01:59:10.939776 16150 catalog-server.cc:313] Publishing update: TABLE:core.b@3 I0925 01:59:10.939782 16150 catalog-server.cc:313] Publishing update: TABLE:core.product@5 I0925 01:59:10.939787 16150 catalog-server.cc:313] Publishing update: TABLE:core.c@4 I0925 01:59:10.939792 16150 catalog-server.cc:313] Publishing update: TABLE:core.product2b@7 I0925 01:59:10.939796 16150 catalog-server.cc:313] Publishing update: TABLE:core.test1@11 I0925 01:59:10.939802 16150 catalog-server.cc:313] Publishing update: TABLE:core.product_bucketed@8 I0925 01:59:10.939807 16150 catalog-server.cc:313] Publishing update: TABLE:core.student@9 I0925 01:59:10.939812 16150 catalog-server.cc:313] Publishing update: DATABASE:default@12 I0925 01:59:10.939820 16150 catalog-server.cc:313] Publishing update: CATALOG:d71b5b5808594fcd:afaa46f809705c83@12 I0925 01:59:10.940039 16156 HiveMetaStoreClient.java:376] Trying to connect to metastore with URI thrift://Catalod_host_FQDN:9083 I0925 01:59:10.946593 16156 HiveMetaStoreClient.java:421] Opened a connection to metastore, current connections: 13 I0925 01:59:10.946977 16156 HiveMetaStoreClient.java:473] Connected to metastore. I0925 01:59:10.947360 16155 MetaStoreClientPool.java:55] Creating MetaStoreClient. Pool Size = 0 I0925 01:59:10.947641 16155 HiveMetaStoreClient.java:376] Trying to connect to metastore with URI thrift://Catalod_host_FQDN:9083 I0925 01:59:10.954646 16155 HiveMetaStoreClient.java:421] Opened a connection to metastore, current connections: 14 I0925 01:59:10.955044 16155 HiveMetaStoreClient.java:473] Connected to metastore. I0925 01:59:10.955150 16154 MetaStoreClientPool.java:55] Creating MetaStoreClient. Pool Size = 0 I0925 01:59:10.955973 16154 HiveMetaStoreClient.java:376] Trying to connect to metastore with URI thrift://Catalod_host_FQDN:9083 I0925 01:59:10.964249 16154 HiveMetaStoreClient.java:421] Opened a connection to metastore, current connections: 15 I0925 01:59:10.964658 16154 HiveMetaStoreClient.java:473] Connected to metastore. I0925 01:59:11.043257 16158 HdfsTable.java:1013] load table: core.test1 I0925 01:59:11.044705 16157 HdfsTable.java:1013] load table: core.a I0925 01:59:11.044816 16159 HdfsTable.java:1013] load table: core.product2b I0925 01:59:11.044934 16152 HdfsTable.java:1013] load table: core.student I0925 01:59:11.045044 16153 HdfsTable.java:1013] load table: core.product_bucketed I0925 01:59:11.045115 16154 HdfsTable.java:1013] load table: core.c I0925 01:59:11.045204 16155 HdfsTable.java:1013] load table: core.b I0925 01:59:11.045284 16156 HdfsTable.java:1013] load table: core.product1 I0925 01:59:11.045358 16149 HdfsTable.java:1013] load table: core.product I0925 01:59:11.055538 16160 HdfsTable.java:1013] load table: core.test I0925 01:59:11.106195 16058 status.cc:112] Couldn't open transport for statestore_host_FQDN:24000 (No more data to read.) @ 0x7aeb93 (unknown) @ 0xbc5cb4 (unknown) @ 0xbc5d93 (unknown) @ 0x9a4bb6 (unknown) @ 0x9a5891 (unknown) @ 0xa6ca83 (unknown) @ 0xa6ea5c (unknown) @ 0x782405 (unknown) @ 0x77b8a6 (unknown) @ 0x3192e1ed5d (unknown) @ 0x77b1b9 (unknown) I0925 01:59:11.106235 16058 thrift-client.cc:55] Unable to connect to statestore_host_FQDN:24000 I0925 01:59:11.106240 16058 thrift-client.cc:61] (Attempt 1 of 10) I0925 01:59:11.111660 16157 Table.java:155] Loading column stats for table: a I0925 01:59:11.114917 16155 Table.java:155] Loading column stats for table: b I0925 01:59:11.116050 16160 Table.java:155] Loading column stats for table: test I0925 01:59:11.116945 16154 Table.java:155] Loading column stats for table: c I0925 01:59:11.118154 16158 Table.java:155] Loading column stats for table: test1 I0925 01:59:11.119752 16149 Table.java:155] Loading column stats for table: product I0925 01:59:11.123488 16156 Table.java:155] Loading column stats for table: product1 I0925 01:59:11.124166 16153 Table.java:155] Loading column stats for table: product_bucketed I0925 01:59:11.124459 16152 Table.java:155] Loading column stats for table: student I0925 01:59:11.124534 16159 Table.java:155] Loading column stats for table: product2b I0925 01:59:11.910172 16160 HiveMetaStoreClient.java:502] Closed a connection to metastore, current connections: 14 I0925 01:59:11.910339 16152 HiveMetaStoreClient.java:502] Closed a connection to metastore, current connections: 13 I0925 01:59:11.910528 16158 HiveMetaStoreClient.java:502] Closed a connection to metastore, current connections: 12 I0925 01:59:11.910676 16155 HiveMetaStoreClient.java:502] Closed a connection to metastore, current connections: 11 I0925 01:59:11.910926 16154 HiveMetaStoreClient.java:502] Closed a connection to metastore, current connections: 10 I0925 01:59:11.911077 16157 HiveMetaStoreClient.java:502] Closed a connection to metastore, current connections: 9 I0925 01:59:11.911568 16159 HiveMetaStoreClient.java:502] Closed a connection to metastore, current connections: 8 I0925 01:59:11.911725 16153 HiveMetaStoreClient.java:502] Closed a connection to metastore, current connections: 7 I0925 01:59:11.912005 16156 HiveMetaStoreClient.java:502] Closed a connection to metastore, current connections: 6 I0925 01:59:14.106725 16058 client-cache.cc:80] ReopenClient(): re-creating client for statestore_host_FQDN:24000 I0925 01:59:14.110769 16058 status.cc:112] Couldn't open transport for statestore_host_FQDN:24000 (No more data to read.) @ 0x7aeb93 (unknown) @ 0xbc5cb4 (unknown) @ 0xbc5d93 (unknown) @ 0x9a4bb6 (unknown) @ 0x9a5254 (unknown) @ 0xa6d5c3 (unknown) @ 0xa6ea5c (unknown) @ 0x782405 (unknown) @ 0x77b8a6 (unknown) @ 0x3192e1ed5d (unknown) @ 0x77b1b9 (unknown) I0925 01:59:14.110781 16058 thrift-client.cc:55] Unable to connect to statestore_host_FQDN:24000 I0925 01:59:14.110785 16058 thrift-client.cc:61] (Attempt 1 of 10) I0925 01:59:16.646400 16149 HiveMetaStoreClient.java:502] Closed a connection to metastore, current connections: 5 I0925 01:59:17.111428 16058 status.cc:45] RPC Error: No more data to read. @ 0x7ae3c6 (unknown) @ 0xa6d724 (unknown) @ 0xa6ea5c (unknown) @ 0x782405 (unknown) @ 0x77b8a6 (unknown) @ 0x3192e1ed5d (unknown) @ 0x77b1b9 (unknown) I0925 01:59:17.111462 16058 statestore-subscriber.cc:196] statestore registration unsuccessful: RPC Error: No more data to read. E0925 01:59:17.111650 16058 catalogd-main.cc:76] RPC Error: No more data to read.   Impalad Log:   Log file created at: 2016/09/25 01:59:12 Running on machine: Impalad_host Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg I0925 01:59:12.879591 2702404 logging.cc:119] stdout will be logged to this file. E0925 01:59:12.879758 2702404 logging.cc:120] stderr will be logged to this file. I0925 01:59:12.884112 2702404 authentication.cc:692] Using internal kerberos principal "impala/Impalad_host_FQDN@SPARTA.COM" I0925 01:59:12.884125 2702404 authentication.cc:1027] Internal communication is authenticated with Kerberos I0925 01:59:12.884238 2702458 authentication.cc:491] Registering impala/Impalad_host_FQDN@SPARTA.COM, keytab file /var/run/cloudera-scm-agent/process/1100-impala-IMPALAD/impala.keytab I0925 01:59:12.884248 2702404 authentication.cc:812] Waiting for Kerberos ticket for principal: impala/Impalad_host_FQDN@SPARTA.COM I0925 01:59:13.102747 2702404 authentication.cc:814] Kerberos ticket granted to impala/Impalad_host_FQDN@SPARTA.COM I0925 01:59:13.102812 2702404 authentication.cc:692] Using external kerberos principal "impala/Impalad_host_FQDN@SPARTA.COM" I0925 01:59:13.102818 2702404 authentication.cc:1043] External communication is authenticated with Kerberos I0925 01:59:13.103026 2702404 init.cc:158] impalad version 2.3.0-cdh5.5.4 RELEASE (build e65ded24350974ae6b4e475557b358b718fad29e) Built on Mon, 25 Apr 2016 11:30:47 PST I0925 01:59:13.103032 2702404 init.cc:159] Using hostname: Impalad_host_FQDN I0925 01:59:13.103698 2702404 logging.cc:155] Flags (see also /varz are on debug webserver): --catalog_service_port=26000 --load_catalog_in_background=false --num_metadata_loading_threads=16 --sentry_config= --disable_optimization_passes=false --dump_ir=false --opt_module_dir= --print_llvm_ir_instruction_count=false --unopt_module_dir= --abort_on_config_error=true --be_port=22000 --be_principal= --compact_catalog_topic=false --disable_mem_pools=false --enable_process_lifetime_heap_profiling=false --heap_profile_dir= --hostname=Impalad_host_FQDN --keytab_file=/var/run/cloudera-scm-agent/process/1100-impala-IMPALAD/impala.keytab --krb5_conf= --krb5_debug_file= --mem_limit=4328521728 --principal=impala/Impalad_host_FQDN@SPARTA.COM --redaction_rules_file= --max_log_files=10 --log_filename=impalad --redirect_stdout_stderr=true --data_source_batch_size=1024 --exchg_node_buffer_size_bytes=10485760 --enable_partitioned_aggregation=true --enable_partitioned_hash_join=true --enable_probe_side_filtering=true --enable_quadratic_probing=true --skip_lzo_version_check=false --convert_legacy_hive_parquet_utc_timestamps=false --max_page_header_size=8388608 --max_row_batches=0 --debug_disable_streaming_gzip=false --enable_phj_probe_side_filtering=true --enable_ldap_auth=false --internal_principals_whitelist=hdfs --kerberos_reinit_interval=60 --ldap_allow_anonymous_binds=false --ldap_baseDN= --ldap_bind_pattern= --ldap_ca_certificate= --ldap_domain= --ldap_manual_config=false --ldap_passwords_in_clear_ok=false --ldap_tls=false --ldap_uri= --sasl_path=/usr/lib/sasl2:/usr/lib64/sasl2:/usr/local/lib/sasl2:/usr/lib/x86_64-linux-gnu/sasl2 --rpc_cnxn_attempts=10 --rpc_cnxn_retry_interval_ms=2000 --disk_spill_encryption=false --insert_inherit_permissions=false --max_cached_file_handles=0 --max_free_io_buffers=128 --min_buffer_size=1024 --num_disks=0 --num_remote_hdfs_io_threads=8 --num_s3_io_threads=16 --num_threads_per_disk=0 --read_size=8388608 --catalog_service_host=catalaogd_host_FQDN --cgroup_hierarchy_path= --enable_rm=false --enable_webserver=true --llama_addresses= --llama_callback_port=28000 --llama_host= --llama_max_request_attempts=5 --llama_port=15000 --llama_registration_timeout_secs=30 --llama_registration_wait_secs=3 --num_hdfs_worker_threads=16 --resource_broker_cnxn_attempts=1 --resource_broker_cnxn_retry_interval_ms=3000 --resource_broker_recv_timeout=0 --resource_broker_send_timeout=0 --staging_cgroup=impala_staging --state_store_host=Impalad_host_FQDN --state_store_subscriber_port=23000 --use_statestore=true --local_library_dir=/vol_01/lib/impala/udfs --serialize_batch=false --status_report_interval=5 --num_threads_per_core=3 --use_local_tz_for_unix_timestamp_conversions=false --scratch_dirs=/vol_01/impala/impalad --queue_wait_timeout_ms=60000 --default_pool_max_queued=200 --default_pool_max_requests=200 --default_pool_mem_limit= --disable_pool_max_requests=false --disable_pool_mem_limits=false --fair_scheduler_allocation_path= --llama_site_path= --log_mem_usage_interval=0 --authorization_policy_file= --authorization_policy_provider_class=org.apache.sentry.provider.common.HadoopGroupResourceAuthorizationProvider --authorized_proxy_user_config= --authorized_proxy_user_config_delimiter=, --load_catalog_at_startup=false --server_name= --abort_on_failed_audit_event=true --abort_on_failed_lineage_event=true --audit_event_log_dir= --be_service_threads=64 --beeswax_port=21000 --cancellation_thread_pool_size=5 --default_query_options= --fe_service_threads=64 --hs2_port=21050 --idle_query_timeout=0 --idle_session_timeout=0 --lineage_event_log_dir=/vol_01/log/impalad/lineage --local_nodemanager_url= --log_query_to_file=true --max_audit_event_log_file_size=5000 --max_lineage_log_file_size=5000 --max_profile_log_file_size=5000 --max_result_cache_size=100000 --profile_log_dir= --query_log_size=25 --ssl_client_ca_certificate= --ssl_private_key= --ssl_private_key_password_cmd= --ssl_server_certificate= --max_vcore_oversubscription_ratio=2.5 --rm_always_use_defaults=false --rm_default_cpu_vcores=2 --rm_default_memory=4G --disable_admission_control=true --require_username=false --statestore_subscriber_cnxn_attempts=10 --statestore_subscriber_cnxn_retry_interval_ms=3000 --statestore_subscriber_timeout_seconds=30 --state_store_port=24000 --statestore_heartbeat_frequency_ms=1000 --statestore_heartbeat_tcp_timeout_seconds=3 --statestore_max_missed_heartbeats=10 --statestore_num_heartbeat_threads=10 --statestore_num_update_threads=10 --statestore_update_frequency_ms=2000 --statestore_update_tcp_timeout_seconds=300 --force_lowercase_usernames=false --num_cores=0 --web_log_bytes=1048576 --non_impala_java_vlog=0 --periodic_counter_update_period_ms=500 --enable_webserver_doc_root=true --webserver_authentication_domain= --webserver_certificate_file= --webserver_doc_root=/vol_01/cloudera/parcels/CDH-5.5.4-1.cdh5.5.4.p0.9/lib/impala --webserver_interface= --webserver_password_file= --webserver_port=25000 --webserver_private_key_file= --webserver_private_key_password_cmd= --flagfile=/var/run/cloudera-scm-agent/process/1100-impala-IMPALAD/impala-conf/impalad_flags --fromenv= --tryfromenv= --undefok= --tab_completion_columns=80 --tab_completion_word= --help=false --helpfull=false --helpmatch= --helpon= --helppackage=false --helpshort=false --helpxml=false --version=false --alsologtoemail= --alsologtostderr=false --drop_log_memory=true --log_backtrace_at= --log_dir=/vol_01/log/impalad --log_link= --log_prefix=true --logbuflevel=0 --logbufsecs=30 --logbufvlevel=1 --logemaillevel=999 --logmailer=/bin/mail --logtostderr=false --max_log_size=200 --minloglevel=0 --stderrthreshold=4 --stop_logging_if_full_disk=false --symbolize_stacktrace=true --v=1 --vmodule= I0925 01:59:13.103770 2702404 init.cc:164] Cpu Info: Model: Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz Cores: 4 L1 Cache: 32.00 KB L2 Cache: 256.00 KB L3 Cache: 25.00 MB Hardware Supports: ssse3 sse4_1 sse4_2 popcnt I0925 01:59:13.103778 2702404 init.cc:165] Disk Info: Num disks 3: sda (rotational=true) sdb (rotational=true) dm- (rotational=true) I0925 01:59:13.103785 2702404 init.cc:166] Physical Memory: 15.58 GB I0925 01:59:13.103790 2702404 init.cc:167] OS version: Linux version 2.6.32-573.18.1.el6.x86_64 (mockbuild@x86-010.build.bos.redhat.com) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-16) (GCC) ) #1 SMP Wed Jan 6 11:20:49 EST 2016 I0925 01:59:13.103793 2702404 init.cc:168] Process ID: 2702404 I0925 01:59:14.821723 2702458 authentication.cc:519] Successfully renewed Keberos ticket I0925 01:59:15.032460 2702404 hbase-table-scanner.cc:157] Detected HBase version >= 0.95.2 I0925 01:59:15.142060 2702404 GlogAppender.java:123] Logging initialized. Impala: VLOG, All other: INFO I0925 01:59:15.144978 2702404 JniFrontend.java:130] Authorization is 'DISABLED'. I0925 01:59:15.145108 2702404 JniFrontend.java:132] Java Version Info: Java(TM) SE Runtime Environment (1.7.0_67-b01) I0925 01:59:15.734531 2702404 simple-scheduler.cc:103] Admission control is disabled. I0925 01:59:15.735497 2702404 impala-server.cc:1091] Default query options:TQueryOptions { 01: abort_on_error (bool) = false, 02: max_errors (i32) = 0, 03: disable_codegen (bool) = false, 04: batch_size (i32) = 0, 05: num_nodes (i32) = 0, 06: max_scan_range_length (i64) = 0, 07: num_scanner_threads (i32) = 0, 08: max_io_buffers (i32) = 0, 09: allow_unsupported_formats (bool) = false, 10: default_order_by_limit (i64) = -1, 11: debug_action (string) = "", 12: mem_limit (i64) = 0, 13: abort_on_default_limit_exceeded (bool) = false, 15: hbase_caching (i32) = 0, 16: hbase_cache_blocks (bool) = false, 17: parquet_file_size (i64) = 0, 18: explain_level (i32) = 1, 19: sync_ddl (bool) = false, 23: disable_cached_reads (bool) = false, 24: disable_outermost_topn (bool) = false, 25: rm_initial_mem (i64) = 0, 26: query_timeout_s (i32) = 0, 28: appx_count_distinct (bool) = false, 29: disable_unsafe_spills (bool) = false, 31: exec_single_node_rows_threshold (i32) = 100, } I0925 01:59:17.514686 2702404 tmp-file-mgr.cc:106] Using scratch directory /vol_01/impala/impalad/impala-scratch on disk 2 I0925 01:59:17.514786 2702404 simple-logger.cc:76] Logging to: /vol_01/log/impalad/profiles//impala_profile_log_1.1-1474786757514 I0925 01:59:17.514906 2702404 impala-server.cc:485] Event logging is disabled I0925 01:59:17.514976 2702404 simple-logger.cc:76] Logging to: /vol_01/log/impalad/lineage/impala_lineage_log_1.0-1474786757514 I0925 01:59:17.530464 2702404 impala-server.cc:1722] Impala Beeswax Service listening on 21000 I0925 01:59:17.531194 2702404 impala-server.cc:1744] Impala HiveServer2 Service listening on 21050 I0925 01:59:17.531886 2702404 impala-server.cc:1764] ImpalaInternalService listening on 22000 I0925 01:59:17.532097 2702404 thrift-server.cc:431] ThriftServer 'backend' started on port: 22000 I0925 01:59:17.532107 2702404 exec-env.cc:296] Starting global services I0925 01:59:17.543001 2702404 exec-env.cc:383] Using global memory limit: 4.03 GB I0925 01:59:17.544988 2702404 webserver.cc:210] Starting webserver on 0.0.0.0:25000 I0925 01:59:17.544999 2702404 webserver.cc:224] Document root: /vol_01/cloudera/parcels/CDH-5.5.4-1.cdh5.5.4.p0.9/lib/impala I0925 01:59:17.545121 2702404 webserver.cc:309] Webserver started I0925 01:59:17.545142 2702404 simple-scheduler.cc:182] Starting simple scheduler I0925 01:59:17.545276 2702404 simple-scheduler.cc:230] Simple-scheduler using Statestore_Host_IP as IP address I0925 01:59:17.545287 2702404 statestore-subscriber.cc:179] Starting statestore subscriber I0925 01:59:17.546177 2702404 thrift-server.cc:431] ThriftServer 'StatestoreSubscriber' started on port: 23000 I0925 01:59:17.546185 2702404 statestore-subscriber.cc:190] Registering with statestore I0925 01:59:17.764664 2702404 status.cc:112] Couldn't open transport for Impalad_host_FQDN:24000 (No more data to read.) @ 0x79f523 (unknown) @ 0xbcdc24 (unknown) @ 0xbcdd03 (unknown) @ 0x9a00f6 (unknown) @ 0x9a0dd1 (unknown) @ 0xa6e6b3 (unknown) @ 0xa7068c (unknown) @ 0x963e66 (unknown) @ 0x77b57c (unknown) @ 0x313081ed5d (unknown) @ 0x77b1b9 (unknown) I0925 01:59:17.764691 2702404 thrift-client.cc:55] Unable to connect to Impalad_host_FQDN:24000 I0925 01:59:17.764695 2702404 thrift-client.cc:61] (Attempt 1 of 10) I0925 01:59:20.764950 2702404 client-cache.cc:80] ReopenClient(): re-creating client for Impalad_host_FQDN:24000 I0925 01:59:20.769177 2702404 status.cc:112] Couldn't open transport for Impalad_host_FQDN:24000 (No more data to read.) @ 0x79f523 (unknown) @ 0xbcdc24 (unknown) @ 0xbcdd03 (unknown) @ 0x9a00f6 (unknown) @ 0x9a0794 (unknown) @ 0xa6f1f3 (unknown) @ 0xa7068c (unknown) @ 0x963e66 (unknown) @ 0x77b57c (unknown) @ 0x313081ed5d (unknown) @ 0x77b1b9 (unknown) I0925 01:59:20.769208 2702404 thrift-client.cc:55] Unable to connect to Impalad_host_FQDN:24000 I0925 01:59:20.769212 2702404 thrift-client.cc:61] (Attempt 1 of 10) I0925 01:59:23.769857 2702404 status.cc:45] RPC Error: No more data to read. @ 0x79ed56 (unknown) @ 0xa6f354 (unknown) @ 0xa7068c (unknown) @ 0x963e66 (unknown) @ 0x77b57c (unknown) @ 0x313081ed5d (unknown) @ 0x77b1b9 (unknown) I0925 01:59:23.769894 2702404 statestore-subscriber.cc:196] statestore registration unsuccessful: RPC Error: No more data to read. E0925 01:59:23.770056 2702404 impalad-main.cc:80] Impalad services did not start correctly, exiting. Error: RPC Error: No more data to read. State Store Subscriber did not start up. ... View more

Hi Sailesh,   Please find the requested log files.   Thanks, Cibi   Statestore log:   Log file created at: 2016/09/25 01:58:34 Running on machine: statestore_host Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg I0925 01:58:34.314796 2701924 logging.cc:119] stdout will be logged to this file. E0925 01:58:34.314962 2701924 logging.cc:120] stderr will be logged to this file. I0925 01:58:34.318999 2701924 authentication.cc:692] Using internal kerberos principal "impala/statestore_host_FQDN@SPARTA.COM" I0925 01:58:34.319008 2701924 authentication.cc:1027] Internal communication is authenticated with Kerberos I0925 01:58:34.319118 2701960 authentication.cc:491] Registering impala/statestore_host_FQDN@SPARTA.COM, keytab file /var/run/cloudera-scm-agent/process/1103-impala-STATESTORE/impala.keytab I0925 01:58:34.319125 2701924 authentication.cc:812] Waiting for Kerberos ticket for principal: impala/statestore_host_FQDN@SPARTA.COM I0925 01:58:34.537320 2701924 authentication.cc:814] Kerberos ticket granted to impala/statestore_host_FQDN@SPARTA.COM I0925 01:58:34.537376 2701924 authentication.cc:692] Using external kerberos principal "impala/statestore_host_FQDN@SPARTA.COM" I0925 01:58:34.537382 2701924 authentication.cc:1043] External communication is authenticated with Kerberos I0925 01:58:34.537528 2701924 init.cc:158] statestored version 2.3.0-cdh5.5.4 RELEASE (build e65ded24350974ae6b4e475557b358b718fad29e) Built on Mon, 25 Apr 2016 11:30:47 PST I0925 01:58:34.537534 2701924 init.cc:159] Using hostname: statestore_host I0925 01:58:34.538158 2701924 logging.cc:155] Flags (see also /varz are on debug webserver): --catalog_service_port=26000 --load_catalog_in_background=false --num_metadata_loading_threads=16 --sentry_config= --disable_optimization_passes=false --dump_ir=false --opt_module_dir= --print_llvm_ir_instruction_count=false --unopt_module_dir= --abort_on_config_error=true --be_port=22000 --be_principal= --compact_catalog_topic=false --disable_mem_pools=false --enable_process_lifetime_heap_profiling=false --heap_profile_dir= --hostname=statestore_host --keytab_file=/var/run/cloudera-scm-agent/process/1103-impala-STATESTORE/impala.keytab --krb5_conf= --krb5_debug_file= --mem_limit=80% --principal=impala/statestore_host_FQDN@SPARTA.COM --redaction_rules_file= --max_log_files=10 --log_filename=statestored --redirect_stdout_stderr=true --data_source_batch_size=1024 --exchg_node_buffer_size_bytes=10485760 --enable_partitioned_aggregation=true --enable_partitioned_hash_join=true --enable_probe_side_filtering=true --enable_quadratic_probing=true --skip_lzo_version_check=false --convert_legacy_hive_parquet_utc_timestamps=false --max_page_header_size=8388608 --max_row_batches=0 --debug_disable_streaming_gzip=false --enable_phj_probe_side_filtering=true --enable_ldap_auth=false --internal_principals_whitelist=hdfs --kerberos_reinit_interval=60 --ldap_allow_anonymous_binds=false --ldap_baseDN= --ldap_bind_pattern= --ldap_ca_certificate= --ldap_domain= --ldap_manual_config=false --ldap_passwords_in_clear_ok=false --ldap_tls=false --ldap_uri= --sasl_path=/usr/lib/sasl2:/usr/lib64/sasl2:/usr/local/lib/sasl2:/usr/lib/x86_64-linux-gnu/sasl2 --rpc_cnxn_attempts=10 --rpc_cnxn_retry_interval_ms=2000 --disk_spill_encryption=false --insert_inherit_permissions=false --max_cached_file_handles=0 --max_free_io_buffers=128 --min_buffer_size=1024 --num_disks=0 --num_remote_hdfs_io_threads=8 --num_s3_io_threads=16 --num_threads_per_disk=0 --read_size=8388608 --catalog_service_host=localhost --cgroup_hierarchy_path= --enable_rm=false --enable_webserver=true --llama_addresses= --llama_callback_port=28000 --llama_host= --llama_max_request_attempts=5 --llama_port=15000 --llama_registration_timeout_secs=30 --llama_registration_wait_secs=3 --num_hdfs_worker_threads=16 --resource_broker_cnxn_attempts=1 --resource_broker_cnxn_retry_interval_ms=3000 --resource_broker_recv_timeout=0 --resource_broker_send_timeout=0 --staging_cgroup=impala_staging --state_store_host=localhost --state_store_subscriber_port=23000 --use_statestore=true --local_library_dir=/tmp --serialize_batch=false --status_report_interval=5 --num_threads_per_core=3 --use_local_tz_for_unix_timestamp_conversions=false --scratch_dirs=/tmp --queue_wait_timeout_ms=60000 --default_pool_max_queued=200 --default_pool_max_requests=200 --default_pool_mem_limit= --disable_pool_max_requests=false --disable_pool_mem_limits=false --fair_scheduler_allocation_path= --llama_site_path= --log_mem_usage_interval=0 --authorization_policy_file= --authorization_policy_provider_class=org.apache.sentry.provider.common.HadoopGroupResourceAuthorizationProvider --authorized_proxy_user_config= --authorized_proxy_user_config_delimiter=, --load_catalog_at_startup=false --server_name= --abort_on_failed_audit_event=true --abort_on_failed_lineage_event=true --audit_event_log_dir= --be_service_threads=64 --beeswax_port=21000 --cancellation_thread_pool_size=5 --default_query_options= --fe_service_threads=64 --hs2_port=21050 --idle_query_timeout=0 --idle_session_timeout=0 --lineage_event_log_dir= --local_nodemanager_url= --log_query_to_file=true --max_audit_event_log_file_size=5000 --max_lineage_log_file_size=5000 --max_profile_log_file_size=5000 --max_result_cache_size=100000 --profile_log_dir= --query_log_size=25 --ssl_client_ca_certificate= --ssl_private_key= --ssl_private_key_password_cmd= --ssl_server_certificate= --max_vcore_oversubscription_ratio=2.5 --rm_always_use_defaults=false --rm_default_cpu_vcores=2 --rm_default_memory=4G --disable_admission_control=true --require_username=false --statestore_subscriber_cnxn_attempts=10 --statestore_subscriber_cnxn_retry_interval_ms=3000 --statestore_subscriber_timeout_seconds=30 --state_store_port=24000 --statestore_heartbeat_frequency_ms=1000 --statestore_heartbeat_tcp_timeout_seconds=3 --statestore_max_missed_heartbeats=10 --statestore_num_heartbeat_threads=10 --statestore_num_update_threads=10 --statestore_update_frequency_ms=2000 --statestore_update_tcp_timeout_seconds=300 --force_lowercase_usernames=false --num_cores=0 --web_log_bytes=1048576 --non_impala_java_vlog=0 --periodic_counter_update_period_ms=500 --enable_webserver_doc_root=true --webserver_authentication_domain= --webserver_certificate_file= --webserver_doc_root=/vol_01/cloudera/parcels/CDH-5.5.4-1.cdh5.5.4.p0.9/lib/impala --webserver_interface= --webserver_password_file= --webserver_port=25010 --webserver_private_key_file= --webserver_private_key_password_cmd= --flagfile=/var/run/cloudera-scm-agent/process/1103-impala-STATESTORE/impala-conf/state_store_flags --fromenv= --tryfromenv= --undefok= --tab_completion_columns=80 --tab_completion_word= --help=false --helpfull=false --helpmatch= --helpon= --helppackage=false --helpshort=false --helpxml=false --version=false --alsologtoemail= --alsologtostderr=false --drop_log_memory=true --log_backtrace_at= --log_dir=/vol_01/log/statestore --log_link= --log_prefix=true --logbuflevel=0 --logbufsecs=30 --logbufvlevel=1 --logemaillevel=999 --logmailer=/bin/mail --logtostderr=false --max_log_size=200 --minloglevel=0 --stderrthreshold=4 --stop_logging_if_full_disk=false --symbolize_stacktrace=true --v=1 --vmodule= I0925 01:58:34.538220 2701924 init.cc:164] Cpu Info: Model: Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz Cores: 4 L1 Cache: 32.00 KB L2 Cache: 256.00 KB L3 Cache: 25.00 MB Hardware Supports: ssse3 sse4_1 sse4_2 popcnt I0925 01:58:34.538228 2701924 init.cc:165] Disk Info: Num disks 3: sda (rotational=true) sdb (rotational=true) dm- (rotational=true) I0925 01:58:34.538235 2701924 init.cc:166] Physical Memory: 15.58 GB I0925 01:58:34.538240 2701924 init.cc:167] OS version: Linux version 2.6.32-573.18.1.el6.x86_64 (mockbuild@x86-010.build.bos.redhat.com) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-16) (GCC) ) #1 SMP Wed Jan 6 11:20:49 EST 2016 I0925 01:58:34.538242 2701924 init.cc:168] Process ID: 2701924 I0925 01:58:34.538297 2701924 webserver.cc:210] Starting webserver on 0.0.0.0:25010 I0925 01:58:34.538313 2701924 webserver.cc:224] Document root: /vol_01/cloudera/parcels/CDH-5.5.4-1.cdh5.5.4.p0.9/lib/impala I0925 01:58:34.538424 2701924 webserver.cc:309] Webserver started I0925 01:58:34.559240 2701924 thrift-server.cc:431] ThriftServer 'StatestoreService' started on port: 24000 I0925 01:58:36.257709 2701960 authentication.cc:519] Successfully renewed Keberos ticket E0925 01:58:37.180516 2701998 authentication.cc:158] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request) I0925 01:58:37.181058 2701998 thrift-util.cc:109] TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context E0925 01:58:37.646294 2701998 authentication.cc:158] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request) I0925 01:58:37.646749 2701998 thrift-util.cc:109] TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context E0925 01:58:39.232471 2701998 authentication.cc:158] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request) I0925 01:58:39.232897 2701998 thrift-util.cc:109] TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context E0925 01:58:39.568882 2701998 authentication.cc:158] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request) I0925 01:58:39.569471 2701998 thrift-util.cc:109] TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context E0925 01:58:40.184850 2701998 authentication.cc:158] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request) I0925 01:58:40.185300 2701998 thrift-util.cc:109] TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context E0925 01:58:40.650048 2701998 authentication.cc:158] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request) I0925 01:58:40.650457 2701998 thrift-util.cc:109] TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context E0925 01:58:42.236927 2701998 authentication.cc:158] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request) I0925 01:58:42.237365 2701998 thrift-util.cc:109] TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context E0925 01:58:42.573323 2701998 authentication.cc:158] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request) I0925 01:58:42.575258 2701998 thrift-util.cc:109] TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context E0925 01:58:47.326885 2701998 authentication.cc:158] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request) I0925 01:58:47.327370 2701998 thrift-util.cc:109] TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context E0925 01:58:48.607534 2701998 authentication.cc:158] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request) I0925 01:58:48.608216 2701998 thrift-util.cc:109] TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context E0925 01:58:50.330967 2701998 authentication.cc:158] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request) I0925 01:58:50.331411 2701998 thrift-util.cc:109] TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context E0925 01:58:50.694953 2701998 authentication.cc:158] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request) I0925 01:58:50.695441 2701998 thrift-util.cc:109] TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context E0925 01:58:51.611749 2701998 authentication.cc:158] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request) I0925 01:58:51.612180 2701998 thrift-util.cc:109] TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context E0925 01:58:52.205765 2701998 authentication.cc:158] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request) I0925 01:58:52.206214 2701998 thrift-util.cc:109] TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context E0925 01:58:53.698994 2701998 authentication.cc:158] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request) I0925 01:58:53.699483 2701998 thrift-util.cc:109] TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context E0925 01:58:55.210412 2701998 authentication.cc:158] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request) I0925 01:58:55.210867 2701998 thrift-util.cc:109] TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context E0925 01:58:58.699170 2701998 authentication.cc:158] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request) I0925 01:58:58.699609 2701998 thrift-util.cc:109] TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context E0925 01:59:00.513659 2701998 authentication.cc:158] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request) I0925 01:59:00.514102 2701998 thrift-util.cc:109] TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context E0925 01:59:01.703227 2701998 authentication.cc:158] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request) I0925 01:59:01.703634 2701998 thrift-util.cc:109] TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context E0925 01:59:03.486690 2701998 authentication.cc:158] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request) I0925 01:59:03.487344 2701998 thrift-util.cc:109] TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context E0925 01:59:03.517756 2701998 authentication.cc:158] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request) I0925 01:59:03.518329 2701998 thrift-util.cc:109] TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context E0925 01:59:05.393232 2701998 authentication.cc:158] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request) I0925 01:59:05.393954 2701998 thrift-util.cc:109] TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context E0925 01:59:06.490890 2701998 authentication.cc:158] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request) I0925 01:59:06.491515 2701998 thrift-util.cc:109] TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context E0925 01:59:08.398248 2701998 authentication.cc:158] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request) I0925 01:59:08.398841 2701998 thrift-util.cc:109] TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context E0925 01:59:11.103200 2701998 authentication.cc:158] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request) I0925 01:59:11.103942 2701998 thrift-util.cc:109] TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context E0925 01:59:13.687938 2701998 authentication.cc:158] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request) I0925 01:59:13.688544 2701998 thrift-util.cc:109] TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context E0925 01:59:14.109560 2701998 authentication.cc:158] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request) I0925 01:59:14.110250 2701998 thrift-util.cc:109] TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context E0925 01:59:16.692235 2701998 authentication.cc:158] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request) I0925 01:59:16.692840 2701998 thrift-util.cc:109] TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context E0925 01:59:17.763322 2701998 authentication.cc:158] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request) I0925 01:59:17.763895 2701998 thrift-util.cc:109] TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context E0925 01:59:19.819659 2701998 authentication.cc:158] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request) I0925 01:59:19.820860 2701998 thrift-util.cc:109] TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context E0925 01:59:20.768082 2701998 authentication.cc:158] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request) I0925 01:59:20.768653 2701998 thrift-util.cc:109] TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context E0925 01:59:22.824905 2701998 authentication.cc:158] SASL message (Kerberos (internal)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request) I0925 01:59:22.825522 2701998 thrift-util.cc:109] TThreadedServer: TServerTransport died on accept: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context     ... View more

Hi Sailesh,   I have the renewable_lifetime value set to 604800, but i don't have a 'kerberos_reinit_interval' Impala startup flag parameter set anywhere.   Does the flag need to be set for Impala in krb5.conf file?   PFB the krb5.conf file contents.   # cat /etc/krb5.conf [libdefaults] default_realm = SPARTA.COM dns_lookup_kdc = false dns_lookup_realm = false ticket_lifetime = 86400 renew_lifetime = 604800 forwardable = true default_tgs_enctypes = rc4-hmac default_tkt_enctypes = rc4-hmac permitted_enctypes = rc4-hmac udp_preference_limit = 1 [realms] SPARTA.COM = { kdc = <kdc_host> admin_server = <kdc_host> }   I can also see the below in the role log file:   Flags (see also /varz are on debug webserver): --catalog_service_port=26000 --load_catalog_in_background=true --num_metadata_loading_threads=16 --sentry_config= --disable_optimization_passes=false --dump_ir=false --opt_module_dir= --print_llvm_ir_instruction_count=false --unopt_module_dir= --abort_on_config_error=true --be_port=22000 --be_principal= --compact_catalog_topic=false --disable_mem_pools=false --enable_process_lifetime_heap_profiling=false --heap_profile_dir= --hostname=ech-10-24-130-11 --keytab_file=/var/run/cloudera-scm-agent/process/977-impala-CATALOGSERVER/impala.keytab --krb5_conf= --krb5_debug_file= --mem_limit=80% --principal=impala/ech-10-24-130-11.mastercard.int@SPARTA.COM --redaction_rules_file= --max_log_files=10 --log_filename=catalogd --redirect_stdout_stderr=true --data_source_batch_size=1024 --exchg_node_buffer_size_bytes=10485760 --enable_partitioned_aggregation=true --enable_partitioned_hash_join=true --enable_probe_side_filtering=true --enable_quadratic_probing=true --skip_lzo_version_check=false --convert_legacy_hive_parquet_utc_timestamps=false --max_page_header_size=8388608 --max_row_batches=0 --debug_disable_streaming_gzip=false --enable_phj_probe_side_filtering=true --enable_ldap_auth=false --internal_principals_whitelist=hdfs --kerberos_reinit_interval=60 --ldap_allow_anonymous_binds=false --ldap_baseDN= --ldap_bind_pattern= --ldap_ca_certificate= --ldap_domain= --ldap_manual_config=false --ldap_passwords_in_clear_ok=false --ldap_tls=false --ldap_uri= --sasl_path=/usr/lib/sasl2:/usr/lib64/sasl2:/usr/local/lib/sasl2:/usr/lib/x86_64-linux-gnu/sasl2 --rpc_cnxn_attempts=10 --rpc_cnxn_retry_interval_ms=2000 --disk_spill_encryption=false --insert_inherit_permissions=false --max_cached_file_handles=0 --max_free_io_buffers=128 --min_buffer_size=1024 --num_disks=0 --num_remote_hdfs_io_threads=8 --num_s3_io_threads=16 --num_threads_per_disk=0 --read_size=8388608 --catalog_service_host=localhost --cgroup_hierarchy_path= --enable_rm=false --enable_webserver=true --llama_addresses= --llama_callback_port=28000 --llama_host= --llama_max_request_attempts=5 --llama_port=15000 --llama_registration_timeout_secs=30 --llama_registration_wait_secs=3 --num_hdfs_worker_threads=16 --resource_broker_cnxn_attempts=1 --resource_broker_cnxn_retry_interval_ms=3000 --resource_broker_recv_timeout=0 --resource_broker_send_timeout=0 --staging_cgroup=impala_staging --state_store_host=ech-10-24-130-74.mastercard.int --state_store_subscriber_port=23020 --use_statestore=true --local_library_dir=/tmp --serialize_batch=false --status_report_interval=5 --num_threads_per_core=3 --use_local_tz_for_unix_timestamp_conversions=false --scratch_dirs=/tmp --queue_wait_timeout_ms=60000 --default_pool_max_queued=200 --default_pool_max_requests=200 --default_pool_mem_limit= --disable_pool_max_requests=false --disable_pool_mem_limits=false --fair_scheduler_allocation_path= --llama_site_path= --log_mem_usage_interval=0 --authorization_policy_file= --authorization_policy_provider_class=org.apache.sentry.provider.common.HadoopGroupResourceAuthorizationProvider --authorized_proxy_user_config= --authorized_proxy_user_config_delimiter=, --load_catalog_at_startup=false --server_name= --abort_on_failed_audit_event=true --abort_on_failed_lineage_event=true --audit_event_log_dir= --be_service_threads=64 --beeswax_port=21000 --cancellation_thread_pool_size=5 --default_query_options= --fe_service_threads=64 --hs2_port=21050 --idle_query_timeout=0 --idle_session_timeout=0 --lineage_event_log_dir= --local_nodemanager_url= --log_query_to_file=true --max_audit_event_log_file_size=5000 --max_lineage_log_file_size=5000 --max_profile_log_file_size=5000 --max_result_cache_size=100000 --profile_log_dir= --query_log_size=25 --ssl_client_ca_certificate= --ssl_private_key= --ssl_private_key_password_cmd= --ssl_server_certificate= --max_vcore_oversubscription_ratio=2.5 --rm_always_use_defaults=false --rm_default_cpu_vcores=2 --rm_default_memory=4G --disable_admission_control=true --require_username=false --statestore_subscriber_cnxn_attempts=10 --statestore_subscriber_cnxn_retry_interval_ms=3000 --statestore_subscriber_timeout_seconds=30 --state_store_port=24000 --statestore_heartbeat_frequency_ms=1000 --statestore_heartbeat_tcp_timeout_seconds=3 --statestore_max_missed_heartbeats=10 --statestore_num_heartbeat_threads=10 --statestore_num_update_threads=10 --statestore_update_frequency_ms=2000 --statestore_update_tcp_timeout_seconds=300 --force_lowercase_usernames=false --num_cores=0 --web_log_bytes=1048576 --non_impala_java_vlog=0 --periodic_counter_update_period_ms=500 --enable_webserver_doc_root=true --webserver_authentication_domain= --webserver_certificate_file= --webserver_doc_root=/vol_01/cloudera/parcels/CDH-5.5.4-1.cdh5.5.4.p0.9/lib/impala --webserver_interface= --webserver_password_file= --webserver_port=25020 --webserver_private_key_file= --webserver_private_key_password_cmd= --flagfile=/var/run/cloudera-scm-agent/process/977-impala-CATALOGSERVER/impala-conf/catalogserver_flags --fromenv= --tryfromenv= --undefok= --tab_completion_columns=80 --tab_completion_word= --help=false --helpfull=false --helpmatch= --helpon= --helppackage=false --helpshort=false --helpxml=false --version=false --alsologtoemail= --alsologtostderr=false --drop_log_memory=true --log_backtrace_at= --log_dir=/vol_01/log/catalogd --log_link= --log_prefix=true --logbuflevel=0 --logbufsecs=30 --logbufvlevel=1 --logemaillevel=999 --logmailer=/bin/mail --logtostderr=false --max_log_size=200 --minloglevel=0 --stderrthreshold=4 --stop_logging_if_full_disk=false --symbolize_stacktrace=true --v=1 --vmodule=   Cpu Info: Model: Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz Cores: 4 L1 Cache: 32.00 KB L2 Cache: 256.00 KB L3 Cache: 25.00 MB Hardware Supports: ssse3 sse4_1 sse4_2 popcnt Disk Info: Num disks 3: sda (rotational=true) sdb (rotational=true) dm- (rotational=true) Physical Memory: 15.58 GB OS version: Linux version 2.6.32-573.18.1.el6.x86_64 (mockbuild@x86-010.build.bos.redhat.com) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-16) (GCC) ) #1 SMP Wed Jan 6 11:20:49 EST 2016 Process ID: 637685 Starting webserver on 0.0.0.0:25020 Document root: /vol_01/cloudera/parcels/CDH-5.5.4-1.cdh5.5.4.p0.9/lib/impala Webserver started Logging initialized. Impala: VLOG, All other: INFO Java Version Info: Java(TM) SE Runtime Environment (1.7.0_25-b15) Creating MetaStoreClient. Pool Size = 0 Trying to connect to metastore with URI thrift://<Hive_metastore_host>:9083 Successfully renewed Keberos ticket Opened a connection to metastore, current connections: 1 Connected to metastore.   Thanks, Cibi ... View more

Hi Sailesh,   My kerberos renewable lifetime is set to 7 days.   Kerberos ticket validity is 10 hours after which it gets expired.   I'm getting this error only in Catalog server and not in Statestore.   Error:   statestore registration unsuccessful: RPC Error: No more data to read. RPC Error: No more data to read.   Thanks, Cibi ... View more

Hi Jason,   I was also facing the same issue of impala services not starting after enabling kerberos. I did added the kerberos authentication types as you suggested but that didn't resolved the problem.   # cat /etc/krb5.conf [libdefaults] default_realm = SPARTA.COM dns_lookup_kdc = false dns_lookup_realm = false ticket_lifetime = 86400 renew_lifetime = 604800 forwardable = true default_tgs_enctypes = rc4-hmac des3-hmac-sha1 arcfour-hmac des-hmac-sha1 des-cbc-md5 des-cbc-crc default_tkt_enctypes = rc4-hmac des3-hmac-sha1 arcfour-hmac des-hmac-sha1 des-cbc-md5 des-cbc-crc permitted_enctypes = rc4-hmac des3-hmac-sha1 arcfour-hmac des-hmac-sha1 des-cbc-md5 des-cbc-crc udp_preference_limit = 1 [realms] SPARTA.COM = { kdc = <kdc-hostname> admin_server = <kdc-hostname> }   Impala catalog server error:   I0912 18:28:45.181133 58918 client-cache.cc:80] ReopenClient(): re-creating client for <Impala_catalogd_host>:24000 I0912 18:28:45.184960 58918 status.cc:112] Couldn't open transport for <Impala_catalogd_host>:24000 (No more data to read.) @ 0x7aeb93 (unknown) @ 0xbc5cb4 (unknown) @ 0xbc5d93 (unknown) @ 0x9a4bb6 (unknown) @ 0x9a5254 (unknown) @ 0xa6d5c3 (unknown) @ 0xa6ea5c (unknown) @ 0x782405 (unknown) @ 0x77b8a6 (unknown) @ 0x3192e1ed5d (unknown) @ 0x77b1b9 (unknown) I0912 18:28:45.184970 58918 thrift-client.cc:55] Unable to connect to <Impala_catalogd_host>:24000 I0912 18:28:45.184974 58918 thrift-client.cc:61] (Attempt 1 of 10) I0912 18:28:48.186059 58918 status.cc:45] RPC Error: No more data to read. @ 0x7ae3c6 (unknown) @ 0xa6d724 (unknown) @ 0xa6ea5c (unknown) @ 0x782405 (unknown) @ 0x77b8a6 (unknown) @ 0x3192e1ed5d (unknown) @ 0x77b1b9 (unknown) I0912 18:28:48.186095 58918 statestore-subscriber.cc:196] statestore registration unsuccessful: RPC Error: No more data to read. E0912 18:28:48.186408 58918 catalogd-main.cc:76] RPC Error: No more data to read.   ... View more