Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

impala catalog server & impala deamons web servers issues

impala catalog server & impala deamons web servers issues

Explorer

Hello !

 

I'm installing CDH5.9.0 on 6 machines (2 masters and 4 workers).

 

I have the Impala service going on red error; impala catalog server and impala deamons are in red status, the impala statestore is in a healthy state (green).

 

I have no data on the cluster, this is just happening after the installation.

 

The other services are all in a good state, and all services databases are well configured.

 

 

Any helps well be very welcome !

 

 

Thank you.

4 REPLIES 4
Highlighted

Re: impala catalog server & impala deamons web servers issues

Champion

it should state you the reason as to why its health are bad in the cloudera manager health test 

 

did you try restarting the impala catalog and impala daemon ? 

were you able to fire impala-shell and excute some simple quries ?

also to troubleshoot check the impala catalog and impala daemon logs. 

Re: impala catalog server & impala deamons web servers issues

Explorer

Thank you csguna,

 

I have tried to restart without but didn't solve the issue.

 

Now I'm running Impala queries from hue or impala-shell without any problem.

 

But the health status of impala deamons and catalog still can not turn to good (green color on CM) ??!!

 

 

I don't grasp excatly what is the problem.

This is the log file of the catalog service

 

Log file created at: 2017/12/28 15:47:44
Running on machine: hd-memento-m2
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
I1228 15:47:44.203192 29911 logging.cc:120] stdout will be logged to this file.
E1228 15:47:44.203328 29911 logging.cc:121] stderr will be logged to this file.
I1228 15:47:44.203564 29911 minidump.cc:222] Setting minidump size limit to 20971520.
I1228 15:47:44.203606 29911 atomicops-internals-x86.cc:93] vendor GenuineIntel family 6 model 15 sse2 1 cmpxchg16b 1
I1228 15:47:44.212666 29911 authentication.cc:1020] Internal communication is not authenticated
I1228 15:47:44.212677 29911 authentication.cc:1041] External communication is not authenticated
I1228 15:47:44.212826 29911 init.cc:204] catalogd version 2.7.0-cdh5.9.0 RELEASE (build 4b4cf1936bd6cdf34fda5e2f32827e7d60c07a9c)
Built on Fri Oct 21 01:07:22 PDT 2016
I1228 15:47:44.212831 29911 init.cc:205] Using hostname: hd-memento-m2
I1228 15:47:44.213428 29911 logging.cc:156] Flags (see also /varz are on debug webserver):
--catalog_service_port=26000
--load_catalog_in_background=true
--num_metadata_loading_threads=16
--sentry_config=
--asm_module_dir=
--disable_optimization_passes=false
--dump_ir=false
--opt_module_dir=
--perf_map=false
--print_llvm_ir_instruction_count=false
--unopt_module_dir=
--abort_on_config_error=true
--be_port=22000
--be_principal=
--compact_catalog_topic=false
--disable_kudu=false
--disable_mem_pools=false
--enable_process_lifetime_heap_profiling=false
--heap_profile_dir=
--hostname=hd-memento-m2
--keytab_file=
--krb5_conf=
--krb5_debug_file=
--load_auth_to_local_rules=false
--max_minidumps=9
--mem_limit=80%
--minidump_path=/var/log/impala-minidumps/catalogd
--minidump_size_limit_hint_kb=20480
--principal=
--redaction_rules_file=
--max_log_files=10
--pause_monitor_sleep_time_ms=500
--pause_monitor_warn_threshold_ms=10000
--log_filename=catalogd
--redirect_stdout_stderr=true
--data_source_batch_size=1024
--exchg_node_buffer_size_bytes=10485760
--enable_partitioned_aggregation=true
--enable_partitioned_hash_join=true
--enable_probe_side_filtering=true
--enable_quadratic_probing=true
--skip_lzo_version_check=false
--parquet_min_filter_reject_ratio=0.10000000000000001
--max_row_batches=0
--runtime_filter_wait_time_ms=1000
--suppress_unknown_disk_id_warnings=false
--kudu_max_row_batches=0
--kudu_scanner_keep_alive_period_us=15000000
--kudu_scanner_keep_alive_period_sec=15
--kudu_scanner_timeout_sec=60
--pick_only_leaders_for_tests=false
--kudu_session_timeout_seconds=60
--convert_legacy_hive_parquet_utc_timestamps=false
--max_page_header_size=8388608
--enable_phj_probe_side_filtering=true
--enable_ldap_auth=false
--internal_principals_whitelist=hdfs
--kerberos_reinit_interval=60
--ldap_allow_anonymous_binds=false
--ldap_baseDN=
--ldap_bind_pattern=
--ldap_ca_certificate=
--ldap_domain=
--ldap_manual_config=false
--ldap_passwords_in_clear_ok=false
--ldap_tls=false
--ldap_uri=
--sasl_path=
--rpc_cnxn_attempts=10
--rpc_cnxn_retry_interval_ms=2000
--disk_spill_encryption=false
--insert_inherit_permissions=false
--datastream_sender_timeout_ms=120000
--max_cached_file_handles=0
--max_free_io_buffers=128
--min_buffer_size=1024
--num_disks=0
--num_remote_hdfs_io_threads=8
--num_s3_io_threads=16
--num_threads_per_disk=0
--read_size=8388608
--backend_client_connection_num_retries=3
--backend_client_rpc_timeout_ms=300000
--catalog_client_connection_num_retries=3
--catalog_client_rpc_timeout_ms=0
--catalog_service_host=localhost
--cgroup_hierarchy_path=
--coordinator_rpc_threads=12
--enable_rm=false
--enable_webserver=true
--llama_addresses=
--llama_callback_port=28000
--llama_host=
--llama_max_request_attempts=5
--llama_port=15000
--llama_registration_timeout_secs=30
--llama_registration_wait_secs=3
--num_hdfs_worker_threads=16
--resource_broker_cnxn_attempts=1
--resource_broker_cnxn_retry_interval_ms=3000
--resource_broker_recv_timeout=0
--resource_broker_send_timeout=0
--staging_cgroup=impala_staging
--state_store_host=hd-memento-m1.casd.fr
--state_store_subscriber_port=23020
--use_statestore=true
--s3a_access_key_cmd=
--s3a_secret_key_cmd=
--local_library_dir=/tmp
--serialize_batch=false
--status_report_interval=5
--max_filter_error_rate=0.75
--num_threads_per_core=3
--use_local_tz_for_unix_timestamp_conversions=false
--scratch_dirs=/tmp
--queue_wait_timeout_ms=60000
--max_vcore_oversubscription_ratio=2.5
--rm_mem_expansion_timeout_ms=5000
--rm_always_use_defaults=false
--rm_default_cpu_vcores=2
--rm_default_memory=4G
--default_pool_max_queued=200
--default_pool_max_requests=-1
--default_pool_mem_limit=
--disable_pool_max_requests=false
--disable_pool_mem_limits=false
--fair_scheduler_allocation_path=
--llama_site_path=
--require_username=false
--disable_admission_control=false
--log_mem_usage_interval=0
--authorization_policy_file=
--authorization_policy_provider_class=org.apache.sentry.provider.common.HadoopGroupResourceAuthorizationProvider
--authorized_proxy_user_config=
--authorized_proxy_user_config_delimiter=,
--load_catalog_at_startup=false
--server_name=
--abort_on_failed_audit_event=true
--abort_on_failed_lineage_event=true
--audit_event_log_dir=
--be_service_threads=64
--beeswax_port=21000
--cancellation_thread_pool_size=5
--default_query_options=
--fe_service_threads=64
--hs2_port=21050
--idle_query_timeout=0
--idle_session_timeout=0
--lineage_event_log_dir=
--local_nodemanager_url=
--log_query_to_file=true
--max_audit_event_log_file_size=5000
--max_lineage_log_file_size=5000
--max_profile_log_file_size=5000
--max_profile_log_files=10
--max_result_cache_size=100000
--profile_log_dir=
--query_log_size=25
--ssl_client_ca_certificate=
--ssl_private_key=BUNDLE-REDACTED 15:47:44.213495 29911 init.cc:210] Cpu Info:
Model: Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz
Cores: 3
L1 Cache: 32.00 KB (Line: 64.00 B)
L2 Cache: 256.00 KB (Line: 64.00 B)
L3 Cache: 20.00 MB (Line: 64.00 B)
Hardware Supports:
ssse3
sse4_1
sse4_2
popcnt
avx2
I1228 15:47:44.213503 29911 init.cc:211] Disk Info:
Num disks 3:
sr (rotational=true)
sda (rotational=true)
dm- (rotational=true)

I1228 15:47:44.213510 29911 init.cc:212] Physical Memory: 23.54 GB
I1228 15:47:44.213517 29911 init.cc:213] OS version: Linux version 3.13.0-32-generic (buildd@kissel) (gcc version 4.8.2 (Ubuntu 4.8.2-19ubuntu1) ) #57-Ubuntu SMP Tue Jul 15 03:51:08 UTC 2014
Clock: clocksource: 'hyperv_clocksource', clockid_t: CLOCK_MONOTONIC_COARSE
I1228 15:47:44.213521 29911 init.cc:214] Process ID: 29911
I1228 15:47:45.261741 29911 webserver.cc:233] Starting webserver on 0.0.0.0:25020
I1228 15:47:45.261761 29911 webserver.cc:247] Document root: /opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/impala
I1228 15:47:45.261914 29911 webserver.cc:331] Webserver started
I1228 15:47:45.299940 29911 GlogAppender.java:126] Logging initialized. Impala: VLOG, All other: INFO
I1228 15:47:45.300709 29911 JniCatalog.java:98] Java Version Info: Java(TM) SE Runtime Environment (1.7.0_67-b01)
I1228 15:47:45.741204 29911 MetaStoreClientPool.java:77] Creating MetaStoreClient. Pool Size = 0
I1228 15:47:45.758981 29911 HiveMetaStoreClient.java:386] Trying to connect to metastore with URI thrift://hd-memento-m1.casd.fr:9083
I1228 15:47:45.780591 29911 HiveMetaStoreClient.java:431] Opened a connection to metastore, current connections: 1
I1228 15:47:45.800804 29911 HiveMetaStoreClient.java:483] Connected to metastore.
I1228 15:47:45.887679 29911 MetaStoreClientPool.java:77] Creating MetaStoreClient. Pool Size = 1
I1228 15:47:45.888039 29911 HiveMetaStoreClient.java:386] Trying to connect to metastore with URI thrift://hd-memento-m1.casd.fr:9083
I1228 15:47:45.888519 29911 HiveMetaStoreClient.java:431] Opened a connection to metastore, current connections: 2
I1228 15:47:45.890002 29911 HiveMetaStoreClient.java:483] Connected to metastore.
I1228 15:47:45.890100 29911 MetaStoreClientPool.java:77] Creating MetaStoreClient. Pool Size = 2
I1228 15:47:45.890396 29911 HiveMetaStoreClient.java:386] Trying to connect to metastore with URI thrift://hd-memento-m1.casd.fr:9083
I1228 15:47:45.890733 29911 HiveMetaStoreClient.java:431] Opened a connection to metastore, current connections: 3
I1228 15:47:45.891934 29911 HiveMetaStoreClient.java:483] Connected to metastore.
I1228 15:47:45.892016 29911 MetaStoreClientPool.java:77] Creating MetaStoreClient. Pool Size = 3
I1228 15:47:45.896288 29911 HiveMetaStoreClient.java:386] Trying to connect to metastore with URI thrift://hd-memento-m1.casd.fr:9083
I1228 15:47:45.896705 29911 HiveMetaStoreClient.java:431] Opened a connection to metastore, current connections: 4
I1228 15:47:45.898458 29911 HiveMetaStoreClient.java:483] Connected to metastore.
I1228 15:47:45.898550 29911 MetaStoreClientPool.java:77] Creating MetaStoreClient. Pool Size = 4
I1228 15:47:45.899025 29911 HiveMetaStoreClient.java:386] Trying to connect to metastore with URI thrift://hd-memento-m1.casd.fr:9083
I1228 15:47:45.899375 29911 HiveMetaStoreClient.java:431] Opened a connection to metastore, current connections: 5
I1228 15:47:45.900509 29911 HiveMetaStoreClient.java:483] Connected to metastore.
I1228 15:47:45.900593 29911 MetaStoreClientPool.java:77] Creating MetaStoreClient. Pool Size = 5
I1228 15:47:45.900882 29911 HiveMetaStoreClient.java:386] Trying to connect to metastore with URI thrift://hd-memento-m1.casd.fr:9083
I1228 15:47:45.901231 29911 HiveMetaStoreClient.java:431] Opened a connection to metastore, current connections: 6
I1228 15:47:45.902308 29911 HiveMetaStoreClient.java:483] Connected to metastore.
I1228 15:47:45.902398 29911 MetaStoreClientPool.java:77] Creating MetaStoreClient. Pool Size = 6
I1228 15:47:45.902690 29911 HiveMetaStoreClient.java:386] Trying to connect to metastore with URI thrift://hd-memento-m1.casd.fr:9083
I1228 15:47:45.903064 29911 HiveMetaStoreClient.java:431] Opened a connection to metastore, current connections: 7
I1228 15:47:45.905139 29911 HiveMetaStoreClient.java:483] Connected to metastore.
I1228 15:47:45.905225 29911 MetaStoreClientPool.java:77] Creating MetaStoreClient. Pool Size = 7
I1228 15:47:45.906241 29911 HiveMetaStoreClient.java:386] Trying to connect to metastore with URI thrift://hd-memento-m1.casd.fr:9083
I1228 15:47:45.906636 29911 HiveMetaStoreClient.java:431] Opened a connection to metastore, current connections: 8
I1228 15:47:45.907928 29911 HiveMetaStoreClient.java:483] Connected to metastore.
I1228 15:47:45.908023 29911 MetaStoreClientPool.java:77] Creating MetaStoreClient. Pool Size = 8
I1228 15:47:45.908314 29911 HiveMetaStoreClient.java:386] Trying to connect to metastore with URI thrift://hd-memento-m1.casd.fr:9083
I1228 15:47:45.908638 29911 HiveMetaStoreClient.java:431] Opened a connection to metastore, current connections: 9
I1228 15:47:45.909941 29911 HiveMetaStoreClient.java:483] Connected to metastore.
I1228 15:47:45.910076 29911 MetaStoreClientPool.java:77] Creating MetaStoreClient. Pool Size = 9
I1228 15:47:45.910385 29911 HiveMetaStoreClient.java:386] Trying to connect to metastore with URI thrift://hd-memento-m1.casd.fr:9083
I1228 15:47:45.913889 29911 HiveMetaStoreClient.java:431] Opened a connection to metastore, current connections: 10
I1228 15:47:45.915392 29911 HiveMetaStoreClient.java:483] Connected to metastore.
I1228 15:47:46.642184 29911 CatalogServiceCatalog.java:501] Loading native functions for database: default
I1228 15:47:46.642590 29911 CatalogServiceCatalog.java:527] Loading Java functions for database: default
I1228 15:47:46.657570 29911 statestore-subscriber.cc:181] Starting statestore subscriber
I1228 15:47:46.658641 29911 thrift-server.cc:434] ThriftServer 'StatestoreSubscriber' started on port: 23020
I1228 15:47:46.658649 29911 statestore-subscriber.cc:193] Registering with statestore
I1228 15:47:46.659277 29911 statestore-subscriber.cc:165] Subscriber registration ID: 3543b299cbbd3532:7313ff2af0caf9bb
I1228 15:47:46.659288 29911 statestore-subscriber.cc:197] statestore registration successful
I1228 15:47:46.686005 30015 catalog-server.cc:316] Publishing update: DATABASE:default@1
I1228 15:47:46.686033 30015 catalog-server.cc:316] Publishing update: CATALOG:8768fc952b414f8f:b7748ac681441736@1
I1228 15:47:46.687074 29911 thrift-server.cc:434] ThriftServer 'CatalogService' started on port: 26000
I1228 15:47:46.687083 29911 catalogd-main.cc:96] CatalogService started on port: 26000
I1228 15:47:48.668690 30019 catalog-server.cc:232] Catalog Version: 1 Last Catalog Version: 0
I1228 15:48:14.219319 29972 logging-support.cc:139] Old log file deleted during log rotation: /var/log/catalogd/catalogd.hd-memento-m2.impala.log.INFO.20171221-160549.2922
I1228 15:48:14.219488 29972 logging-support.cc:139] Old log file deleted during log rotation: /var/log/catalogd/catalogd.hd-memento-m2.impala.log.WARNING.20171221-160549.2922
I1228 15:48:14.219636 29972 logging-support.cc:139] Old log file deleted during log rotation: /var/log/catalogd/catalogd.hd-memento-m2.impala.log.ERROR.20171221-160549.2922
I1228 15:57:50.007673 30019 catalog-server.cc:232] Catalog Version: 1 Last Catalog Version: 1
I1228 16:07:51.331156 30019 catalog-server.cc:232] Catalog Version: 1 Last Catalog Version: 1
I1228 16:17:52.666932 30019 catalog-server.cc:232] Catalog Version: 1 Last Catalog Version: 1
I1228 16:27:54.070752 30019 catalog-server.cc:232] Catalog Version: 1 Last Catalog Version: 1
I1228 16:37:55.393751 30019 catalog-server.cc:232] Catalog Version: 1 Last Catalog Version: 1
I1228 16:47:56.709848 30019 catalog-server.cc:232] Catalog Version: 1 Last Catalog Version: 1
I1228 16:57:58.042268 30019 catalog-server.cc:232] Catalog Version: 1 Last Catalog Version: 1
I1228 17:07:59.351016 30019 catalog-server.cc:232] Catalog Version: 1 Last Catalog Version: 1
I1228 17:18:00.679327 30019 catalog-server.cc:232] Catalog Version: 1 Last Catalog Version: 1
I1228 17:28:02.010982 30019 catalog-server.cc:232] Catalog Version: 1 Last Catalog Version: 1
I1228 17:38:03.327862 30019 catalog-server.cc:232] Catalog Version: 1 Last Catalog Version: 1
I1228 17:48:04.765046 30019 catalog-server.cc:232] Catalog Version: 1 Last Catalog Version: 1
I1228 17:58:06.073541 30019 catalog-server.cc:232] Catalog Version: 1 Last Catalog Version: 1
I1228 18:08:07.398394 30019 catalog-server.cc:232] Catalog Version: 1 Last Catalog Version: 1
I1228 18:18:08.735150 30019 catalog-server.cc:232] Catalog Version: 1 Last Catalog Version: 1
I1228 18:28:10.045891 30019 catalog-server.cc:232] Catalog Version: 1 Last Catalog Version: 1
I1228 18:38:11.356907 30019 catalog-server.cc:232] Catalog Version: 1 Last Catalog Version: 1

Re: impala catalog server & impala deamons web servers issues

Champion

@skhadrao could you click on the bad health status of those services  and share it as a screen shot in this thread ? 

Re: impala catalog server & impala deamons web servers issues

Explorer

screen1.pngscrenn2.png