Reply
Explorer
Posts: 6
Registered: ‎12-28-2017

impala catalog server & impala deamons web servers issues

Hello !

 

I'm installing CDH5.9.0 on 6 machines (2 masters and 4 workers).

 

I have the Impala service going on red error; impala catalog server and impala deamons are in red status, the impala statestore is in a healthy state (green).

 

I have no data on the cluster, this is just happening after the installation.

 

The other services are all in a good state, and all services databases are well configured.

 

 

Any helps well be very welcome !

 

 

Thank you.

Champion
Posts: 731
Registered: ‎05-16-2016

Re: impala catalog server & impala deamons web servers issues

[ Edited ]

it should state you the reason as to why its health are bad in the cloudera manager health test 

 

did you try restarting the impala catalog and impala daemon ? 

were you able to fire impala-shell and excute some simple quries ?

also to troubleshoot check the impala catalog and impala daemon logs. 

Explorer
Posts: 6
Registered: ‎12-28-2017

Re: impala catalog server & impala deamons web servers issues

Thank you csguna,

 

I have tried to restart without but didn't solve the issue.

 

Now I'm running Impala queries from hue or impala-shell without any problem.

 

But the health status of impala deamons and catalog still can not turn to good (green color on CM) ??!!

 

 

I don't grasp excatly what is the problem.

This is the log file of the catalog service

 

Log file created at: 2017/12/28 15:47:44
Running on machine: hd-memento-m2
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
I1228 15:47:44.203192 29911 logging.cc:120] stdout will be logged to this file.
E1228 15:47:44.203328 29911 logging.cc:121] stderr will be logged to this file.
I1228 15:47:44.203564 29911 minidump.cc:222] Setting minidump size limit to 20971520.
I1228 15:47:44.203606 29911 atomicops-internals-x86.cc:93] vendor GenuineIntel family 6 model 15 sse2 1 cmpxchg16b 1
I1228 15:47:44.212666 29911 authentication.cc:1020] Internal communication is not authenticated
I1228 15:47:44.212677 29911 authentication.cc:1041] External communication is not authenticated
I1228 15:47:44.212826 29911 init.cc:204] catalogd version 2.7.0-cdh5.9.0 RELEASE (build 4b4cf1936bd6cdf34fda5e2f32827e7d60c07a9c)
Built on Fri Oct 21 01:07:22 PDT 2016
I1228 15:47:44.212831 29911 init.cc:205] Using hostname: hd-memento-m2
I1228 15:47:44.213428 29911 logging.cc:156] Flags (see also /varz are on debug webserver):
--catalog_service_port=26000
--load_catalog_in_background=true
--num_metadata_loading_threads=16
--sentry_config=
--asm_module_dir=
--disable_optimization_passes=false
--dump_ir=false
--opt_module_dir=
--perf_map=false
--print_llvm_ir_instruction_count=false
--unopt_module_dir=
--abort_on_config_error=true
--be_port=22000
--be_principal=
--compact_catalog_topic=false
--disable_kudu=false
--disable_mem_pools=false
--enable_process_lifetime_heap_profiling=false
--heap_profile_dir=
--hostname=hd-memento-m2
--keytab_file=
--krb5_conf=
--krb5_debug_file=
--load_auth_to_local_rules=false
--max_minidumps=9
--mem_limit=80%
--minidump_path=/var/log/impala-minidumps/catalogd
--minidump_size_limit_hint_kb=20480
--principal=
--redaction_rules_file=
--max_log_files=10
--pause_monitor_sleep_time_ms=500
--pause_monitor_warn_threshold_ms=10000
--log_filename=catalogd
--redirect_stdout_stderr=true
--data_source_batch_size=1024
--exchg_node_buffer_size_bytes=10485760
--enable_partitioned_aggregation=true
--enable_partitioned_hash_join=true
--enable_probe_side_filtering=true
--enable_quadratic_probing=true
--skip_lzo_version_check=false
--parquet_min_filter_reject_ratio=0.10000000000000001
--max_row_batches=0
--runtime_filter_wait_time_ms=1000
--suppress_unknown_disk_id_warnings=false
--kudu_max_row_batches=0
--kudu_scanner_keep_alive_period_us=15000000
--kudu_scanner_keep_alive_period_sec=15
--kudu_scanner_timeout_sec=60
--pick_only_leaders_for_tests=false
--kudu_session_timeout_seconds=60
--convert_legacy_hive_parquet_utc_timestamps=false
--max_page_header_size=8388608
--enable_phj_probe_side_filtering=true
--enable_ldap_auth=false
--internal_principals_whitelist=hdfs
--kerberos_reinit_interval=60
--ldap_allow_anonymous_binds=false
--ldap_baseDN=
--ldap_bind_pattern=
--ldap_ca_certificate=
--ldap_domain=
--ldap_manual_config=false
--ldap_passwords_in_clear_ok=false
--ldap_tls=false
--ldap_uri=
--sasl_path=
--rpc_cnxn_attempts=10
--rpc_cnxn_retry_interval_ms=2000
--disk_spill_encryption=false
--insert_inherit_permissions=false
--datastream_sender_timeout_ms=120000
--max_cached_file_handles=0
--max_free_io_buffers=128
--min_buffer_size=1024
--num_disks=0
--num_remote_hdfs_io_threads=8
--num_s3_io_threads=16
--num_threads_per_disk=0
--read_size=8388608
--backend_client_connection_num_retries=3
--backend_client_rpc_timeout_ms=300000
--catalog_client_connection_num_retries=3
--catalog_client_rpc_timeout_ms=0
--catalog_service_host=localhost
--cgroup_hierarchy_path=
--coordinator_rpc_threads=12
--enable_rm=false
--enable_webserver=true
--llama_addresses=
--llama_callback_port=28000
--llama_host=
--llama_max_request_attempts=5
--llama_port=15000
--llama_registration_timeout_secs=30
--llama_registration_wait_secs=3
--num_hdfs_worker_threads=16
--resource_broker_cnxn_attempts=1
--resource_broker_cnxn_retry_interval_ms=3000
--resource_broker_recv_timeout=0
--resource_broker_send_timeout=0
--staging_cgroup=impala_staging
--state_store_host=hd-memento-m1.casd.fr
--state_store_subscriber_port=23020
--use_statestore=true
--s3a_access_key_cmd=
--s3a_secret_key_cmd=
--local_library_dir=/tmp
--serialize_batch=false
--status_report_interval=5
--max_filter_error_rate=0.75
--num_threads_per_core=3
--use_local_tz_for_unix_timestamp_conversions=false
--scratch_dirs=/tmp
--queue_wait_timeout_ms=60000
--max_vcore_oversubscription_ratio=2.5
--rm_mem_expansion_timeout_ms=5000
--rm_always_use_defaults=false
--rm_default_cpu_vcores=2
--rm_default_memory=4G
--default_pool_max_queued=200
--default_pool_max_requests=-1
--default_pool_mem_limit=
--disable_pool_max_requests=false
--disable_pool_mem_limits=false
--fair_scheduler_allocation_path=
--llama_site_path=
--require_username=false
--disable_admission_control=false
--log_mem_usage_interval=0
--authorization_policy_file=
--authorization_policy_provider_class=org.apache.sentry.provider.common.HadoopGroupResourceAuthorizationProvider
--authorized_proxy_user_config=
--authorized_proxy_user_config_delimiter=,
--load_catalog_at_startup=false
--server_name=
--abort_on_failed_audit_event=true
--abort_on_failed_lineage_event=true
--audit_event_log_dir=
--be_service_threads=64
--beeswax_port=21000
--cancellation_thread_pool_size=5
--default_query_options=
--fe_service_threads=64
--hs2_port=21050
--idle_query_timeout=0
--idle_session_timeout=0
--lineage_event_log_dir=
--local_nodemanager_url=
--log_query_to_file=true
--max_audit_event_log_file_size=5000
--max_lineage_log_file_size=5000
--max_profile_log_file_size=5000
--max_profile_log_files=10
--max_result_cache_size=100000
--profile_log_dir=
--query_log_size=25
--ssl_client_ca_certificate=
--ssl_private_key=BUNDLE-REDACTED 15:47:44.213495 29911 init.cc:210] Cpu Info:
Model: Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz
Cores: 3
L1 Cache: 32.00 KB (Line: 64.00 B)
L2 Cache: 256.00 KB (Line: 64.00 B)
L3 Cache: 20.00 MB (Line: 64.00 B)
Hardware Supports:
ssse3
sse4_1
sse4_2
popcnt
avx2
I1228 15:47:44.213503 29911 init.cc:211] Disk Info:
Num disks 3:
sr (rotational=true)
sda (rotational=true)
dm- (rotational=true)

I1228 15:47:44.213510 29911 init.cc:212] Physical Memory: 23.54 GB
I1228 15:47:44.213517 29911 init.cc:213] OS version: Linux version 3.13.0-32-generic (buildd@kissel) (gcc version 4.8.2 (Ubuntu 4.8.2-19ubuntu1) ) #57-Ubuntu SMP Tue Jul 15 03:51:08 UTC 2014
Clock: clocksource: 'hyperv_clocksource', clockid_t: CLOCK_MONOTONIC_COARSE
I1228 15:47:44.213521 29911 init.cc:214] Process ID: 29911
I1228 15:47:45.261741 29911 webserver.cc:233] Starting webserver on 0.0.0.0:25020
I1228 15:47:45.261761 29911 webserver.cc:247] Document root: /opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/impala
I1228 15:47:45.261914 29911 webserver.cc:331] Webserver started
I1228 15:47:45.299940 29911 GlogAppender.java:126] Logging initialized. Impala: VLOG, All other: INFO
I1228 15:47:45.300709 29911 JniCatalog.java:98] Java Version Info: Java(TM) SE Runtime Environment (1.7.0_67-b01)
I1228 15:47:45.741204 29911 MetaStoreClientPool.java:77] Creating MetaStoreClient. Pool Size = 0
I1228 15:47:45.758981 29911 HiveMetaStoreClient.java:386] Trying to connect to metastore with URI thrift://hd-memento-m1.casd.fr:9083
I1228 15:47:45.780591 29911 HiveMetaStoreClient.java:431] Opened a connection to metastore, current connections: 1
I1228 15:47:45.800804 29911 HiveMetaStoreClient.java:483] Connected to metastore.
I1228 15:47:45.887679 29911 MetaStoreClientPool.java:77] Creating MetaStoreClient. Pool Size = 1
I1228 15:47:45.888039 29911 HiveMetaStoreClient.java:386] Trying to connect to metastore with URI thrift://hd-memento-m1.casd.fr:9083
I1228 15:47:45.888519 29911 HiveMetaStoreClient.java:431] Opened a connection to metastore, current connections: 2
I1228 15:47:45.890002 29911 HiveMetaStoreClient.java:483] Connected to metastore.
I1228 15:47:45.890100 29911 MetaStoreClientPool.java:77] Creating MetaStoreClient. Pool Size = 2
I1228 15:47:45.890396 29911 HiveMetaStoreClient.java:386] Trying to connect to metastore with URI thrift://hd-memento-m1.casd.fr:9083
I1228 15:47:45.890733 29911 HiveMetaStoreClient.java:431] Opened a connection to metastore, current connections: 3
I1228 15:47:45.891934 29911 HiveMetaStoreClient.java:483] Connected to metastore.
I1228 15:47:45.892016 29911 MetaStoreClientPool.java:77] Creating MetaStoreClient. Pool Size = 3
I1228 15:47:45.896288 29911 HiveMetaStoreClient.java:386] Trying to connect to metastore with URI thrift://hd-memento-m1.casd.fr:9083
I1228 15:47:45.896705 29911 HiveMetaStoreClient.java:431] Opened a connection to metastore, current connections: 4
I1228 15:47:45.898458 29911 HiveMetaStoreClient.java:483] Connected to metastore.
I1228 15:47:45.898550 29911 MetaStoreClientPool.java:77] Creating MetaStoreClient. Pool Size = 4
I1228 15:47:45.899025 29911 HiveMetaStoreClient.java:386] Trying to connect to metastore with URI thrift://hd-memento-m1.casd.fr:9083
I1228 15:47:45.899375 29911 HiveMetaStoreClient.java:431] Opened a connection to metastore, current connections: 5
I1228 15:47:45.900509 29911 HiveMetaStoreClient.java:483] Connected to metastore.
I1228 15:47:45.900593 29911 MetaStoreClientPool.java:77] Creating MetaStoreClient. Pool Size = 5
I1228 15:47:45.900882 29911 HiveMetaStoreClient.java:386] Trying to connect to metastore with URI thrift://hd-memento-m1.casd.fr:9083
I1228 15:47:45.901231 29911 HiveMetaStoreClient.java:431] Opened a connection to metastore, current connections: 6
I1228 15:47:45.902308 29911 HiveMetaStoreClient.java:483] Connected to metastore.
I1228 15:47:45.902398 29911 MetaStoreClientPool.java:77] Creating MetaStoreClient. Pool Size = 6
I1228 15:47:45.902690 29911 HiveMetaStoreClient.java:386] Trying to connect to metastore with URI thrift://hd-memento-m1.casd.fr:9083
I1228 15:47:45.903064 29911 HiveMetaStoreClient.java:431] Opened a connection to metastore, current connections: 7
I1228 15:47:45.905139 29911 HiveMetaStoreClient.java:483] Connected to metastore.
I1228 15:47:45.905225 29911 MetaStoreClientPool.java:77] Creating MetaStoreClient. Pool Size = 7
I1228 15:47:45.906241 29911 HiveMetaStoreClient.java:386] Trying to connect to metastore with URI thrift://hd-memento-m1.casd.fr:9083
I1228 15:47:45.906636 29911 HiveMetaStoreClient.java:431] Opened a connection to metastore, current connections: 8
I1228 15:47:45.907928 29911 HiveMetaStoreClient.java:483] Connected to metastore.
I1228 15:47:45.908023 29911 MetaStoreClientPool.java:77] Creating MetaStoreClient. Pool Size = 8
I1228 15:47:45.908314 29911 HiveMetaStoreClient.java:386] Trying to connect to metastore with URI thrift://hd-memento-m1.casd.fr:9083
I1228 15:47:45.908638 29911 HiveMetaStoreClient.java:431] Opened a connection to metastore, current connections: 9
I1228 15:47:45.909941 29911 HiveMetaStoreClient.java:483] Connected to metastore.
I1228 15:47:45.910076 29911 MetaStoreClientPool.java:77] Creating MetaStoreClient. Pool Size = 9
I1228 15:47:45.910385 29911 HiveMetaStoreClient.java:386] Trying to connect to metastore with URI thrift://hd-memento-m1.casd.fr:9083
I1228 15:47:45.913889 29911 HiveMetaStoreClient.java:431] Opened a connection to metastore, current connections: 10
I1228 15:47:45.915392 29911 HiveMetaStoreClient.java:483] Connected to metastore.
I1228 15:47:46.642184 29911 CatalogServiceCatalog.java:501] Loading native functions for database: default
I1228 15:47:46.642590 29911 CatalogServiceCatalog.java:527] Loading Java functions for database: default
I1228 15:47:46.657570 29911 statestore-subscriber.cc:181] Starting statestore subscriber
I1228 15:47:46.658641 29911 thrift-server.cc:434] ThriftServer 'StatestoreSubscriber' started on port: 23020
I1228 15:47:46.658649 29911 statestore-subscriber.cc:193] Registering with statestore
I1228 15:47:46.659277 29911 statestore-subscriber.cc:165] Subscriber registration ID: 3543b299cbbd3532:7313ff2af0caf9bb
I1228 15:47:46.659288 29911 statestore-subscriber.cc:197] statestore registration successful
I1228 15:47:46.686005 30015 catalog-server.cc:316] Publishing update: DATABASE:default@1
I1228 15:47:46.686033 30015 catalog-server.cc:316] Publishing update: CATALOG:8768fc952b414f8f:b7748ac681441736@1
I1228 15:47:46.687074 29911 thrift-server.cc:434] ThriftServer 'CatalogService' started on port: 26000
I1228 15:47:46.687083 29911 catalogd-main.cc:96] CatalogService started on port: 26000
I1228 15:47:48.668690 30019 catalog-server.cc:232] Catalog Version: 1 Last Catalog Version: 0
I1228 15:48:14.219319 29972 logging-support.cc:139] Old log file deleted during log rotation: /var/log/catalogd/catalogd.hd-memento-m2.impala.log.INFO.20171221-160549.2922
I1228 15:48:14.219488 29972 logging-support.cc:139] Old log file deleted during log rotation: /var/log/catalogd/catalogd.hd-memento-m2.impala.log.WARNING.20171221-160549.2922
I1228 15:48:14.219636 29972 logging-support.cc:139] Old log file deleted during log rotation: /var/log/catalogd/catalogd.hd-memento-m2.impala.log.ERROR.20171221-160549.2922
I1228 15:57:50.007673 30019 catalog-server.cc:232] Catalog Version: 1 Last Catalog Version: 1
I1228 16:07:51.331156 30019 catalog-server.cc:232] Catalog Version: 1 Last Catalog Version: 1
I1228 16:17:52.666932 30019 catalog-server.cc:232] Catalog Version: 1 Last Catalog Version: 1
I1228 16:27:54.070752 30019 catalog-server.cc:232] Catalog Version: 1 Last Catalog Version: 1
I1228 16:37:55.393751 30019 catalog-server.cc:232] Catalog Version: 1 Last Catalog Version: 1
I1228 16:47:56.709848 30019 catalog-server.cc:232] Catalog Version: 1 Last Catalog Version: 1
I1228 16:57:58.042268 30019 catalog-server.cc:232] Catalog Version: 1 Last Catalog Version: 1
I1228 17:07:59.351016 30019 catalog-server.cc:232] Catalog Version: 1 Last Catalog Version: 1
I1228 17:18:00.679327 30019 catalog-server.cc:232] Catalog Version: 1 Last Catalog Version: 1
I1228 17:28:02.010982 30019 catalog-server.cc:232] Catalog Version: 1 Last Catalog Version: 1
I1228 17:38:03.327862 30019 catalog-server.cc:232] Catalog Version: 1 Last Catalog Version: 1
I1228 17:48:04.765046 30019 catalog-server.cc:232] Catalog Version: 1 Last Catalog Version: 1
I1228 17:58:06.073541 30019 catalog-server.cc:232] Catalog Version: 1 Last Catalog Version: 1
I1228 18:08:07.398394 30019 catalog-server.cc:232] Catalog Version: 1 Last Catalog Version: 1
I1228 18:18:08.735150 30019 catalog-server.cc:232] Catalog Version: 1 Last Catalog Version: 1
I1228 18:28:10.045891 30019 catalog-server.cc:232] Catalog Version: 1 Last Catalog Version: 1
I1228 18:38:11.356907 30019 catalog-server.cc:232] Catalog Version: 1 Last Catalog Version: 1

Champion
Posts: 731
Registered: ‎05-16-2016

Re: impala catalog server & impala deamons web servers issues

@skhadrao could you click on the bad health status of those services  and share it as a screen shot in this thread ? 

Highlighted
Explorer
Posts: 6
Registered: ‎12-28-2017

Re: impala catalog server & impala deamons web servers issues

[ Edited ]

screen1.pngscrenn2.png

Announcements