Created on 08-13-2014 04:09 AM - edited 09-16-2022 02:04 AM
I encounter such error when trying to use Impala:
ERROR: AnalysisException: This Impala daemon is not ready to accept user requests. Status: Waiting for catalog update from the StateStore.
However, Cloudera Manager shows that all Impala Daemons and Impala StateStore Daemon are healthy.
How should I configure Impala to fix this error.
My CM version is 4.5.2 and Impala version is impalad version 1.3.1-cdh4
Created 08-13-2014 09:46 PM
Created 08-13-2014 12:07 PM
Created 08-13-2014 08:25 PM
My CDH version is 4.5. The hive is healthy.
Is it the /var/log/statestore/statestored.INFO the StateStore log?
Here is the content of this file:
Log file created at: 2014/08/13 10:10:50
Running on machine: bi-hadoop-m
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
I0813 10:10:50.259363 3245 init.cc:131] statestored version 1.3.1-cdh4 RELEASE (build 907481bf45b248a7bb3bb077d54831a71f484e5f)
Built on Wed, 30 Apr 2014 14:38:46 PST
I0813 10:10:50.273645 3245 init.cc:132] Using hostname: bi-hadoop-m
I0813 10:10:50.274845 3245 logging.cc:100] Flags (see also /varz are on debug webserver):
--catalog_service_port=26000
--load_catalog_in_background=true
--num_metadata_loading_threads=16
--dump_ir=false
--opt_module_output=
--unopt_module_output=
--abort_on_config_error=true
--be_port=22000
--be_principal=
--enable_process_lifetime_heap_profiling=false
--heap_profile_dir=
--hostname=bi-hadoop-m
--keytab_file=
--mem_limit=80%
--principal=
--log_filename=statestored
--exchg_node_buffer_size_bytes=10485760
--max_row_batches=0
--enable_ldap_auth=false
--kerberos_reinit_interval=60
--ldap_manual_config=false
--ldap_tls=false
--ldap_uri=
--sasl_path=/usr/lib/sasl2:/usr/lib64/sasl2:/usr/local/lib/sasl2:/usr/lib/x86_64-linux-gnu/sasl2
--rpc_cnxn_attempts=10
--rpc_cnxn_retry_interval_ms=2000
--insert_inherit_permissions=false
--min_buffer_size=1024
--num_disks=0
--num_threads_per_disk=0
--read_size=8388608
--reuse_io_buffers=true
--catalog_service_host=localhost
--cgroup_hierarchy_path=
--enable_rm=false
--enable_webserver=true
--llama_callback_port=28000
--llama_host=127.0.0.1
--llama_port=15000
--num_hdfs_worker_threads=16
--resource_broker_cnxn_attempts=10
--resource_broker_cnxn_retry_interval_ms=3000
--resource_broker_recv_timeout=0
--resource_broker_send_timeout=0
--staging_cgroup=impala_staging
--state_store_host=localhost
--state_store_subscriber_port=23000
--use_statestore=true
--local_library_dir=/tmp
--serialize_batch=false
--status_report_interval=5
--num_threads_per_core=3
--queue_wait_timeout_ms=60000
--default_pool_max_queued=50
--default_pool_max_requests=20
--default_pool_mem_limit=
--disable_pool_max_requests=false
--disable_pool_mem_limits=false
--fair_scheduler_allocation_path=
--llama_site_path=
--authorization_policy_file=
--authorization_policy_provider_class=org.apache.sentry.provider.file.HadoopGroupResourceAuthorizationProvider
--authorized_proxy_user_config=
--load_catalog_at_startup=false
--server_name=
--abort_on_failed_audit_event=true
--audit_event_log_dir=
--be_service_threads=64
--beeswax_port=21000
--cancellation_thread_pool_size=5
--default_query_options=
--fe_service_threads=64
--hs2_port=21050
--idle_query_timeout=0
--idle_session_timeout=0
--local_nodemanager_url=
--log_mem_usage_interval=0
--log_query_to_file=true
--max_audit_event_log_file_size=5000
--max_profile_log_file_size=5000
--max_result_cache_size=100000
--profile_log_dir=
--query_log_size=25
--ssl_client_ca_certificate=
--ssl_private_key=
--ssl_server_certificate=
--max_vcore_oversubscription_ratio=2.5
--rm_always_use_defaults=false
--rm_default_cpu_vcores=2
--rm_default_memory=4G
--disable_admission_control=true
--require_username=false
--statestore_subscriber_cnxn_attempts=10
--statestore_subscriber_cnxn_retry_interval_ms=3000
--statestore_subscriber_timeout_seconds=30
--state_store_port=24000
--statestore_heartbeat_frequency_ms=500
--statestore_max_missed_heartbeats=10
--statestore_num_heartbeat_threads=10
--statestore_suspect_heartbeats=5
--num_cores=0
--web_log_bytes=1048576
--non_impala_java_vlog=0
--periodic_counter_update_period_ms=500
--enable_webserver_doc_root=true
--webserver_authentication_domain=
--webserver_certificate_file=
--webserver_doc_root=/opt/cloudera/parcels/IMPALA-1.3.1-1.impala1.3.1.p0.1172/lib/impala
--webserver_interface=
--webserver_password_file=
--webserver_port=25010
--flagfile=/var/run/cloudera-scm-agent/process/1886-impala-STATESTORE/impala-conf/state_store_flags
--fromenv=
--tryfromenv=
--undefok=
--tab_completion_columns=80
--tab_completion_word=
--help=false
--helpfull=false
--helpmatch=
--helpon=
--helppackage=false
--helpshort=false
--helpxml=false
--version=false
--alsologtoemail=
--alsologtostderr=false
--drop_log_memory=true
--log_backtrace_at=
--log_dir=/var/log/statestore
--log_link=
--log_prefix=true
--logbuflevel=-1
--logbufsecs=30
--logbufvlevel=1
--logemaillevel=999
--logmailer=/bin/mail
--logtostderr=false
--max_log_size=200
--minloglevel=0
--stderrthreshold=2
--stop_logging_if_full_disk=false
--symbolize_stacktrace=true
--v=1
--vmodule=
I0813 10:10:50.275037 3245 init.cc:137] Cpu Info:
Model: Intel(R) Xeon(R) CPU E5-2407 0 @ 2.20GHz
Cores: 8
L1 Cache: 32.00 KB
L2 Cache: 256.00 KB
L3 Cache: 10.00 MB
Hardware Supports:
ssse3
sse4_1
sse4_2
popcnt
I0813 10:10:50.275087 3245 init.cc:138] Disk Info:
Num disks 2:
sda (rotational=true)
dm- (rotational=true)
I0813 10:10:50.275172 3245 init.cc:139] Physical Memory: 31.39 GB
I0813 10:10:50.275213 3245 init.cc:140] OS version: Linux version 2.6.18-348.el5 (mockbuild@builder10.centos.org) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-54)) #1 SMP Tue Jan 8 17:53:53 EST 2013
I0813 10:10:50.275233 3245 init.cc:141] Process ID: 3245
I0813 10:10:50.275317 3245 webserver.cc:155] Starting webserver on 0.0.0.0:25010
I0813 10:10:50.275346 3245 webserver.cc:169] Document root: /opt/cloudera/parcels/IMPALA-1.3.1-1.impala1.3.1.p0.1172/lib/impala
I0813 10:10:50.275583 3245 webserver.cc:235] Webserver started
I0813 10:10:50.281631 3245 thrift-server.cc:387] ThriftServer 'StatestoreService' started on port: 24000
I0813 10:10:52.497861 3382 statestore.cc:293] Creating new topic: ''catalog-update' on behalf of subscriber: 'bi-hadoop-sm:22000
I0813 10:10:52.497964 3382 statestore.cc:293] Creating new topic: ''impala-membership' on behalf of subscriber: 'bi-hadoop-sm:22000
I0813 10:10:52.497993 3382 statestore.cc:300] Registering: bi-hadoop-sm:22000
I0813 10:10:52.498136 3325 client-cache.cc:94] CreateClient(): creating new client for bi-hadoop-sm:23000
I0813 10:10:52.498955 3382 statestore.cc:331] Subscriber 'bi-hadoop-sm:22000' registered (registration id: d44b566c7d4f17e5:12d261558a25a1bf)
I0813 10:10:53.000346 3325 statestore.cc:458] Preparing initial impala-membership topic update for bi-hadoop-sm:22000. Size = 106.00 B
I0813 10:10:55.190691 3432 statestore.cc:300] Registering: bi-hadoop-s38:22000
I0813 10:10:55.190860 3432 statestore.cc:331] Subscriber 'bi-hadoop-s38:22000' registered (registration id: 404e299449525642:3919423961822d8f)
I0813 10:10:55.191412 3332 statestore.cc:458] Preparing initial impala-membership topic update for bi-hadoop-s38:22000. Size = 106.00 B
I0813 10:10:55.191493 3332 client-cache.cc:94] CreateClient(): creating new client for bi-hadoop-s38:23000
I0813 10:10:55.669672 3444 statestore.cc:300] Registering: bi-hadoop-s39:22000
I0813 10:10:55.669821 3444 statestore.cc:331] Subscriber 'bi-hadoop-s39:22000' registered (registration id: d049c9d286b0c0a5:90098d1a4e53bc81)
I0813 10:10:55.691418 3326 statestore.cc:458] Preparing initial impala-membership topic update for bi-hadoop-s39:22000. Size = 214.00 B
I0813 10:10:55.691493 3326 client-cache.cc:94] CreateClient(): creating new client for bi-hadoop-s39:23000
I0813 10:10:56.498275 3467 statestore.cc:300] Registering: bi-hadoop-m:22000
I0813 10:10:56.498389 3467 statestore.cc:331] Subscriber 'bi-hadoop-m:22000' registered (registration id: 2c47ab5e0b47ac69:bb1e5229cc792da1)
I0813 10:10:56.498438 3333 statestore.cc:458] Preparing initial impala-membership topic update for bi-hadoop-m:22000. Size = 322.00 B
I0813 10:10:56.498477 3333 client-cache.cc:94] CreateClient(): creating new client for bi-hadoop-m:23000
I0813 10:10:56.810374 3619 statestore.cc:300] Registering: bi-hadoop-s35:22000
I0813 10:10:56.810462 3619 statestore.cc:331] Subscriber 'bi-hadoop-s35:22000' registered (registration id: 3b4548042d00bf08:40a6116030a766a3)
I0813 10:10:56.810511 3330 statestore.cc:458] Preparing initial impala-membership topic update for bi-hadoop-s35:22000. Size = 426.00 B
I0813 10:10:56.810546 3330 client-cache.cc:94] CreateClient(): creating new client for bi-hadoop-s35:23000
I0813 10:10:57.740465 3683 statestore.cc:300] Registering: bi-hadoop-s33:22000
I0813 10:10:57.740603 3683 statestore.cc:331] Subscriber 'bi-hadoop-s33:22000' registered (registration id: 414c2f5002aaed82:8cab6c8740212088)
I0813 10:10:57.740674 3331 statestore.cc:458] Preparing initial impala-membership topic update for bi-hadoop-s33:22000. Size = 534.00 B
I0813 10:10:57.740735 3331 client-cache.cc:94] CreateClient(): creating new client for bi-hadoop-s33:23000
I0813 10:10:57.959748 3699 statestore.cc:300] Registering: bi-hadoop-s36:22000
I0813 10:10:57.959874 3699 statestore.cc:331] Subscriber 'bi-hadoop-s36:22000' registered (registration id: 4e49d7914f3c0ca9:eaeb5fccce2f6ebc)
I0813 10:10:57.959945 3328 statestore.cc:458] Preparing initial impala-membership topic update for bi-hadoop-s36:22000. Size = 642.00 B
I0813 10:10:57.960011 3328 client-cache.cc:94] CreateClient(): creating new client for bi-hadoop-s36:23000
I0813 10:10:58.872663 3707 statestore.cc:300] Registering: bi-hadoop-s34:22000
I0813 10:10:58.872807 3707 statestore.cc:331] Subscriber 'bi-hadoop-s34:22000' registered (registration id: 73486c443003bff4:c92cc94de366c85)
I0813 10:10:58.872871 3327 statestore.cc:458] Preparing initial impala-membership topic update for bi-hadoop-s34:22000. Size = 750.00 B
I0813 10:10:58.872941 3327 client-cache.cc:94] CreateClient(): creating new client for bi-hadoop-s34:23000
I0813 10:11:00.013226 3710 statestore.cc:300] Registering: bi-hadoop-s40:22000
I0813 10:11:00.013355 3710 statestore.cc:331] Subscriber 'bi-hadoop-s40:22000' registered (registration id: 84487823e24a4c01:e11f38a681d38a2)
I0813 10:11:00.013402 3334 statestore.cc:458] Preparing initial impala-membership topic update for bi-hadoop-s40:22000. Size = 858.00 B
I0813 10:11:00.013456 3334 client-cache.cc:94] CreateClient(): creating new client for bi-hadoop-s40:23000
Created 08-13-2014 09:46 PM
Created 08-13-2014 10:02 PM
Created 08-14-2014 11:11 AM
Created 08-17-2014 11:59 PM