Error in using Impala

[#1]

I encounter such error when trying to use Impala:

ERROR: AnalysisException: This Impala daemon is not ready to accept user requests. Status: Waiting for catalog update from the StateStore.

However, Cloudera Manager shows that all Impala Daemons and Impala StateStore Daemon are healthy.

How should I configure Impala to fix this error.

My CM version is 4.5.2 and Impala version is impalad version 1.3.1-cdh4

[#2]

What is your CDH version?

What do the StateStore logs say? Check both the role log and the stderr.

Make sure that Hive is up and healthy as well.

[#3]

My CDH version is 4.5. The hive is healthy.

Is it the /var/log/statestore/statestored.INFO the StateStore log?

Here is the content of this file:

Log file created at: 2014/08/13 10:10:50
Running on machine: bi-hadoop-m
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
I0813 10:10:50.259363  3245 init.cc:131] statestored version 1.3.1-cdh4 RELEASE (build 907481bf45b248a7bb3bb077d54831a71f484e5f)
Built on Wed, 30 Apr 2014 14:38:46 PST
I0813 10:10:50.273645  3245 init.cc:132] Using hostname: bi-hadoop-m
I0813 10:10:50.274845  3245 logging.cc:100] Flags (see also /varz are on debug webserver):
--catalog_service_port=26000
--load_catalog_in_background=true
--num_metadata_loading_threads=16
--dump_ir=false
--opt_module_output=
--unopt_module_output=
--abort_on_config_error=true
--be_port=22000
--be_principal=
--enable_process_lifetime_heap_profiling=false
--heap_profile_dir=
--hostname=bi-hadoop-m
--keytab_file=
--mem_limit=80%
--principal=
--log_filename=statestored
--exchg_node_buffer_size_bytes=10485760
--max_row_batches=0
--enable_ldap_auth=false
--kerberos_reinit_interval=60
--ldap_manual_config=false
--ldap_tls=false
--ldap_uri=
--sasl_path=/usr/lib/sasl2:/usr/lib64/sasl2:/usr/local/lib/sasl2:/usr/lib/x86_64-linux-gnu/sasl2
--rpc_cnxn_attempts=10
--rpc_cnxn_retry_interval_ms=2000
--insert_inherit_permissions=false
--min_buffer_size=1024
--num_disks=0
--num_threads_per_disk=0
--read_size=8388608
--reuse_io_buffers=true
--catalog_service_host=localhost
--cgroup_hierarchy_path=
--enable_rm=false
--enable_webserver=true
--llama_callback_port=28000
--llama_host=127.0.0.1
--llama_port=15000
--num_hdfs_worker_threads=16
--resource_broker_cnxn_attempts=10
--resource_broker_cnxn_retry_interval_ms=3000
--resource_broker_recv_timeout=0
--resource_broker_send_timeout=0
--staging_cgroup=impala_staging
--state_store_host=localhost
--state_store_subscriber_port=23000
--use_statestore=true
--local_library_dir=/tmp
--serialize_batch=false
--status_report_interval=5
--num_threads_per_core=3
--queue_wait_timeout_ms=60000
--default_pool_max_queued=50
--default_pool_max_requests=20
--default_pool_mem_limit=
--disable_pool_max_requests=false
--disable_pool_mem_limits=false
--fair_scheduler_allocation_path=
--llama_site_path=
--authorization_policy_file=
--authorization_policy_provider_class=org.apache.sentry.provider.file.HadoopGroupResourceAuthorizationProvider
--authorized_proxy_user_config=
--load_catalog_at_startup=false
--server_name=
--abort_on_failed_audit_event=true
--audit_event_log_dir=
--be_service_threads=64
--beeswax_port=21000
--cancellation_thread_pool_size=5
--default_query_options=
--fe_service_threads=64
--hs2_port=21050
--idle_query_timeout=0
--idle_session_timeout=0
--local_nodemanager_url=
--log_mem_usage_interval=0
--log_query_to_file=true
--max_audit_event_log_file_size=5000
--max_profile_log_file_size=5000
--max_result_cache_size=100000
--profile_log_dir=
--query_log_size=25
--ssl_client_ca_certificate=
--ssl_private_key=
--ssl_server_certificate=
--max_vcore_oversubscription_ratio=2.5
--rm_always_use_defaults=false
--rm_default_cpu_vcores=2
--rm_default_memory=4G
--disable_admission_control=true
--require_username=false
--statestore_subscriber_cnxn_attempts=10
--statestore_subscriber_cnxn_retry_interval_ms=3000
--statestore_subscriber_timeout_seconds=30
--state_store_port=24000
--statestore_heartbeat_frequency_ms=500
--statestore_max_missed_heartbeats=10
--statestore_num_heartbeat_threads=10
--statestore_suspect_heartbeats=5
--num_cores=0
--web_log_bytes=1048576
--non_impala_java_vlog=0
--periodic_counter_update_period_ms=500
--enable_webserver_doc_root=true
--webserver_authentication_domain=
--webserver_certificate_file=
--webserver_doc_root=/opt/cloudera/parcels/IMPALA-1.3.1-1.impala1.3.1.p0.1172/lib/impala
--webserver_interface=
--webserver_password_file=
--webserver_port=25010
--flagfile=/var/run/cloudera-scm-agent/process/1886-impala-STATESTORE/impala-conf/state_store_flags
--fromenv=
--tryfromenv=
--undefok=
--tab_completion_columns=80
--tab_completion_word=
--help=false
--helpfull=false
--helpmatch=
--helpon=
--helppackage=false
--helpshort=false
--helpxml=false
--version=false
--alsologtoemail=
--alsologtostderr=false
--drop_log_memory=true
--log_backtrace_at=
--log_dir=/var/log/statestore
--log_link=
--log_prefix=true
--logbuflevel=-1
--logbufsecs=30
--logbufvlevel=1
--logemaillevel=999
--logmailer=/bin/mail
--logtostderr=false
--max_log_size=200
--minloglevel=0
--stderrthreshold=2
--stop_logging_if_full_disk=false
--symbolize_stacktrace=true
--v=1
--vmodule=
I0813 10:10:50.275037  3245 init.cc:137] Cpu Info:
  Model: Intel(R) Xeon(R) CPU E5-2407 0 @ 2.20GHz
  Cores: 8
  L1 Cache: 32.00 KB
  L2 Cache: 256.00 KB
  L3 Cache: 10.00 MB
  Hardware Supports:
    ssse3
    sse4_1
    sse4_2
    popcnt
I0813 10:10:50.275087  3245 init.cc:138] Disk Info:
  Num disks 2:
    sda (rotational=true)
    dm- (rotational=true)

I0813 10:10:50.275172  3245 init.cc:139] Physical Memory: 31.39 GB
I0813 10:10:50.275213  3245 init.cc:140] OS version: Linux version 2.6.18-348.el5 (mockbuild@builder10.centos.org) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-54)) #1 SMP Tue Jan 8 17:53:53 EST 2013
I0813 10:10:50.275233  3245 init.cc:141] Process ID: 3245
I0813 10:10:50.275317  3245 webserver.cc:155] Starting webserver on 0.0.0.0:25010
I0813 10:10:50.275346  3245 webserver.cc:169] Document root: /opt/cloudera/parcels/IMPALA-1.3.1-1.impala1.3.1.p0.1172/lib/impala
I0813 10:10:50.275583  3245 webserver.cc:235] Webserver started
I0813 10:10:50.281631  3245 thrift-server.cc:387] ThriftServer 'StatestoreService' started on port: 24000
I0813 10:10:52.497861  3382 statestore.cc:293] Creating new topic: ''catalog-update' on behalf of subscriber: 'bi-hadoop-sm:22000
I0813 10:10:52.497964  3382 statestore.cc:293] Creating new topic: ''impala-membership' on behalf of subscriber: 'bi-hadoop-sm:22000
I0813 10:10:52.497993  3382 statestore.cc:300] Registering: bi-hadoop-sm:22000
I0813 10:10:52.498136  3325 client-cache.cc:94] CreateClient(): creating new client for bi-hadoop-sm:23000
I0813 10:10:52.498955  3382 statestore.cc:331] Subscriber 'bi-hadoop-sm:22000' registered (registration id: d44b566c7d4f17e5:12d261558a25a1bf)
I0813 10:10:53.000346  3325 statestore.cc:458] Preparing initial impala-membership topic update for bi-hadoop-sm:22000. Size = 106.00 B
I0813 10:10:55.190691  3432 statestore.cc:300] Registering: bi-hadoop-s38:22000
I0813 10:10:55.190860  3432 statestore.cc:331] Subscriber 'bi-hadoop-s38:22000' registered (registration id: 404e299449525642:3919423961822d8f)
I0813 10:10:55.191412  3332 statestore.cc:458] Preparing initial impala-membership topic update for bi-hadoop-s38:22000. Size = 106.00 B
I0813 10:10:55.191493  3332 client-cache.cc:94] CreateClient(): creating new client for bi-hadoop-s38:23000
I0813 10:10:55.669672  3444 statestore.cc:300] Registering: bi-hadoop-s39:22000
I0813 10:10:55.669821  3444 statestore.cc:331] Subscriber 'bi-hadoop-s39:22000' registered (registration id: d049c9d286b0c0a5:90098d1a4e53bc81)
I0813 10:10:55.691418  3326 statestore.cc:458] Preparing initial impala-membership topic update for bi-hadoop-s39:22000. Size = 214.00 B
I0813 10:10:55.691493  3326 client-cache.cc:94] CreateClient(): creating new client for bi-hadoop-s39:23000
I0813 10:10:56.498275  3467 statestore.cc:300] Registering: bi-hadoop-m:22000
I0813 10:10:56.498389  3467 statestore.cc:331] Subscriber 'bi-hadoop-m:22000' registered (registration id: 2c47ab5e0b47ac69:bb1e5229cc792da1)
I0813 10:10:56.498438  3333 statestore.cc:458] Preparing initial impala-membership topic update for bi-hadoop-m:22000. Size = 322.00 B
I0813 10:10:56.498477  3333 client-cache.cc:94] CreateClient(): creating new client for bi-hadoop-m:23000
I0813 10:10:56.810374  3619 statestore.cc:300] Registering: bi-hadoop-s35:22000
I0813 10:10:56.810462  3619 statestore.cc:331] Subscriber 'bi-hadoop-s35:22000' registered (registration id: 3b4548042d00bf08:40a6116030a766a3)
I0813 10:10:56.810511  3330 statestore.cc:458] Preparing initial impala-membership topic update for bi-hadoop-s35:22000. Size = 426.00 B
I0813 10:10:56.810546  3330 client-cache.cc:94] CreateClient(): creating new client for bi-hadoop-s35:23000
I0813 10:10:57.740465  3683 statestore.cc:300] Registering: bi-hadoop-s33:22000
I0813 10:10:57.740603  3683 statestore.cc:331] Subscriber 'bi-hadoop-s33:22000' registered (registration id: 414c2f5002aaed82:8cab6c8740212088)
I0813 10:10:57.740674  3331 statestore.cc:458] Preparing initial impala-membership topic update for bi-hadoop-s33:22000. Size = 534.00 B
I0813 10:10:57.740735  3331 client-cache.cc:94] CreateClient(): creating new client for bi-hadoop-s33:23000
I0813 10:10:57.959748  3699 statestore.cc:300] Registering: bi-hadoop-s36:22000
I0813 10:10:57.959874  3699 statestore.cc:331] Subscriber 'bi-hadoop-s36:22000' registered (registration id: 4e49d7914f3c0ca9:eaeb5fccce2f6ebc)
I0813 10:10:57.959945  3328 statestore.cc:458] Preparing initial impala-membership topic update for bi-hadoop-s36:22000. Size = 642.00 B
I0813 10:10:57.960011  3328 client-cache.cc:94] CreateClient(): creating new client for bi-hadoop-s36:23000
I0813 10:10:58.872663  3707 statestore.cc:300] Registering: bi-hadoop-s34:22000
I0813 10:10:58.872807  3707 statestore.cc:331] Subscriber 'bi-hadoop-s34:22000' registered (registration id: 73486c443003bff4:c92cc94de366c85)
I0813 10:10:58.872871  3327 statestore.cc:458] Preparing initial impala-membership topic update for bi-hadoop-s34:22000. Size = 750.00 B
I0813 10:10:58.872941  3327 client-cache.cc:94] CreateClient(): creating new client for bi-hadoop-s34:23000
I0813 10:11:00.013226  3710 statestore.cc:300] Registering: bi-hadoop-s40:22000
I0813 10:11:00.013355  3710 statestore.cc:331] Subscriber 'bi-hadoop-s40:22000' registered (registration id: 84487823e24a4c01:e11f38a681d38a2)
I0813 10:11:00.013402  3334 statestore.cc:458] Preparing initial impala-membership topic update for bi-hadoop-s40:22000. Size = 858.00 B
I0813 10:11:00.013456  3334 client-cache.cc:94] CreateClient(): creating new client for bi-hadoop-s40:23000

[#4 · Accepted solution]

Sorry I didn't realize this earlier, but the problem is you are using a version of CM that only works with Impala < 1.2.

You'll need to upgrade to CM 4.8 or higher in order to use Impala 1.2 or higher. This is because Impala introduced a new role, the Catalog Server, in Impala 1.2, and Cloudera Manager < 4.8 didn't know about it.

Note that CM 5.x works just fine with CDH 4.

Be sure to follow the appropriate upgrade documentation when performing your upgrade. For the lastest CM version, you'll want to start reading here, then click on "Upgrading Cloudera Manager 4 to Cloudera Manager 5":
http://www.cloudera.com/content/cloudera-content/cloudera-docs/CM5/latest/Cloudera-Manager-Administr...

Thanks,
Darren

[#5]

Ok, I see.

By the way, my cluster is not connected with Internet, so do you have any advices about upgrading offline?

[#6]

The upgrade guide should include instructions for offline installs. If it doesn't, check the install guides.

[#7]

Since upgrading CM needs more testing, I downgraded impala to 1.0 instead.

Now using impala through impala-shell is ok.

However, the impala query component in hue is still not working. It can recognize all the database created in hive, but always return "No server logs for this query" to all queries.