Cannot monitor impala queries on Cloudera Manager Impala Queries Tab

[#1]

I got installed a new cloudera cluster with versions CM-6.3.1 and CDH-6.3.2 and now facing a problem never had with previous cloudera distributions.

Cloudera Management Services (including Cloudera Service Monitor), Impala (Sentry Enabled), and all other related processes are working fine. Individual users are able to authenticate with LDAP and can query tables to which they are authorized.

The problem is, the queries sent and executed to impala daemons do not appear in the Impala Queries tab in Cloudera Manager, although they can be observed in Completed Impala Queries chart, that is, there are changes in the queries/second metric when someone send a query, i checked the things discussed in the link below.

https://community.cloudera.com/t5/Support-Questions/Empty-Impala-queries-in-Cloudera-Manager/td-p/79...

No troubles in either /var/log/cloudera-scm-firehose/mgmt-cmf-mgmt-SERVICEMONITOR-hostname.log.out or /var/log/cloudera-scm-server/cloudera-scm-server.log logs.

There is enough disk space in the Service Monitor Storage Directory.

And tried to set to different sizes from 1GB to 3GB Impala Storage (firehose_impala_storage_bytes)

Everything seems fine, yet there are no queries in the Cloudera Manager Impala Queries Web-UI. What might be the cause of this situation?

Any help would be highly appreciated,

Thanks.

[#2]

@legendarier,

Please go to CM > Cloudera Management Services > Instances > Service Monitor > Charts Library > Service Monitor Storage, and check impala-query-monitoring and see how old the data is (whether it has any data or not).

See my screenshot below:

This can help to confirm if data is stored correctly or not.

In detail, CM agent will get two types of Impala queries and sends to Service Monitor, which will in term display in CM interface:

1. in flight queries

CM agent will fetch in flight query IDs via API call: http(s)://{host}:{impalad-port}/inflight_query_ids

CM agent will fetch in flight query details via API call: http(s)://{host}:{impalad-port}/query_profile_encoded?query_id={query_id}

Above details will be combined and sent to Service Monitor service

2. finished queries

CM agent will parse the query profiles under /var/log/impalad/profiles on the impala daemon host and sends data to Service monitor.

So to troubleshoot the issue, you can check:

a. CM agent log on the impala coordinator host to see if there is any impala related errors

b. Service Monitor server log to see if there is any impala related errors

Hope above helps.

Cheers
Eric