Created on 05-04-202304:06 AM - edited 05-05-202304:18 PM
These steps will enable DevOps and Data Management teams to…
Troubleshoot critical impala query failures more accurately and near real-time using a combination of Cloudera Manager (CM) API, Nifi, and BI tools to:
Gain automated visibility, transparency, and alerts of impala query failures
Continuously monitor and track Impala's health
Identify, classify, and label critical failure types (10) to address and types to ignore
Quickly identify critical failure causes
Visualize failures in one place including trends by count and failure rates
Access actionable insights to recover from incidents quickly and prevent future failures
Automate end-user issue notification
Recommendations provided later will focus only on the following 10 high-impacting critical query failures:
Sr No.
Label Name
Query Failure Type
1
oom
Query failed with a status of Out Of Memory Error.
2
stats Missing
Query failed with a status of statmissing = true. Indicates missing computing info about table/column level statistics.
3
stats corrupt
Query failed with a status of statscorrupt = true. Indicates corrupt table level block statistics.
4
disk io
Query failed with a status of disk IO error. Indicates extensive disk input/output operations. Could also be related to service/platform level issues.
5
unreachable hosts
Query failed with a status of unreachable impala demon. Indicates connectivity and unreachability to impala demons.
6
rpc failed
Query failed with rpc error. Indicates impala demon connectivity issues.
7
canceled
Queries failed with Canceled exception highlights. Due to query abortion from the user or service end.
8
session closed
Query failed with a status of client inactivity or no active connection.
9
udf error
The query failed with a status of UDF error. Indicates the user's incorrect function or data parsing logic.
10
rejected from pool
Queries failed with a status of Rejected query form pool.
Tools Required
Solution Pattern
Note: This approach works for CDP Base and Data Hub, and is not supported for CDW Public or Private cloud.
The implementation steps in the PDF attached to this article will guide you to implement the impala query monitoring and alerting mechanism and build the dashboards.
Keyword Search
Critical Impala query exception, Monitor critical Impala query failures, Manage critical Impala query failures, Visibility and Transparency, Gain Actionable insights into Impala query failures, Root cause Impala failures, Impala Query Failures, Identifying critical Impala query failures, Impala query failures.