These steps will enable DevOps and Data Management teams to…
Troubleshoot critical impala query failures more accurately and near real-time using a combination of Cloudera Manager (CM) API, Nifi, and BI tools to:
- Gain automated visibility, transparency, and alerts of impala query failures
- Continuously monitor and track Impala's health
- Identify, classify, and label critical failure types (10) to address and types to ignore
- Quickly identify critical failure causes
- Visualize failures in one place including trends by count and failure rates
- Access actionable insights to recover from incidents quickly and prevent future failures
- Automate end-user issue notification
Recommendations provided later will focus only on the following 10 high-impacting critical query failures:
Sr No. | Label Name | Query Failure Type |
1 | oom | Query failed with a status of Out Of Memory Error. |
2 | stats Missing | Query failed with a status of statmissing = true. Indicates missing computing info about table/column level statistics. |
3 | stats corrupt | Query failed with a status of statscorrupt = true. Indicates corrupt table level block statistics. |
4 | disk io | Query failed with a status of disk IO error. Indicates extensive disk input/output operations. Could also be related to service/platform level issues. |
5 | unreachable hosts | Query failed with a status of unreachable impala demon. Indicates connectivity and unreachability to impala demons. |
6 | rpc failed | Query failed with rpc error. Indicates impala demon connectivity issues. |
7 | canceled | Queries failed with Canceled exception highlights. Due to query abortion from the user or service end. |
8 | session closed | Query failed with a status of client inactivity or no active connection. |
9 | udf error | The query failed with a status of UDF error. Indicates the user's incorrect function or data parsing logic. |
10 | rejected from pool | Queries failed with a status of Rejected query form pool. |
Tools Required
Solution Pattern
Note: This approach works for CDP Base and Data Hub, and is not supported for CDW Public or Private cloud.
The implementation steps in the PDF attached to this article will guide you to implement the impala query monitoring and alerting mechanism and build the dashboards.
Keyword Search
Critical Impala query exception, Monitor critical Impala query failures, Manage critical Impala query failures, Visibility and Transparency, Gain Actionable insights into Impala query failures, Root cause Impala failures, Impala Query Failures, Identifying critical Impala query failures, Impala query failures.
