Cloudera Data Analytics (CDA) Articles

avatar
Cloudera Employee

These steps will enable DevOps and Data Management teams to…

 

Troubleshoot critical impala query failures more accurately and near real-time using a combination of Cloudera Manager (CM) API, Nifi, and BI tools to:

  • Gain automated visibility, transparency, and alerts of impala query failures
  • Continuously monitor and track Impala's health
  • Identify, classify, and label critical failure types (10) to address and types to ignore
  • Quickly identify critical failure causes
  • Visualize failures in one place including trends by count and failure rates
  • Access actionable insights to recover from incidents quickly and prevent future failures
  • Automate end-user issue notification 

 

Recommendations provided later will focus only on the following 10 high-impacting critical query failures: 

 

Sr No.

Label Name

Query Failure Type 

1

oom

Query failed with a status of Out Of Memory Error.

2

stats Missing 

Query failed with a status of statmissing = true. Indicates missing computing info about table/column level statistics. 

3

stats corrupt

Query failed with a status of statscorrupt = true. Indicates corrupt table level block statistics. 

4

disk io

Query failed with a status of disk IO error. Indicates extensive disk input/output operations. Could also be related to service/platform level issues. 

5

unreachable hosts

Query failed with a status of unreachable impala demon. Indicates connectivity and unreachability to impala demons. 

6

rpc failed

Query failed with rpc error. Indicates impala demon connectivity issues.

7

canceled

Queries failed with Canceled exception highlights. Due to query abortion from the user or service end.

8

session closed

Query failed with a status of client inactivity or no active connection.

9

udf error

The query failed with a status of  UDF error. Indicates the user's incorrect function or data parsing logic.

10

rejected from pool

Queries failed with a status of Rejected query form pool. 

Tools Required

image1.png

Solution Pattern

 

image2.png

Note: This approach works for CDP Base and Data Hub, and is not supported for CDW Public or Private cloud.

 

The implementation steps in the PDF attached to this article will guide you to implement the impala query monitoring and alerting mechanism and build the dashboards

 

Keyword Search 

Critical Impala query exception, Monitor critical Impala query failures, Manage critical Impala query failures, Visibility and Transparency, Gain Actionable insights into Impala query failures, Root cause Impala failures, Impala Query Failures, Identifying critical Impala query failures, Impala query failures.

1,307 Views
0 Kudos