These steps will enable DevOps and Data Management teams to… Gain visibility and quantify high Impla resource usage by database, table, user, and query. Build near real-time visuals on daily Impala usage using Impala API calls, NiFi, and Tableau. Generate insights to action resource-intensive and problematic queries. Improve platform optimization and control user behaviors.  Tools Required   Solution Pattern  Note: This approach works for CDP Base and Data Hub, and is not supported for CDW Public or Private cloud.    The Implementation Steps in the PDF attached to this article will guide you to implement the impala usage monitoring and build the dashboards. Keyword Search  Monitor, control, and manage Impala usage, Impala Usage, Visibility and Transparency, Managing Impala Utilisation,  Impala Resource Utilization, Cloudera Manager APIs, Gain actionable insights into your impala utilization through a customized analytics and reporting use-case. ... View more

These steps will enable DevOps and Data Management teams to…   Troubleshoot critical impala query failures more accurately and near real-time using a combination of Cloudera Manager (CM) API, Nifi, and BI tools to: Gain automated visibility, transparency, and alerts of impala query failures Continuously monitor and track Impala's health Identify, classify, and label critical failure types (10) to address and types to ignore Quickly identify critical failure causes Visualize failures in one place including trends by count and failure rates Access actionable insights to recover from incidents quickly and prevent future failures Automate end-user issue notification    Recommendations provided later will focus only on the following 10 high-impacting critical query failures:    Sr No. Label Name Query Failure Type  1 oom Query failed with a status of Out Of Memory Error. 2 stats Missing  Query failed with a status of statmissing = true. Indicates missing computing info about table/column level statistics.  3 stats corrupt Query failed with a status of statscorrupt = true. Indicates corrupt table level block statistics.  4 disk io Query failed with a status of disk IO error. Indicates extensive disk input/output operations. Could also be related to service/platform level issues.  5 unreachable hosts Query failed with a status of unreachable impala demon. Indicates connectivity and unreachability to impala demons.  6 rpc failed Query failed with rpc error. Indicates impala demon connectivity issues. 7 canceled Queries failed with Canceled exception highlights. Due to query abortion from the user or service end. 8 session closed Query failed with a status of client inactivity or no active connection. 9 udf error The query failed with a status of  UDF error. Indicates the user's incorrect function or data parsing logic. 10 rejected from pool Queries failed with a status of Rejected query form pool.  Tools Required Solution Pattern   Note: This approach works for CDP Base and Data Hub, and is not supported for CDW Public or Private cloud.   The implementation steps in the PDF attached to this article will guide you to implement the impala query monitoring and alerting mechanism and build the dashboards.    Keyword Search  Critical Impala query exception, Monitor critical Impala query failures, Manage critical Impala query failures, Visibility and Transparency, Gain Actionable insights into Impala query failures, Root cause Impala failures, Impala Query Failures, Identifying critical Impala query failures, Impala query failures. ... View more

These steps will enable DevOps and Data Management teams to… Out of the box, Cloudera Manager provides plenty of valuable health checks to highlight platform and service issues that may need to be addressed along with an intuitive interface for troubleshooting issues. Nevertheless, the greater the quantity of pipelines and processes the platform supports (especially at peak loads/times), the higher the probability that you would benefit from additional visibility into overall cluster performance. Cloudera offers additional CM dashboards.    You can also augment those helpful Cloudera managers (CM) charts and additional dashboards by tapping into CM API calls to gain full visibility into your cluster-generated alerts. Doing so will provide an end-to-end customizable health alert monitoring system and enable you to: Expand your visibility into historical service-level and cluster-level alerts Automate manual monitoring activities Track weekly and daily metrics, KPIs, and alert trends in dashboards (templates provided)  Suppress noise Find hidden patterns in recurring cluster issues to troubleshoot and diagnose more quickly and effectively   Act promptly to resolve root-cause issues Become more proactive with incident resolution  Tools Required  Solution Pattern The implementation steps in the PDF will guide you to implement the API calls and build the dashboards.    Keyword Search CM Alerts, Cloudera Manager Alerts, Monitor Alerts, Manage Alerts, Visibility, Transparency, Visibility and transparency, Monitor and Manage Cloudera Manager Alerts, Gain actionable insights into your CM alerts through customized analytics and reporting, Setup customized reporting and analytical monitoring framework that helps you manage and monitor your Cloudera Manager generated Alert,  Additional CM report, Additional Cloudera Manager report, Root cause CM issues, Root cause Cloudera Manager issues, CM API, Cloudera Manager API.     ... View more