In a series of articles Cloudera’s Data Analytics team shares insights into how they started their journey towards operational excellence. These articles provide guidelines on how you can leverage Cloudera and supporting tool APIs to harness the platform and the process data. But technology is just a facilitator, inside these articles you will find insights into methodology to gain insights into your platform , the processes running in it, how to customize the insights to your particular ecosystem and how to use it to achieve operational excellence. Modern CDP platform built on an open source technology stack provides great flexibility to harvest platform data and it is up to us to use that data to build meaningful and actionable insights. From one operational data practitioner to another, these articles are a must read. Here are the links to the articles in this series. # Topic Link Address 1 Gaining Control of Your CDP Environment https://community.cloudera.com/t5/Cloudera-Data-Analytics-CDA/Gaining-Control-of-Your-CDP-Environment/ta-p/369657 2 Visibility and Transparency https://community.cloudera.com/t5/Cloudera-Data-Analytics-CDA/Visibility-and-Transparency/ta-p/369919 2.1 Deploy your Cloudera Manager Dashboards https://community.cloudera.com/t5/Cloudera-Data-Analytics-CDA/Deploy-your-Cloudera-Manager-Dashboards/ta-p/369923 2.2 CDSW Monitoring - Operational Data Engineering Process Monitoring https://community.cloudera.com/t5/Cloudera-Data-Analytics-CDA/CDSW-Monitoring-Operational-Data-Engineering-Process/ta-p/370005 2.3 Operational Data Engineering Process Monitoring - Transparency and Visibility https://community.cloudera.com/t5/Cloudera-Data-Analytics-CDA/Operational-Data-Engineering-Process-Monitoring-Transparency/ta-p/370013 2.4 Yarn/Spark jobs Monitoring Implementation Steps - Operational Data Engineering Process Monitoring https://community.cloudera.com/t5/Cloudera-Data-Analytics-CDA/Yarn-Spark-jobs-Monitoring-Implementation-Steps-nbsp/ta-p/370015 2.5 Airflow jobs Monitoring Implementation Steps - Operational Data Engineering Process Monitoring https://community.cloudera.com/t5/Cloudera-Data-Analytics-CDA/Airflow-jobs-Monitoring-Implementation-Steps-nbsp/ta-p/370017 2.6 Cron jobs Monitoring Implementation Steps - Operational Data Engineering Process Monitoring https://community.cloudera.com/t5/Cloudera-Data-Analytics-CDA/Cron-jobs-Monitoring-Implementation-Steps-nbsp-Operational/ta-p/370022 2.7 NiFi Job Monitoring Implementation Steps - Operational Data Engineering Process Monitoring https://community.cloudera.com/t5/Cloudera-Data-Analytics-CDA/NiFi-Job-Monitoring-Implementation-Steps-Operational-Data/ta-p/370023 2.8 Handling Impala Critical Query Exceptions/Failures - Transparency and Visibility https://community.cloudera.com/t5/Cloudera-Data-Analytics-CDA/Handling-Impala-Critical-Query-Exceptions-Failures/ta-p/370048 2.9 Monitoring Impala Usage - Transparency and Visibility https://community.cloudera.com/t5/Cloudera-Data-Analytics-CDA/Monitoring-Impala-Usage-Transparency-and-Visibility/ta-p/370058 2.10 Leveraging Cloudera Manager Cluster Generated Alerts To Gain Control Of Your CDP Environment - Transparency and Visibility https://community.cloudera.com/t5/Cloudera-Data-Analytics-CDA/Leveraging-Cloudera-Manager-Cluster-Generated-Alerts-To-Gain/ta-p/370040   ... View more

Transparency and Visibility - Operational Data Engineering Process Monitoring - NiFi jobs Monitoring Implementation Steps   Part of the Visibility and Transparency series this article provides steps for the implementation of NiFi processes or workloads monitoring and alerting system. The framework uses Apache Hive, Apache NiFi, and BI visualization tools. The features include customized dashboards for insights and alert mechanisms.   The attachment to this article provides the details of the implementation steps. ... View more

Part of the Visibility and Transparency series this article provides steps for the implementation of the Cron processes alerting system. The framework uses Apache Hive, Python, and Airflow. The features include an alert mechanism.   The attachment to this article details the implementation steps. ... View more

Part of the Visibility and Transparency series this article provides steps for the implementation of Airflow processes or workloads monitoring and alerting system. The framework uses Apache Hive, Python, and BI visualization tools. The features include customized dashboards for insights and alert mechanism.   The attachment to this article details the implementation steps. ... View more

Part of the Visibility and Transparency series this article provides steps for the implementation of Yarn/Spark processes or workloads monitoring and alerting system. The framework uses Apache Hive, Apache NiFi, and BI visualization tools. The features include customized visualization dashboards for insights and alert mechanisms.   The attachment to this article details the implementation steps. ... View more

These steps will enable DevOps and Data Management teams to… Gain near real-time visibility into day-to-day operational data engineering processes. Automate manual monitoring activities and suppress noise. Build near real-time visualizations to monitor those processes and trends using Cloudera Manager (CM) API, NIFI Pipeline, and NiFi APIs. Find hidden patterns in process issues to troubleshoot, diagnose, and quickly action failing, troublesome, and resource-intensive processes. Mine for insights into Yarn, Cloudera Data Science Workbench (CDSW), AirFlow, and NIFI processor logs/database. Improve platform optimization and control user behaviors. Become more proactive with incident resolution. We will provide you with a framework that runs locally on your cluster and is customizable by security-restricted telemetry tools that require data to be sent outside the cluster. The framework provides insights into the processes that trigger workloads on the Cloudera Private Cloud Base platform, including:   KPI Monitoring Framework  Scheduling / ETL Tool Workloads Triggered API / Data Source Used Airflow Jobs Monitor  Airflow - Python, SPARK Yarn jobs status Python script - Airflow database CDSW Jobs Monitor  CDSW - Python CDSW jobs status  Python script - Cdsw database NIFI Processor Monitor  NiFi - Python NIFI processor status NIFI Flow - NIFI app logs  Spark/Yarn jobs Monitor NIFI - Python Spark/Yarn jobs status NIFI Flow - Python script Tools Required Solution Pattern These steps will guide you to implement monitoring and alerting mechanisms into your engineering service processes and build the dashboards.   Refer to the following links for articles on implementation steps for various monitoring processes,   Airflow Monitoring CDSW Monitoring NiFi Monitoring Spark and Yarn Monitoring Cron Jobs Monitoring Keyword Search  Transparency, Visibility, Log data, Database data, Monitor AirFlow, Monitor CDSW, Monitor SPARK, Monitor NiFi Processor, Manage AirFlow, Manage CDSW, Manage SPARK, Manage NiFi Processor, Actionable insights, Operational processes, Data engineering processes, Yarn, Cloudera Manager (CM) API, NiFi Process logs, Visualize failures, Visualize jobs, Visualize processes. ... View more

Part of the Visibility and Transparency series this article provides steps for the implementation of Cloudera Data Science Workbench processes or workloads monitoring and alerting system. The framework uses Apache Hive, Apache Airflow, Bash shell scripts, and BI Visualization tools. The framework features include a customized visualization dashboard for insights and alert mechanisms.   The attachment to this article details the implementation steps. ... View more

Keyword Search Deploy Cloudera Manager Dashboards Free Manager Dashboards Cloudera Manager Troubleshooting Monitor CDP Cluster Proactively resolve Cloudera Manager alerts Cloudera Manager Monitoring JSON Health monitoring Impala health monitoring YARN health monitoring HBase tuning HDFS caching Datanode health monitoring HDFS NN & JN health monitoring Hive health monitoring Impala monitor coordinators Infra-Solr health monitoring Solr health monitoring HDFS JVM GC Sizing YARN JVM GC Sizing MGMT Cluster JVM GC Sizing ZK JVM GC Sizing Oozie JVM GC Sizing Hive JVM GC Sizing Summary Out of the box, Cloudera Manager provides plenty of valuable health checks to highlight platform and service issues that may need to be addressed along with an intuitive interface for troubleshooting issues. Nevertheless, the greater the quantity of pipelines and processes the platform supports (especially at peak loads/times), the higher the probability that you would benefit from additional visibility into overall cluster performance. To that end, we’re offering you JSON files of custom Cloudera Manager dashboards to import into your CM ’managed cluster: Here are the links (Change this to Snapshots) to the dashboards we will preview below: NOTE - you must change the user ID that is within each JSON file from ‘mbush’ to a suitable admin user account within your own environment, prior to uploading them in Cloudera Manager.     Three Daily Dashboards We recommend using the following dashboards every day: MB - Overall Health MB - Impala Health MB - YARN Health The Overall Health dashboard depicts a mix of insights across the entire cluster to identify abnormalities affecting any node when it comes to CPU, memory, etc.   NOTE: the ‘Time Series’ in the top right corner of every screenshot which shows the monitoring time period. The longer the time period set, the more skewed the statistics can become within the CM dashboard due to metric aggregation.    MB - Overall Health (screenshots)     In any CDP cluster, it is important to verify that the Impala service is healthy. It is common to see that ETL-style workloads have been deployed within the Impala service, where Hive, Spark, or YARN would have been more suitable. The 3 dashboards depicted below can help analyze workload balance and provide valuable insights you can use to begin migrating those workloads from Impala to YARN.   MB - Impala Health (screenshots)   MB - YARN Health (screenshots) Seven Weekly Dashboards We recommend using the following dashboards weekly, or more frequently to identify unexpected behavior or instability within your cluster: MB - HBase Tuning HDFS Caching & Datanode Health MB - HDFS NN & JN Health MB - Hive Health MB - Impala Monitor Co-ordinators MB - Infra-Solr Health MB - Solr Health MB - HBase Tuning (screenshots)   HDFS Caching & Datanode Health (screenshots) MB - HDFS NN & JN Health (screenshots) MB - Hive Health (screenshots) MB - Impala Monitor Co-ordinators (screenshots) MB - Infra-Solr Health (screenshots) MB - Solr Health (screenshots) Three Monthly Dashboards As with the Weekly Used Dashboards, these dashboards can quite often be used daily if identifying unexpected behavior or instability within the cluster but we recommend using them at least monthly to proactively identify that service/role heap sizing is in a healthy state: MB - HDFS & YARN - JVM GC Sizing MB - MGMT Cluster - JVM GC Sizing MB - ZK, Oozie & Hive - JVM GC Sizing MB - HDFS & YARN - JVM GC Sizing (screenshots) MB - MGMT Cluster - JVM GC Sizing (screenshots) MB - ZK, Oozie & Hive - JVM GC Sizing (screenshots) ... View more

Summary Out of the box, Cloudera Manager provides plenty of valuable health checks to highlight platform and service issues that may need to be addressed along with an intuitive interface for troubleshooting issues. Nevertheless, the greater the quantity of pipelines and processes the platform supports (especially at peak loads/times), the higher the probability that you would benefit from additional visibility into overall cluster performance. To that end, we’re offering you JSON files of custom Cloudera Manager dashboards to import into your CM ’managed cluster.   The attachment to this article details steps for creating and deploying Cloudera Manager dashboards. Keyword Search Deploy Cloudera Manager Dashboards, Free Manager Dashboards, Cloudera Manager Troubleshooting, Monitor CDP Cluster, Proactively resolve Cloudera Manager alerts, Cloudera Manager Monitoring, JSON, Health monitoring, Impala health monitoring, YARN health monitoring, HBase tuning, HDFS caching, Datanode health monitoring, HDFS NN & JN health monitoring, Hive health monitoring, Impala monitor coordinators, Infra-Solr health monitoring, Solr health monitoring, HDFS JVM GC Sizing, YARN JVM GC Sizing, MGMT Cluster JVM GC Sizing, ZK JVM GC Sizing, Oozie JVM GC Sizing, Hive JVM GC Sizing ... View more