Created on 05-04-202312:01 AM - edited on 05-05-202306:45 AM by VidyaSargur
These steps will enable DevOps and Data Management teams to…
Gain near real-time visibility into day-to-day operational data engineering processes.
Automate manual monitoring activities and suppress noise.
Build near real-time visualizations to monitor those processes and trends using Cloudera Manager (CM) API, NIFI Pipeline, and NiFi APIs.
Find hidden patterns in process issues to troubleshoot, diagnose, and quickly action failing, troublesome, and resource-intensive processes.
Mine for insights into Yarn, Cloudera Data Science Workbench (CDSW), AirFlow, and NIFI processor logs/database.
Improve platform optimization and control user behaviors.
Become more proactive with incident resolution.
We will provide you with a framework that runs locally on your cluster and is customizable by security-restricted telemetry tools that require data to be sent outside the cluster. The framework provides insights into the processes that trigger workloads on the Cloudera Private Cloud Base platform, including:
KPI Monitoring Framework
Scheduling / ETL Tool
API / Data Source Used
Airflow Jobs Monitor
Airflow - Python, SPARK
Yarn jobs status
Python script - Airflow database
CDSW Jobs Monitor
CDSW - Python
CDSW jobs status
Python script - Cdsw database
NIFI Processor Monitor
NiFi - Python
NIFI processor status
NIFI Flow - NIFI app logs
Spark/Yarn jobs Monitor
NIFI - Python
Spark/Yarn jobs status
NIFI Flow - Python script
These steps will guide you to implement monitoring and alerting mechanisms into your engineering service processes and build the dashboards.
Refer to the following links for articles on implementation steps for various monitoring processes,