Operational Data Engineering Process Monitoring - Transparency and Visibility

[#1]

These steps will enable DevOps and Data Management teams to…

• Gain near real-time visibility into day-to-day operational data engineering processes.

• Automate manual monitoring activities and suppress noise.

• Build near real-time visualizations to monitor those processes and trends using Cloudera Manager (CM) API, NIFI Pipeline, and NiFi APIs.

• Find hidden patterns in process issues to troubleshoot, diagnose, and quickly action failing, troublesome, and resource-intensive processes.

• Mine for insights into Yarn, Cloudera Data Science Workbench (CDSW), AirFlow, and NIFI processor logs/database.

• Improve platform optimization and control user behaviors.

• Become more proactive with incident resolution.

We will provide you with a framework that runs locally on your cluster and is customizable by security-restricted telemetry tools that require data to be sent outside the cluster. The framework provides insights into the processes that trigger workloads on the Cloudera Private Cloud Base platform, including:

KPI Monitoring Framework	Scheduling / ETL Tool	Workloads Triggered	API / Data Source Used
Airflow Jobs Monitor	Airflow - Python, SPARK	Yarn jobs status	Python script - Airflow database
CDSW Jobs Monitor	CDSW - Python	CDSW jobs status	Python script - Cdsw database
NIFI Processor Monitor	NiFi - Python	NIFI processor status	NIFI Flow - NIFI app logs
Spark/Yarn jobs Monitor	NIFI - Python	Spark/Yarn jobs status	NIFI Flow - Python script

Tools Required

Solution Pattern

These steps will guide you to implement monitoring and alerting mechanisms into your engineering service processes and build the dashboards.

Refer to the following links for articles on implementation steps for various monitoring processes,

• Airflow Monitoring
• CDSW Monitoring
• NiFi Monitoring
• Spark and Yarn Monitoring
• Cron Jobs Monitoring

Keyword Search 

Transparency, Visibility, Log data, Database data, Monitor AirFlow, Monitor CDSW, Monitor SPARK, Monitor NiFi Processor, Manage AirFlow, Manage CDSW, Manage SPARK, Manage NiFi Processor, Actionable insights, Operational processes, Data engineering processes, Yarn, Cloudera Manager (CM) API, NiFi Process logs, Visualize failures, Visualize jobs, Visualize processes.