- Subscribe to RSS Feed
- Mark as New
- Mark as Read
- Bookmark
- Subscribe
- Printer Friendly Page
- Report Inappropriate Content
Created on
05-04-2023
12:01 AM
- edited on
05-05-2023
06:45 AM
by
VidyaSargur
These steps will enable DevOps and Data Management teams to…
-
Gain near real-time visibility into day-to-day operational data engineering processes.
-
Automate manual monitoring activities and suppress noise.
-
Build near real-time visualizations to monitor those processes and trends using Cloudera Manager (CM) API, NIFI Pipeline, and NiFi APIs.
-
Find hidden patterns in process issues to troubleshoot, diagnose, and quickly action failing, troublesome, and resource-intensive processes.
-
Mine for insights into Yarn, Cloudera Data Science Workbench (CDSW), AirFlow, and NIFI processor logs/database.
-
Improve platform optimization and control user behaviors.
-
Become more proactive with incident resolution.
We will provide you with a framework that runs locally on your cluster and is customizable by security-restricted telemetry tools that require data to be sent outside the cluster. The framework provides insights into the processes that trigger workloads on the Cloudera Private Cloud Base platform, including:
KPI Monitoring Framework |
Scheduling / ETL Tool |
Workloads Triggered |
API / Data Source Used |
Airflow Jobs Monitor |
Airflow - Python, SPARK |
Yarn jobs status |
Python script - Airflow database |
CDSW Jobs Monitor |
CDSW - Python |
CDSW jobs status |
Python script - Cdsw database |
NIFI Processor Monitor |
NiFi - Python |
NIFI processor status |
NIFI Flow - NIFI app logs |
Spark/Yarn jobs Monitor |
NIFI - Python |
Spark/Yarn jobs status |
NIFI Flow - Python script |
Tools Required
Solution Pattern
Refer to the following links for articles on implementation steps for various monitoring processes,
Keyword Search
Transparency, Visibility, Log data, Database data, Monitor AirFlow, Monitor CDSW, Monitor SPARK, Monitor NiFi Processor, Manage AirFlow, Manage CDSW, Manage SPARK, Manage NiFi Processor, Actionable insights, Operational processes, Data engineering processes, Yarn, Cloudera Manager (CM) API, NiFi Process logs, Visualize failures, Visualize jobs, Visualize processes.