Cloudera Data Analytics (CDA) Articles

Announcements
Celebrating as our community reaches 100,000 members! Thank you!
avatar
Cloudera Employee

image1.png

These steps will enable DevOps and Data Management teams to…

  • Gain near real-time visibility into day-to-day operational data engineering processes.

  • Automate manual monitoring activities and suppress noise.

  • Build near real-time visualizations to monitor those processes and trends using Cloudera Manager (CM) API, NIFI Pipeline, and NiFi APIs.

  • Find hidden patterns in process issues to troubleshoot, diagnose, and quickly action failing, troublesome, and resource-intensive processes.

  • Mine for insights into Yarn, Cloudera Data Science Workbench (CDSW), AirFlow, and NIFI processor logs/database.

  • Improve platform optimization and control user behaviors.

  • Become more proactive with incident resolution.

We will provide you with a framework that runs locally on your cluster and is customizable by security-restricted telemetry tools that require data to be sent outside the cluster. The framework provides insights into the processes that trigger workloads on the Cloudera Private Cloud Base platform, including:

 

KPI Monitoring Framework 

Scheduling / ETL Tool

Workloads Triggered

API / Data Source Used

Airflow Jobs Monitor 

Airflow - Python, SPARK

Yarn jobs status

Python script - Airflow database

CDSW Jobs Monitor 

CDSW - Python

CDSW jobs status 

Python script - Cdsw database

NIFI Processor Monitor 

NiFi - Python

NIFI processor status

NIFI Flow - NIFI app logs 

Spark/Yarn jobs Monitor

NIFI - Python

Spark/Yarn jobs status

NIFI Flow - Python script

Tools Required

image1.png

Solution Pattern

image3.png

These steps will guide you to implement monitoring and alerting mechanisms into your engineering service processes and build the dashboards.

 

Refer to the following links for articles on implementation steps for various monitoring processes,

 

Keyword Search 

Transparency, Visibility, Log data, Database data, Monitor AirFlow, Monitor CDSW, Monitor SPARK, Monitor NiFi Processor, Manage AirFlow, Manage CDSW, Manage SPARK, Manage NiFi Processor, Actionable insights, Operational processes, Data engineering processes, Yarn, Cloudera Manager (CM) API, NiFi Process logs, Visualize failures, Visualize jobs, Visualize processes.

677 Views
0 Kudos