Operational Data Engineering Process Monitoring - Transparency and Visibility

These steps will enable DevOps and Data Management teams to…

• Gain near real-time visibility into day-to-day operational data engineering processes.

• Automate manual monitoring activities and suppress noise.

• Build near real-time visualizations to monitor those processes and trends using Cloudera Manager (CM) API, NIFI Pipeline, and NiFi APIs.

• Find hidden patterns in process issues to troubleshoot, diagnose, and quickly action failing, troublesome, and resource-intensive processes.

• Mine for insights into Yarn, Cloudera AI Workbench, AirFlow, and NIFI processor logs/database.

• Improve platform optimization and control user behaviors.

• Become more proactive with incident resolution.

We will provide you with a framework that runs locally on your cluster and is customizable by security-restricted telemetry tools that require data to be sent outside the cluster. The framework provides insights into the processes that trigger workloads on the Cloudera Private Cloud Base platform, including:

Tools Required

Solution Pattern

Refer to the following links for articles on implementation steps for various monitoring processes,

• Airflow Monitoring
• CDSW Monitoring
• NiFi Monitoring
• Spark and Yarn Monitoring
• Cron Jobs Monitoring

Keyword Search 

Transparency, Visibility, Log data, Database data, Monitor AirFlow, Monitor CDSW, Monitor SPARK, Monitor NiFi Processor, Manage AirFlow, Manage CDSW, Manage SPARK, Manage NiFi Processor, Actionable insights, Operational processes, Data engineering processes, Yarn, Cloudera Manager (CM) API, NiFi Process logs, Visualize failures, Visualize jobs, Visualize processes.