Hi There,
We are currently using Cloudera On-Prem (CDP 7.1.9SP1, CFM 2.1.7, CM 7.13.1), where monitoring of Spark jobs (such as memory utilization, CPU usage, active jobs, completed jobs, and related metrics) appears to be primarily available through the UI.
We explored several approaches for retrieving Spark Structured Streaming and job execution metrics programmatically. However, most publicly available documentation references accessing the Spark Driver UI via port 4040 (same case with 4041,4042,4043 etc) on the driver node. In our Cloudera environment, this endpoint does not appear to be operational or accessible, which limits our ability to leverage those documented methods.
Could you please point us in the right direction or suggest alternative approaches for retrieving such monitoring data through command-line interfaces, APIs, logs, event stores, or any other available mechanisms within Cloudera?
At this stage, we are not looking for a specific metric alone. Our objective is to understand all available monitoring options and data sources that could provide visibility into Spark and Structured Streaming workloads. A comprehensive understanding of the underlying monitoring capabilities will help us establish the fundamentals required for future automation and operational observability.
The only significant constraint is that the solution should preferably be accessible via command-line or scriptable interfaces, as our longer-term goal is to automate collection, analysis, and alerting across a broad range of Spark runtime and streaming metrics.
