question Re: Comprehensive monitoring command line Spark 3x in Support Questions

Comprehensive monitoring command line Spark 3x

vowxx007 — Tue, 15 Sep 2026 04:00:07 GMT

Hi There,

We are currently using Cloudera On-Prem (CDP 7.1.9SP1, CFM 2.1.7, CM 7.13.1), where monitoring of Spark jobs (such as memory utilization, CPU usage, active jobs, completed jobs, and related metrics) appears to be primarily available through the UI.

We explored several approaches for retrieving Spark Structured Streaming and job execution metrics programmatically. However, most publicly available documentation references accessing the Spark Driver UI via port 4040 (same case with 4041,4042,4043 etc) on the driver node. In our Cloudera environment, this endpoint does not appear to be operational or accessible, which limits our ability to leverage those documented methods.

Could you please point us in the right direction or suggest alternative approaches for retrieving such monitoring data through command-line interfaces, APIs, logs, event stores, or any other available mechanisms within Cloudera?

At this stage, we are not looking for a specific metric alone. Our objective is to understand all available monitoring options and data sources that could provide visibility into Spark and Structured Streaming workloads. A comprehensive understanding of the underlying monitoring capabilities will help us establish the fundamentals required for future automation and operational observability.

The only significant constraint is that the solution should preferably be accessible via command-line or scriptable interfaces, as our longer-term goal is to automate collection, analysis, and alerting across a broad range of Spark runtime and streaming metrics.

Re: Comprehensive monitoring command line Spark 3x

VidyaSargur — Tue, 15 Sep 2026 08:59:24 GMT

@vowxx007, Welcome to our community! To help you get the best possible answer, I have tagged in our CDP experts @Scharan @shehbazk @vaishaakb who may be able to assist you further.

Please feel free to provide any additional information or details about your query, and we hope that you will find a satisfactory solution to your question.

Re: Comprehensive monitoring command line Spark 3x

vafs — Tue, 15 Sep 2026 16:38:56 GMT

Hello @vowxx007,

The driver port is ephimeral and only exist during the app executions, so for your intention it will not work.

There are some options, for example the Spark History Server REST API.
For example, http://<server-url>:18080/api/v1/applications returns a JSON with data for both active and completed apps.
Then you can play with the options this API offers.
Note: check and confirm the SHS Port on the Spark service configuration.

Another tool that may work is the YARN Resource Manager REST API there you can use options such as "Cluster Applications API".
This one will be something like this: http://rm-http-address:port/ws/v1/cluster/apps
That option will give information about the user, start and finish time, status, and more.

Hopefully this help you on your monitoring tasks.