question Re: Capture airflow run duration in Support Questions

Capture airflow run duration

wert_1311 — Tue, 21 Apr 2026 06:27:37 GMT

I have a requirement to gather run duration (time) for the last 3 months, for a particular airflow job.

In our CDE environment we use airflow to call spark DBT jobs, of late the run duration of job have increased drastically.
I assume there would be a way to gather this jobs runtime duration for further analysis etc. Hoping for some assistance / guidance in getting his done.

CDE Version - 1.19.3-b29

Thanks

Wert

Re: Capture airflow run duration

vaishaakb — Mon, 19 Aug 2024 14:28:07 GMT

Hello @wert_1311 Thank you for bringing this to our Community.

I see this has already by requested by you here:
[0a]https://community.cloudera.com/t5/Support-Questions/Monitor-alert-long-running-Airflow-jobs/m-p/388314/highlight/true#M246627

Did it not help? If not, Please try the following:

Airflow has the following metrics that can be used:
[0b]https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/logging-monitoring/metrics.html#metric-descriptions

For the integration, I'd suggest you explore the feasibility based on your use-case. For example, the Metrics configuration setup is given here for StatD and OpenTelemetry.
[0c] https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/logging-monitoring/metrics.html#metrics-configuration

Also, I am citing the third-party apps that may be helpful:
[0d] https://github.com/search?q=airflow+prometheus&type=repositories

Let us know how it goes.

Re: Capture airflow run duration

smdas — Tue, 01 Apr 2025 14:15:13 GMT

Hello @wert_1311

Greetings from Cloudera Support. Based on the Post, Your Team wish to capture the Airflow's DAG Run over the past 3 Months.

The Info is Captured in the Airflow mySQL DB (For Public Cloud). You can connect to the Airflow mySQL DB [1] & Capture the Info from the Metadata Tables for DAG Runs. Note that Caution is recommended while accessing the Metadata Tables to avoid any Undue Changes, which may break the CDE Setup.

Regards, Smarak

[1] https://community.cloudera.com/t5/Customer/How-to-connect-to-CDE-AirFlow-database-In-CDP-Public-Cloud/ta-p/324309

Re: Capture airflow run duration

smdas — Tue, 08 Apr 2025 13:38:17 GMT

Hello @wert_1311

We hope the above Post answers your Q on the Use-Case of fetching the DAG Run over the past 3 months via Airflow DB. We shall mark the Post as Solved, yet Feel free to Update the Post if there's any further Q.

Regards, Smarak