How to Obtain total_io_mb, mapreduce_inputBytes, and mapreduce_outputBytes for Each Application in Yarn Logs?

[#1]

Hello everyone,

I am Emmanuel Katto currently working on evaluating the disk I/O of our CDH (Cloudera Distribution for Hadoop) cluster, which consists of several hundred bare metal machines. I would like to obtain the following values for each application within a certain period of time:

• total_io_mb
• mapreduce_inputBytes
• mapreduce_outputBytes

These values, I believe, are logged in the YARN logs, but I’m not sure how to configure YARN or the logging system to ensure these values are written in the log files.

So far, through Cloudera Manager, we’ve only been able to get metrics like the yarn_application_hdfs_bytes_read_rate, but that’s not enough for evaluating overall disk I/O.

Could anyone share any advice or alternatives on how to extract these specific I/O values for each application? Also, if there’s a way to configure YARN or Cloudera Manager to write these metrics into the logs, I’d appreciate your insights.

Thanks in advance!

Regards

Emmanuel Katto

[#2]

@emmanuelkatto24, Welcome to our community! To help you get the best possible answer, I have tagged our Airflow experts @smdas who may be able to assist you further.

Please feel free to provide any additional information or details about your query, and we hope that you will find a satisfactory solution to your question.