question Re: How to get total_io_mb of eatch applications in yarn log in Support Questions

How to get total_io_mb of eatch applications in yarn log

yoshio_ono — Thu, 17 Oct 2024 11:58:33 GMT

I would like to know how to obtain the values of total_io_mb, mapreduce_inputBytes, and mapreduce_outputBytes for each application within a certain period of time.
I believe that these values are written to the yarn log file. So, I would like to know SME's advice on how to do this, that is, how to configure these values to be written to the log.

By Cloudera Manager, we are not able to get but the rate(ex. yarn_application_hdfs_bytes_read_rate). However, alternative ideas are welcome.

The purpose is to evaluate the disk IO of the entire CDH cluster, which is made up of several hundred bare metal machines, and to estimate the disk IO required for the next generation of bare metal and/or storage nodes.

Re: How to get total_io_mb of eatch applications in yarn log

DianaTorres — Fri, 18 Oct 2024 03:49:32 GMT

@yoshio_ono Welcome to the Cloudera Community!

To help you get the best possible solution, I have tagged our YARN experts @AKR @Xiaomin @srathore who may be able to assist you further.

Please keep us updated on your post, and we hope you find a satisfactory solution to your query.

Re: How to get total_io_mb of eatch applications in yarn log

AyazHussain — Wed, 30 Oct 2024 06:53:40 GMT

Hi @yoshio_ono ,

Please check this article https://my.cloudera.com/knowledge/How-to-calculate-memory-and-v-core-utilization-for-each-of-the?id=271149.

Re: How to get total_io_mb of eatch applications in yarn log

yoshio_ono — Thu, 31 Oct 2024 03:52:44 GMT

Thanks. I guess it is not for total_io_mb but for "memory and v-core utilization".
Disk io should be in yarn logs. I will try seek it.