How to get total_io_mb of eatch applications in yarn log

Question

I would like to know how to obtain the values of total_io_mb, mapreduce_inputBytes, and mapreduce_outputBytes for each application within a certain period of time.
I believe that these values are written to the yarn log file. So, I would like to know SME's advice on how to do this, that is, how to configure these values to be written to the log.

By Cloudera Manager, we are not able to get but the rate(ex. yarn_application_hdfs_bytes_read_rate). However, alternative ideas are welcome.

The purpose is to evaluate the disk IO of the entire CDH cluster, which is made up of several hundred bare metal machines, and to estimate the disk IO required for the next generation of bare metal and/or storage nodes.

AyazHussain · Accepted Answer

Hi @yoshio_ono ,Please check this article https://my.cloudera.com/knowledge/How-to-calculate-memory-and-v-core-utilization-for-each-of-the?id=271149.

DianaTorres · Answer

@yoshio_ono Welcome to the Cloudera Community!

To help you get the best possible solution, I have tagged our YARN experts @AKR @Xiaomin @srathore who may be able to assist you further.

Please keep us updated on your post, and we hope you find a satisfactory solution to your query.

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.