Access Running Job Metrics via JMX

[#1]

Can you access Job Metrics via JMX? I would like to see all running jobs and how long they have been running, how many containers were allocated,etc. It looks like the ResourceManager REST API provides what I'm looking for, but was wondering if it was available in JMX.

[#2 · Accepted solution]

@mhendricksT

Try this: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_hdfs_admin_tools/content/ch07.html

[#3]

@mhendricks

I have the exact same interest on monitoring all the containers of the job via JMX. However, my question is how to get the list of containers for a job assuming we can turn ON JMX for yarn containers?

[#4]

https://community.hortonworks.com/questions/6488/yarn-jmx-access.html

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.2/bk_hdfs_admin_tools/content/ch07.html

http://www.slideshare.net/Hadoop_Summit/w-525hall1shenv2

[#5]

@Timothy Spann

Tim,

Thanks for the info. I can pull daemon JMX for sure since they are running all the time. However, I am more interested on per job based container. How to query YARN restful api to get a list of container for a job?

[#6]

seems like everything is here:

https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_...

http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.2/bk_yarn_resource_mgt/content/ch_yarn_rest_ap...

https://hadoop.apache.org/docs/r2.7.2/hadoop-yarn/hadoop-yarn-site/YarnCommands.html#container

command line

yarn application -list

get your list of job ids

yarn container -list application_1475786720159_0003

[#7]

My problem is how to set different JMX ports on the containers running on the same slave node.

[#8]

Have you looked at this article?

https://community.hortonworks.com/content/kbentry/61188/enable-jmx-metrics-on-hadoop-using-jmxterm.h...