Community Articles
Find and share helpful community-sourced technical articles
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.
Super Guru

To start pulling jmx metrics from hadoop you need to first enable it via jvm parameters. Go to ambari-->Yarn-->config-->yarn-env

Enable jmx by adding the following params to YARN_RESOURCEMANAGER_OPTS

"-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=8001"

Set the port to whatever port is available on your cluster

8473-2016-10-12-10-25-38.jpg

Save the new config. This will require restart of yarn.

Now you have jmx enabled but you need to client to start pulling jmx metrics. Go to your data node and and download (from here http://wiki.cyclopsgroup.org/jmxterm/download.html) the latest jxmterm-xxx-xxx-uber.jar. For this article I use jmxterm-1.0-alpha-4-uber.jar

Once you have the jmxterm client downloaded lets connect to jxm using

java -jar jmxterm-1.0-alpha-4-uber.jar -l service:jmx:rmi:///jndi/rmi://localhost:<YOURPORT>/jmxrmi

In this example I am set port to 8012.

java -jar jmxterm-1.0-alpha-4-uber.jar -l service:jmx:rmi:///jndi/rmi://localhost:8012/jmxrmi

8474-2016-10-12-10-32-44.jpg

Now I am connected to jxm. Lets look at all the beans available by issuing beans commands

8475-3.jpg

Well now I see all the beans available to pull metrics from. Lets say I want to pull metrics about resourcemanager cluster metrics. That is bean Hadoop:name=ClusterMetrics,service=ResourceManager. Lets find all the attributes available for that beans by issuing info -b Hadoop:name=ClusterMetrics,service=ResourceManager

8476-4.jpg

All the attributes are shown and notice there is a notification attribute. You can use this for notification into your enterprise operational system. To pull metrics for a specific attribute within a bean use

get -b bean_name attribute

For this example I want to know number of active Node Managers which is attribute NumActiveNMs.

get -b Hadoop:name=ClusterMetrics,service=ResourceManager NumActiveNMs

8477-5.jpg

So there are 4 active node managers. I want to know how many node managers are down. That is attribute NumLostNMs

get -b Hadoop:name=ClusterMetrics,service=ResourceManager NumLostNMs

8478-6.jpg

Returns 0 meaning all my node managers are available.

Hope this helps you get started on interacting with JMX.

3,099 Views
Comments
Contributor

Great article. Is there any way to pull individual yarn container JMX? It would be great for me to see what is going on when I start a job and monitor the job containers at real time.

Super Guru

@ScipioTheYounger most definitely. Simply use exact same process and connect to yarn jvm. you will get all attributes (metrics) available.

Contributor

Sunile.

For daemons such as NM and RM, I can hard-code JMS port number. However, how to set different JMX port number on a number of JVM containers running on the same slave node? For example, I can configure 18 container per node, how could I set the port number to 8001-8018? Maybe I can use ZK, or some kind of script.

Don't have an account?
Coming from Hortonworks? Activate your account here
Version history
Revision #:
2 of 2
Last update:
‎08-17-2019 08:53 AM
Updated by:
 
Contributors
Top Kudoed Authors