Our storm topology has custom metrics. I want to publish them into AMS. To do this, I understand I need a metric source to push the metrics to Hadoop Metrics2 framework and metrics sink to push the metrics from Metrics2 to AMS. Is this a valid approach? If so, what class should I use as the Metric Sink and how should I configure it?
For the metric source, I integrated the following metrics2 source for publishing my metrics into Hadoop Metrics2.
I see a variety of sinks are available but not sure which performs the push to ambari. What should be used? What is the ambari-metrics-storm-sink? I
hbase.class=org.apache.hadoop.metrics2.sink.timeline.HadoopTimelineMetricsSink hbase.period=10 hbase.collector=localhost.localdomain:6188
In my case here, localhost.localdomain:6188 is the location where the Metrics Collector is running (you would need to change this obviously). You should be able to do something like the following,
mystormmetrics.class=org.apache.hadoop.metrics2.sink.timeline.HadoopTimelineMetricsSink mystormmetrics.period=10 mystormmetrics.collector=<metrics-collector-fqdn>:6188
The important part is to make sure you match the "mystormmetrics" in the filename and the entries in the file. You can also use an asterisk ("*") to send all metrics2 information to that sink. Be aware that Hadoop Metrics2 will also pick up any file that matches "hadoop-metrics2-*.properties" that are available on the classpath. I'm not sure if Storm will already have one being picked up somewhere.
Finally, then you can configure and start the Reporter once in your Storm application to start pushing any metrics you write via the dropwizard API to AMS (see the README on the github project). To play around with it, you can also try configuring the FileSink to test the setup before pushing to AMS -- it's quicker to verify that metrics are being sent correctly.
I suppose it also goes without saying, but this definitely needs to be documented better on my little project :)
You have been an enormous help. Thanks!
Here's what I see. Maybe a version difference or storm-ism? There are 2 items. After setting logging to trace, I see:
The *TimelineMetricsSink clases do not extend MetricsPlugin and we get a class cast exception
"cannot be cast to org.apache.hadoop.metrics2.MetricsPlugin"
I'll keep looking.
Hrmm. What version of Hadoop/Storm are you using? If you can share a trivial example, I can try to pull it down myself and poke around with it.
You have to look at the implementation of org.apache.hadoop.metrics2.sink.storm.StormTimelineMetricsReporter. This class is the one AMS uses to push Storm metrics to the Metrics Collector.
HadoopTimelineMetricsSink is an implementation that uses Hadoop Metrics2 framework, and is not supported by storm.
The Storm Sink jar is linked to the Storm lib directory through a symlink.
ls -al /usr/hdp/current/storm-client/lib/ambari-metrics-storm-sink.jar
lrwxrwxrwx. 1 root root 72 Apr 29 21:20 /usr/hdp/current/storm-client/lib/ambari-metrics-storm-sink.jar -> /usr/lib/storm/lib/ambari-metrics-storm-sink-with-common-220.127.116.11.459.jar
Interesting. I didn't realize that there are so many projects with an extra layer of metrics indirection to get data to AMS (storm, kafka, and flume I can see). Thanks for sharing, Aravindan.