Support Questions
Find answers, ask questions, and share your expertise

Method to publish custom topology metrics to Ambari Metrics

Highlighted

Method to publish custom topology metrics to Ambari Metrics

New Contributor

Our storm topology has custom metrics. I want to publish them into AMS. To do this, I understand I need a metric source to push the metrics to Hadoop Metrics2 framework and metrics sink to push the metrics from Metrics2 to AMS. Is this a valid approach? If so, what class should I use as the Metric Sink and how should I configure it?

Notes:

Metrics Source:

For the metric source, I integrated the following metrics2 source for publishing my metrics into Hadoop Metrics2.

https://github.com/joshelser/dropwizard-hadoop-metrics2

Metrics Sink:

I see a variety of sinks are available but not sure which performs the push to ambari. What should be used? What is the ambari-metrics-storm-sink? I

7 REPLIES 7
Highlighted

Re: Method to publish custom topology metrics to Ambari Metrics

I took a look at a local HBase installation by Ambari (as I know HBase is configured to send metrics to AMS), and I saw the following in hadoop-metrics2-hbase.properties:
hbase.class=org.apache.hadoop.metrics2.sink.timeline.HadoopTimelineMetricsSink
hbase.period=10
hbase.collector=localhost.localdomain:6188

In my case here, localhost.localdomain:6188 is the location where the Metrics Collector is running (you would need to change this obviously). You should be able to do something like the following,

In hadoop-metrics2-mystormmetrics.properties:

mystormmetrics.class=org.apache.hadoop.metrics2.sink.timeline.HadoopTimelineMetricsSink
mystormmetrics.period=10
mystormmetrics.collector=<metrics-collector-fqdn>:6188

The important part is to make sure you match the "mystormmetrics" in the filename and the entries in the file. You can also use an asterisk ("*") to send all metrics2 information to that sink. Be aware that Hadoop Metrics2 will also pick up any file that matches "hadoop-metrics2-*.properties" that are available on the classpath. I'm not sure if Storm will already have one being picked up somewhere.

Finally, then you can configure and start the Reporter once in your Storm application to start pushing any metrics you write via the dropwizard API to AMS (see the README on the github project). To play around with it, you can also try configuring the FileSink to test the setup before pushing to AMS -- it's quicker to verify that metrics are being sent correctly.

Highlighted

Re: Method to publish custom topology metrics to Ambari Metrics

I suppose it also goes without saying, but this definitely needs to be documented better on my little project :)

Highlighted

Re: Method to publish custom topology metrics to Ambari Metrics

New Contributor

You have been an enormous help. Thanks!

Here's what I see. Maybe a version difference or storm-ism? There are 2 items. After setting logging to trace, I see:

Property name:

  • With a prop of this form (and the corresponding prop filename), the configuration does identify the prop/value but doesn't do anything with it. mystormmetrics.class=
  • Using this form, it does attempt to install the sink but there is the next problem. mystormmetrics.sink.inst.class=

Class type:

The *TimelineMetricsSink clases do not extend MetricsPlugin and we get a class cast exception

"cannot be cast to org.apache.hadoop.metrics2.MetricsPlugin"

I'll keep looking.

Highlighted

Re: Method to publish custom topology metrics to Ambari Metrics

Hrmm. What version of Hadoop/Storm are you using? If you can share a trivial example, I can try to pull it down myself and poke around with it.

Highlighted

Re: Method to publish custom topology metrics to Ambari Metrics

Expert Contributor

Hi John,

You have to look at the implementation of org.apache.hadoop.metrics2.sink.storm.StormTimelineMetricsReporter. This class is the one AMS uses to push Storm metrics to the Metrics Collector.

HadoopTimelineMetricsSink is an implementation that uses Hadoop Metrics2 framework, and is not supported by storm.

The Storm Sink jar is linked to the Storm lib directory through a symlink.

ls -al /usr/hdp/current/storm-client/lib/ambari-metrics-storm-sink.jar

lrwxrwxrwx. 1 root root 72 Apr 29 21:20 /usr/hdp/current/storm-client/lib/ambari-metrics-storm-sink.jar -> /usr/lib/storm/lib/ambari-metrics-storm-sink-with-common-2.2.2.0.459.jar

Highlighted

Re: Method to publish custom topology metrics to Ambari Metrics

Interesting. I didn't realize that there are so many projects with an extra layer of metrics indirection to get data to AMS (storm, kafka, and flume I can see). Thanks for sharing, Aravindan.

Highlighted

Re: Method to publish custom topology metrics to Ambari Metrics

New Contributor

Does this mean I should extend StormTimelineMetricsReporter with my own reportMetrics() method?