Collecting Storm Metrics with Collectd

[#1]

Abstract:
Nimbus metrics are critical to operations as well as development teams for monitoring the performance and stability of Storm applications / topology. Usually most production environments have a metrics / operations monitoring systems including solr, elasticsearch, tsdbs etc. This post shows you; how you can use Collectd to forward these metrics over to your desired metrics environment and alert on them.

Solution:

Collectd is a standard metrics collection tool that can be run natively on linux operating systems. It's capable of capturing a wide variety of metrics, you can find more information on Collectd here: https://collectd.org/

So to capture Storm nimbus metrics, here's a collectd plugin that needs to be complied and built: https://github.com/srotya/storm-collectd (using Maven). Simply run:
mvn clean package assembly:single

In addition, you will need to install collectd and ensure that it has Java plugin capability. Here's a great post on how to do that:
http://blog.asquareb.com/blog/2014/06/09/enabling-java-plugin-for-collectd/ (Please note that the JAR="/path/to/jar" JAVAC="/path/to/javac" variables need to be fixed before you can run it)

Once installed, you will need to configure collectd using the following: (DON'T FORGET TO CONFIGURE OUTPUT PLUGIN)

LoadPlugin java

<Plugin "java">  
# required JVM argument is the classpath  
# JVMArg "-Djava.class.path=/installpath/collectd/share/collectd/java" 
# Since version 4.8.4 (commit c983405) the API and GenericJMX plugin are 
# provided as .jar files.  

JVMARG "-Djava.class.path=<ABSOLUTE PATH>/lib/collectd-api.jar:<ABSOLUTE PATH>/target/storm-collectd-0.0.1-SNAPSHOT-jar-with-dependencies.jar"  






LoadPlugin "com.srotya.collectd.storm.StormNimbusMetrics"

<Plugin "storm">
	address "http://localhost:8084/"
	kerberos false	
	jaas "<PATH TO JAAS CONF>"  
</Plugin>

</Plugin>