Monitoring Streaming application Flume->Kafka->Storm->Hive Also some Oozie jobs

[#1]

I have a question about monitoring an application that reads files from one datasource ( oozie ) and streaming logs through a streaming application Logs -> Flume -> Kafka -> Storm -> HDFS.

I would like to monitor this application and apart from Email actions in the Oozie workflow and Ambari monitoring errors in the setup I was wondering if anybody had done something like that before.

1) Storm Topology/Broker/Flume agent failures

Is there any way to add ambari alerts or an ambari view that shows this in one?

2) Data problems

For example if data stops flowing from the source

Some things I have tried:

Push number of received, inserted tuples from Storm into ambari metrics and show it on Ambari. Anybody did something like that? Are custom charts in Ambari supported now?

Any simpler solutions.

[#2 · Accepted solution]

Write custome MBeans to derive custom metrics (like you mentioned) in storm and monitor them from JConsole or use tools Ganglia. https://dropwizard.github.io/metrics/3.1.0/manual...

Kafka is JMX compliant, so enable JMX and monitor the same way as above.

Please note that these metrics cannot be shown on ambari or ambari views. You may use Ganglia UI.

For alerts you can use http://code.google.com/p/ganglia-alert/

All the Best:)

[#3]

@Benjamin Leonhardi I'm trolling you