question Monitoring Streaming application Flume->Kafka->Storm->Hive Also some Oozie jobs in Archives of Support Questions (Read Only)

Monitoring Streaming application Flume->Kafka->Storm->Hive Also some Oozie jobs

bleonhardi — Tue, 08 Dec 2015 18:14:54 GMT

I have a question about monitoring an application that reads files from one datasource ( oozie ) and streaming logs through a streaming application Logs -> Flume -> Kafka -> Storm -> HDFS.

I would like to monitor this application and apart from Email actions in the Oozie workflow and Ambari monitoring errors in the setup I was wondering if anybody had done something like that before.

1) Storm Topology/Broker/Flume agent failures

Is there any way to add ambari alerts or an ambari view that shows this in one?

2) Data problems

For example if data stops flowing from the source

Some things I have tried:

Push number of received, inserted tuples from Storm into ambari metrics and show it on Ambari. Anybody did something like that? Are custom charts in Ambari supported now?

Any simpler solutions.

Re: Monitoring Streaming application Flume->Kafka->Storm->Hive Also some Oozie jobs

vt0084883 — Thu, 10 Dec 2015 17:18:53 GMT

Write custome MBeans to derive custom metrics (like you mentioned) in storm and monitor them from JConsole or use tools Ganglia. https://dropwizard.github.io/metrics/3.1.0/manual...

Kafka is JMX compliant, so enable JMX and monitor the same way as above.

Please note that these metrics cannot be shown on ambari or ambari views. You may use Ganglia UI.

For alerts you can use http://code.google.com/p/ganglia-alert/

All the Best:)

Re: Monitoring Streaming application Flume->Kafka->Storm->Hive Also some Oozie jobs

aervits — Tue, 02 Feb 2016 23:02:56 GMT

@Benjamin Leonhardi I'm trolling you