Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Monitoring Streaming application Flume->Kafka->Storm->Hive Also some Oozie jobs

avatar
Master Guru

I have a question about monitoring an application that reads files from one datasource ( oozie ) and streaming logs through a streaming application Logs -> Flume -> Kafka -> Storm -> HDFS.

I would like to monitor this application and apart from Email actions in the Oozie workflow and Ambari monitoring errors in the setup I was wondering if anybody had done something like that before.

1) Storm Topology/Broker/Flume agent failures

Is there any way to add ambari alerts or an ambari view that shows this in one?

2) Data problems

For example if data stops flowing from the source

Some things I have tried:

Push number of received, inserted tuples from Storm into ambari metrics and show it on Ambari. Anybody did something like that? Are custom charts in Ambari supported now?

Any simpler solutions.

1 ACCEPTED SOLUTION

avatar
Explorer

Write custome MBeans to derive custom metrics (like you mentioned) in storm and monitor them from JConsole or use tools Ganglia. https://dropwizard.github.io/metrics/3.1.0/manual...

Kafka is JMX compliant, so enable JMX and monitor the same way as above.

Please note that these metrics cannot be shown on ambari or ambari views. You may use Ganglia UI.

For alerts you can use http://code.google.com/p/ganglia-alert/

All the Best:)

View solution in original post

2 REPLIES 2

avatar
Explorer

Write custome MBeans to derive custom metrics (like you mentioned) in storm and monitor them from JConsole or use tools Ganglia. https://dropwizard.github.io/metrics/3.1.0/manual...

Kafka is JMX compliant, so enable JMX and monitor the same way as above.

Please note that these metrics cannot be shown on ambari or ambari views. You may use Ganglia UI.

For alerts you can use http://code.google.com/p/ganglia-alert/

All the Best:)

avatar
Master Mentor

@Benjamin Leonhardi I'm trolling you