Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Spark Streaming Monitoring options to monitor input data rate in last 24 hours?

Spark Streaming Monitoring options to monitor input data rate in last 24 hours?

Contributor

Hi All,

Just looking for various open source options available to monitor the input data rate to show some metrics on the input data for last 24 hours etc. Also will be using this monitoring tools for other options as well to purely monitor everything related t spark streaming? Any ideas in terms of the tools i know we currently have Spark Streaming UI, which will show the number of rdds processed but currently i just wanted to know the input data rate before these are processed?

Any ideas in terms of the open source tools to use will be appreciated?

1. Ganglia?

2. Push pull receivers?

3. Spark logs i.e. as soon as the messages are received shall we log them and connect a UI tool to it.

Please let me know if there are any more efficient options. Thank you

Don't have an account?
Coming from Hortonworks? Activate your account here