Just looking for various open source options available to monitor the input data rate to show some metrics on the input data for last 24 hours etc. Also will be using this monitoring tools for other options as well to purely monitor everything related t spark streaming? Any ideas in terms of the tools i know we currently have Spark Streaming UI, which will show the number of rdds processed but currently i just wanted to know the input data rate before these are processed?
Any ideas in terms of the open source tools to use will be appreciated?
2. Push pull receivers?
3. Spark logs i.e. as soon as the messages are received shall we log them and connect a UI tool to it.
Please let me know if there are any more efficient options. Thank you