Support Questions
Find answers, ask questions, and share your expertise

How can we write the intermediate outputs from a Spark Streaming job to Web UI?

New Contributor

I am using a Spark Streaming job and processing batches of data at an interval of 1 min. I am actually getting input data from flume and then use that input data and retrieve some more data from HBase tables and after processing updating back to HBase.In the program I would need some intermediate outputs like how much time it took to write to Hbase and how many items are processed etc,.At present we are tracking these by using sendToFlume and checking the HDFS files under Flume.Is there any Web interface available to route these kind of data generated inside the Spark program instead of using sendToFlume?