I have created Spark Streaming application which consumes message from Kafka topic. After started the Spark Streaming application in YARN Cluster mode the Spark UI has to show the Streaming Tab to understand the metrics but it is not available.
Is there any configuration available to enable Streaming tab.
I can see the tab in my local machine Spark 2.0 version.
Since you're running in cluster mode, your node may not be the driver. So it won't come at Spark UI of that but of some other node which yarn assigns as the driver.
You can get this info through logs or resource manager UI.
1. Spark UI is available only during the lifetime of a SparkContext at <drivernode>:4040 by default. If you exit the context, it's gone.
2. The port number increments if you have multiple SparkContexts running. Suppose you created another spark context and ran some streaming job there, it would be accessible via <drivernode>:4041 now.