HDP 2.6: Spark 2.1 Streaming tab not available in the Spark UI

[#1]

I have created Spark Streaming application which consumes message from Kafka topic. After started the Spark Streaming application in YARN Cluster mode the Spark UI has to show the Streaming Tab to understand the metrics but it is not available.

Is there any configuration available to enable Streaming tab.

I can see the tab in my local machine Spark 2.0 version.

Thanks,

Uvaraj.S

[#2]

Since you're running in cluster mode, your node may not be the driver. So it won't come at Spark UI of that but of some other node which yarn assigns as the driver.

For example you may be accessing Spark UI via http://localnode:4040, but in cluster mode it'll now be http://clusternode:4040. (because sc and driver are created there).

You can get this info through logs or resource manager UI.

Also:

1. Spark UI is available only during the lifetime of a SparkContext at <drivernode>:4040 by default. If you exit the context, it's gone.

2. The port number increments if you have multiple SparkContexts running. Suppose you created another spark context and ran some streaming job there, it would be accessible via <drivernode>:4041 now.

Thanks

[#3]

@tsharma . I am also getting the same issue, I am trying the monitor the incomplete application from the history server. My history server is running in 18080 port. And I am running using following http://<myserver>-node:18080/?showIncomplete=true . But missing the Streaming tab.