Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

HDP 2.6: Spark 2.1 Streaming tab not available in the Spark UI

Highlighted

HDP 2.6: Spark 2.1 Streaming tab not available in the Spark UI

New Contributor

I have created Spark Streaming application which consumes message from Kafka topic. After started the Spark Streaming application in YARN Cluster mode the Spark UI has to show the Streaming Tab to understand the metrics but it is not available.

16681-spark-streaming-tab-missing.png

Is there any configuration available to enable Streaming tab.

I can see the tab in my local machine Spark 2.0 version.

Thanks,

Uvaraj.S

2 REPLIES 2

Re: HDP 2.6: Spark 2.1 Streaming tab not available in the Spark UI

Expert Contributor

Since you're running in cluster mode, your node may not be the driver. So it won't come at Spark UI of that but of some other node which yarn assigns as the driver.

For example you may be accessing Spark UI via http://localnode:4040, but in cluster mode it'll now be http://clusternode:4040. (because sc and driver are created there).

You can get this info through logs or resource manager UI.

Also:

1. Spark UI is available only during the lifetime of a SparkContext at <drivernode>:4040 by default. If you exit the context, it's gone.

2. The port number increments if you have multiple SparkContexts running. Suppose you created another spark context and ran some streaming job there, it would be accessible via <drivernode>:4041 now.

Thanks

Re: HDP 2.6: Spark 2.1 Streaming tab not available in the Spark UI

Explorer

@tsharma . I am also getting the same issue, I am trying the monitor the incomplete application from the history server. My history server is running in 18080 port. And I am running using following http://<myserver>-node:18080/?showIncomplete=true . But missing the Streaming tab.

94443-screen-shot-2018-12-05-at-85936-pm.png

Don't have an account?
Coming from Hortonworks? Activate your account here