HDP 2,3.4- Running jobs is not getting displayed in Spark UI

[#1]

Hi,

I am running HDP-2.3.4 cluster on EC2.When I go to http://ec2-xx-xx-xxx-xxx.ap-southeast-1.compute.amazonaws.com:4041/jobs/ ,it doesnt display any currently running jobs.

Whereas in spark-shell ,it shows the spark UI started at 4041

16/02/14 21:45:13 INFO AbstractConnector: Started SelectChannelConnector@0.0.0.0:4041
16/02/14 21:45:13 INFO Utils: Successfully started service 'SparkUI' on port 4041.
16/02/14 21:45:13 INFO SparkUI: Started SparkUI at http://xxx.xx.xx.xxx:4041

But if I go to history server at It shows all the old jobs ran in the past .

I am new bee to Spark.

P.S. Attaching the screen shots of Spark Web UI and History UI

Thanks,

Divya

[#2]

@Divya Gehlot

You may be running the job local mode. Specify master to get the job listing.

http://spark.apache.org/docs/latest/running-on-yarn.html

[#3]

@Neeraj Sabharwal :

I am running my spark below :

spark-shell --properties-file /TestDivya/Spark/Oracle.properties --jars /usr/hdp/2.3.4.0-3485/spark/lib/ojdbc6.jar --driver-class-path /usr/hdp/2.3.4.0-3485/spark/lib/ojdbc6.jar --packages com.databricks:spark-csv_2.10:1.1.0 --master yarn-client -i /TestDivya/Spark/Test.scala

Do I need to modify my options to view jobs in Spark UI ?

[#4]

@Divya Gehlot I guess you are using ambari. You can see Spark and quick link...Go to spark history server to get details. Your job looks ok. If that does not work then replace --master yarn-client with --master yarn

[#5]

@Neeraj Sabharwal :I am able to view jobs in Spark history server but unable to view current running jobs in Spark Web UI even I though I am specifying --master yarn-client or --master yarn

When the job finishes I could see the "

This webpage is not available

ERR_CONNECTION_REFUSED "

And When job running it displays the page but doesnt show any running jobs .

Am I missing any settings ?

[#6]

@Divya Gehlot

Hi Divya, Please see this thread https://community.hortonworks.com/questions/11382/hdp-234-spark-web-ui.html

You can access it only through the lifetime of the job.

[#7]

@Guilherme Braccialli

[#8]

@Divya Gehlot please follow the steps in this document http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.4/bk_spark-guide/content/ch_tuning-spark.html#...

[#9 · Accepted solution]

@Divya Gehlot

Spark History only shows your job after it's finished.

To access Spark UI while your job is running on YARN, got to YARN UI (usually port :8088) and click on Application Master link of your job.

[#10]

As SparkUI port was internal ip of Ec2 ,that's the reason I wasn't able to view the current running jobs in Spark UI.

To resolve this issue ,had to configure SSH Tunnel.