Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

HDP 2,3.4- Running jobs is not getting displayed in Spark UI

avatar
Expert Contributor

Hi,

I am running HDP-2.3.4 cluster on EC2.When I go to http://ec2-xx-xx-xxx-xxx.ap-southeast-1.compute.amazonaws.com:4041/jobs/ ,it doesnt display any currently running jobs.

Whereas in spark-shell ,it shows the spark UI started at 4041

16/02/14 21:45:13 INFO AbstractConnector: Started SelectChannelConnector@0.0.0.0:4041
16/02/14 21:45:13 INFO Utils: Successfully started service 'SparkUI' on port 4041.
16/02/14 21:45:13 INFO SparkUI: Started SparkUI at http://xxx.xx.xx.xxx:4041

But if I go to history server at It shows all the old jobs ran in the past .

I am new bee to Spark.

P.S. Attaching the screen shots of Spark Web UI and History UI

2127-spark-history-server.png

Thanks,

Divya


nosparkjobsrunning.png
1 ACCEPTED SOLUTION

avatar
@Divya Gehlot

Spark History only shows your job after it's finished.

To access Spark UI while your job is running on YARN, got to YARN UI (usually port :8088) and click on Application Master link of your job.

View solution in original post

10 REPLIES 10

avatar
Master Mentor
@Divya Gehlot

You may be running the job local mode. Specify master to get the job listing.

http://spark.apache.org/docs/latest/running-on-yarn.html

avatar
Expert Contributor

@Neeraj Sabharwal :

I am running my spark below :

spark-shell --properties-file /TestDivya/Spark/Oracle.properties --jars /usr/hdp/2.3.4.0-3485/spark/lib/ojdbc6.jar --driver-class-path /usr/hdp/2.3.4.0-3485/spark/lib/ojdbc6.jar --packages com.databricks:spark-csv_2.10:1.1.0 --master yarn-client -i /TestDivya/Spark/Test.scala

Do I need to modify my options to view jobs in Spark UI ?

avatar
Master Mentor

@Divya Gehlot I guess you are using ambari. You can see Spark and quick link...Go to spark history server to get details. Your job looks ok. If that does not work then replace --master yarn-client with --master yarn

avatar
Expert Contributor

@Neeraj Sabharwal :I am able to view jobs in Spark history server but unable to view current running jobs in Spark Web UI even I though I am specifying --master yarn-client or --master yarn

When the job finishes I could see the "

This webpage is not available

ERR_CONNECTION_REFUSED "

And When job running it displays the page but doesnt show any running jobs .

Am I missing any settings ?

2137-spark-shell-spark-jobs.png

avatar
Master Mentor

@Divya Gehlot

Hi Divya, Please see this thread https://community.hortonworks.com/questions/11382/hdp-234-spark-web-ui.html

You can access it only through the lifetime of the job.

avatar
Master Mentor

avatar
Master Mentor

avatar
@Divya Gehlot

Spark History only shows your job after it's finished.

To access Spark UI while your job is running on YARN, got to YARN UI (usually port :8088) and click on Application Master link of your job.

avatar
Expert Contributor

As SparkUI port was internal ip of Ec2 ,that's the reason I wasn't able to view the current running jobs in Spark UI.

To resolve this issue ,had to configure SSH Tunnel.