question Re: HDP 2,3.4- Running jobs is not getting displayed in Spark UI in Archives of Support Questions (Read Only)

HDP 2,3.4- Running jobs is not getting displayed in Spark UI

DivyaGehlot13 — Mon, 19 Aug 2019 08:14:50 GMT

Hi,

I am running HDP-2.3.4 cluster on EC2.When I go to http://ec2-xx-xx-xxx-xxx.ap-southeast-1.compute.amazonaws.com:4041/jobs/ ,it doesnt display any currently running jobs.

Whereas in spark-shell ,it shows the spark UI started at 4041

16/02/14 21:45:13 INFO AbstractConnector: Started SelectChannelConnector@0.0.0.0:4041
16/02/14 21:45:13 INFO Utils: Successfully started service 'SparkUI' on port 4041.
16/02/14 21:45:13 INFO SparkUI: Started SparkUI at http://xxx.xx.xx.xxx:4041

But if I go to history server at It shows all the old jobs ran in the past .

I am new bee to Spark.

P.S. Attaching the screen shots of Spark Web UI and History UI

Thanks,

Divya

Re: HDP 2,3.4- Running jobs is not getting displayed in Spark UI

nsabharwal — Mon, 15 Feb 2016 11:01:39 GMT

@Divya Gehlot

You may be running the job local mode. Specify master to get the job listing.

http://spark.apache.org/docs/latest/running-on-yarn.html

Re: HDP 2,3.4- Running jobs is not getting displayed in Spark UI

aervits — Mon, 15 Feb 2016 11:11:39 GMT

@Divya Gehlot please follow the steps in this document http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.4/bk_spark-guide/content/ch_tuning-spark.html#spark-job-status

Re: HDP 2,3.4- Running jobs is not getting displayed in Spark UI

DivyaGehlot13 — Mon, 15 Feb 2016 12:24:23 GMT

@Neeraj Sabharwal :

I am running my spark below :

spark-shell --properties-file /TestDivya/Spark/Oracle.properties --jars /usr/hdp/2.3.4.0-3485/spark/lib/ojdbc6.jar --driver-class-path /usr/hdp/2.3.4.0-3485/spark/lib/ojdbc6.jar --packages com.databricks:spark-csv_2.10:1.1.0 --master yarn-client -i /TestDivya/Spark/Test.scala

Do I need to modify my options to view jobs in Spark UI ?

Re: HDP 2,3.4- Running jobs is not getting displayed in Spark UI

nsabharwal — Mon, 15 Feb 2016 12:27:12 GMT

@Divya Gehlot I guess you are using ambari. You can see Spark and quick link...Go to spark history server to get details. Your job looks ok. If that does not work then replace --master yarn-client with --master yarn

Re: HDP 2,3.4- Running jobs is not getting displayed in Spark UI

DivyaGehlot13 — Mon, 19 Aug 2019 08:14:42 GMT

@Neeraj Sabharwal :I am able to view jobs in Spark history server but unable to view current running jobs in Spark Web UI even I though I am specifying --master yarn-client or --master yarn

When the job finishes I could see the "

This webpage is not available

ERR_CONNECTION_REFUSED "

And When job running it displays the page but doesnt show any running jobs .

Am I missing any settings ?

Re: HDP 2,3.4- Running jobs is not getting displayed in Spark UI

nsabharwal — Mon, 15 Feb 2016 20:31:40 GMT

@Divya Gehlot

Hi Divya, Please see this thread https://community.hortonworks.com/questions/11382/hdp-234-spark-web-ui.html

You can access it only through the lifetime of the job.

Re: HDP 2,3.4- Running jobs is not getting displayed in Spark UI

nsabharwal — Tue, 16 Feb 2016 09:52:33 GMT

@Guilherme Braccialli

Re: HDP 2,3.4- Running jobs is not getting displayed in Spark UI

gbraccialli3 — Tue, 16 Feb 2016 10:08:38 GMT

@Divya Gehlot

Spark History only shows your job after it's finished.

To access Spark UI while your job is running on YARN, got to YARN UI (usually port :8088) and click on Application Master link of your job.

Re: HDP 2,3.4- Running jobs is not getting displayed in Spark UI

DivyaGehlot13 — Thu, 18 Feb 2016 16:43:21 GMT

As SparkUI port was internal ip of Ec2 ,that's the reason I wasn't able to view the current running jobs in Spark UI.

To resolve this issue ,had to configure SSH Tunnel.

Re: HDP 2,3.4- Running jobs is not getting displayed in Spark UI

jtasipit — Fri, 24 Mar 2017 08:58:17 GMT

Hope this will help anyone:

Add or modify below properties

Ambari > Spark > Configs > Custom spark-defaults

spark.eventLog.enabled = true

spark.history.fs.logDirectory = hdfs://<fs.defaultFS>/<directory-to-your-spark-logs>

Ambari > Spark > Configs > Advanced spark-defaults

spark.history.provider = org.apache.spark.deploy.history.FsHistoryProvider