HDP 2,3.4- Running jobs is not getting displayed in Spark UI

[#1]

Hi,

I am running HDP-2.3.4 cluster on EC2.When I go to http://ec2-xx-xx-xxx-xxx.ap-southeast-1.compute.amazonaws.com:4041/jobs/ ,it doesnt display any currently running jobs.

Whereas in spark-shell ,it shows the spark UI started at 4041

16/02/14 21:45:13 INFO AbstractConnector: Started SelectChannelConnector@0.0.0.0:4041
16/02/14 21:45:13 INFO Utils: Successfully started service 'SparkUI' on port 4041.
16/02/14 21:45:13 INFO SparkUI: Started SparkUI at http://xxx.xx.xx.xxx:4041

But if I go to history server at It shows all the old jobs ran in the past .

I am new bee to Spark.

P.S. Attaching the screen shots of Spark Web UI and History UI

Thanks,

Divya

[#2]

Hope this will help anyone:

Add or modify below properties

Ambari > Spark > Configs > Custom spark-defaults

spark.eventLog.enabled = true

spark.history.fs.logDirectory = hdfs://<fs.defaultFS>/<directory-to-your-spark-logs>

Ambari > Spark > Configs > Advanced spark-defaults

spark.history.provider = org.apache.spark.deploy.history.FsHistoryProvider