@Bharath If the job is still running then you can go to YARN-Resource manager UI(usually 8088 port) and click on applicationmaster link,This will redirect to spark job UI and you can get the detailed information how many tasks are running/finished/failed with time took for each task to finish.
-
If you want to debug the job that is finished/failed then go to spark history server(usually runs on 18080 port), then you are able to see time taken for each task to execute and DAG of the spark job also.
-
Starting from Spark2(scala only) we can use spark.time to calculate the time took for the action to execute.
-
You can also execute explain() plan to get detailed information of the execution of spark job.
Useful links: link1 link2 link3