I guess, one solution is to launch in spark job in cluster mode. spark.yarn.submit.waitAppCompletion (default: true) is a flag to control whether to wait for the application to finish before exiting the launcher process in cluster mode Example spark-submit --class SparkBatchLogs --conf "spark.yarn.submit.waitAppCompletion=false" --master yarn-cluster SparkTest-0.0.1-SNAPSHOT.jar ... View more