The application might be expecting the log folder to be there in order to generate logs in it. Seems like your problem can be solved by creating the folder in the driver node: /some/path/to/edgeNode/   I hope you also know that you have mentioned the log4j file only for driver program. In order for executors to generate logs you may need to specify the following option in spark-submit "spark.executor.extraJavaOptions=-Dlog4j.configuration=driver_log4j.properties" ... View more