New Contributor
Posts: 5
Registered: ‎11-02-2018

Config log4j in Spark - Driver Logs



I am trying to use the custom log4j to gather Spark driver logs( submitting jobs under CLUSTER mode), but unable to achieve it.

Here is my custom file content:


#Below is the unix server path from where job is getting submitted

log4j.appender.FILE.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n


And cmnd to submit job:

spark2-submit --files /apps/test/config/ --conf "" --master yarn --deploy-mode cluster --num-executors 2 --executor-cores 4 --driver-memory 1g --executor-memory 16g --keytab XXXXX.keytab --principal XXXXX --class com.test.spark.par_1_submit par_submit.jar


Error I'm getting: /some/path/to/edgeNode/SparkDriver.log (No such file or directory)
Cloudera Employee
Posts: 35
Registered: ‎04-05-2016

Re: Config log4j in Spark - Driver Logs

As your intent seems to capture the driver logs in a separate file while executing the app in the cluster mode, make sure that '/some/path/to/edgeNode/' dir is present on all of the NodeManager essentially as in cluster mode the driver will be running in the Yarn app's application master.

If you can't make sure that follow a general practice to provide log file path to some pre-existing paths e.g. "/var/log/SparkDriver.log".