Logging Spark Jobs with log4j

[#1]

Hi!

We need to log Spark jobs with log4j, saving the output into an external logfile in HDFS (we want to separate the application log from the native YARN logs).

We do not have Apache Flume and Apache Kafka on our cluster. (HDP 3) so we are looking for HDFS appender to use with log4j.

Any suggestions?

Thanks in advance.

R.A.