Support Questions
Find answers, ask questions, and share your expertise

Logging Spark Jobs with log4j

Logging Spark Jobs with log4j

New Contributor


We need to log Spark jobs with log4j, saving the output into an external logfile in HDFS (we want to separate the application log from the native YARN logs).

We do not have Apache Flume and Apache Kafka on our cluster. (HDP 3) so we are looking for HDFS appender to use with log4j.

Any suggestions?

Thanks in advance.