Spark logs location

[#1]

Hi,

We are running spark jobs and knew that YARN will create logs on hdfs at /app-logs/<running User>/logs/application_1463538185607_99971

To know more details about logs we can run yarn logs -applicationId application_1463538185607_99971

But we are working on Spark Automation process and trying to keep the logs in Custom location. In-order to achieve this we added "log4j.appender.rolling.file" property in "Custom spark-log4j-properties" section through Ambari.

log4j.appender.rolling.file= ${spark.yarn.app.container.log.dir}/spark.log

Here I'm not sure where Spark is going to create logs for sucessfull/Failed jobs.

Can you suggest me where can we check this spark logs?

[#2]

@Divakar Annapureddy

I doubt log4j will work with hdfs.

Try setting the file location on native linux path, something like /var/log/spark/spark.log

[#3]

Thanks for this..I tried it earlier but it's not creating any logs here. I'm seeing only .OUT files.

[#4]

I looked at "yarn.nodemanager.log-dirs" in YARN but it seems YARN will clear all the logs immediately after completion of the job.

[#5 · Accepted solution]

I have added below properties in advanced log4j properties and spark is creating logs in local directory.

log4j.appender.rolling=org.apache.log4j.RollingFileAppender

log4j.appender.rolling.encoding=UTF-8

log4j.appender.rolling.layout=org.apache.log4j.PatternLayout

log4j.appender.rolling.layout.conversionPattern=[%d] %p %m (%c)%n

log4j.appender.rolling.maxBackupIndex=5

log4j.appender.rolling.maxFileSize=50MB

log4j.logger.org.apache.spark=WARN

log4j.logger.org.eclipse.jetty=WARN log4j.rootLogger=INFO, rolling #log4j.appender.rolling.file=${spark.yarn.app.container.log.dir}/spark.log

log4j.appender.rolling.file=/var/log/spark/spark.log

${spark.yarn.app.container.log.dir}/spark.log doesn't work for me to write logs in HDFS.