Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Spark logs location

avatar

Hi,

We are running spark jobs and knew that YARN will create logs on hdfs at /app-logs/<running User>/logs/application_1463538185607_99971

To know more details about logs we can run yarn logs -applicationId application_1463538185607_99971

But we are working on Spark Automation process and trying to keep the logs in Custom location. In-order to achieve this we added "log4j.appender.rolling.file" property in "Custom spark-log4j-properties" section through Ambari.

log4j.appender.rolling.file= ${spark.yarn.app.container.log.dir}/spark.log

Here I'm not sure where Spark is going to create logs for sucessfull/Failed jobs.

Can you suggest me where can we check this spark logs?

1 ACCEPTED SOLUTION

avatar

I have added below properties in advanced log4j properties and spark is creating logs in local directory.

log4j.appender.rolling=org.apache.log4j.RollingFileAppender

log4j.appender.rolling.encoding=UTF-8

log4j.appender.rolling.layout=org.apache.log4j.PatternLayout

log4j.appender.rolling.layout.conversionPattern=[%d] %p %m (%c)%n

log4j.appender.rolling.maxBackupIndex=5

log4j.appender.rolling.maxFileSize=50MB

log4j.logger.org.apache.spark=WARN

log4j.logger.org.eclipse.jetty=WARN log4j.rootLogger=INFO, rolling #log4j.appender.rolling.file=${spark.yarn.app.container.log.dir}/spark.log

log4j.appender.rolling.file=/var/log/spark/spark.log

${spark.yarn.app.container.log.dir}/spark.log doesn't work for me to write logs in HDFS.

View solution in original post

4 REPLIES 4

avatar

@Divakar Annapureddy

I doubt log4j will work with hdfs.

Try setting the file location on native linux path, something like /var/log/spark/spark.log

avatar

Thanks for this..I tried it earlier but it's not creating any logs here. I'm seeing only .OUT files.

avatar

I looked at "yarn.nodemanager.log-dirs" in YARN but it seems YARN will clear all the logs immediately after completion of the job.

avatar

I have added below properties in advanced log4j properties and spark is creating logs in local directory.

log4j.appender.rolling=org.apache.log4j.RollingFileAppender

log4j.appender.rolling.encoding=UTF-8

log4j.appender.rolling.layout=org.apache.log4j.PatternLayout

log4j.appender.rolling.layout.conversionPattern=[%d] %p %m (%c)%n

log4j.appender.rolling.maxBackupIndex=5

log4j.appender.rolling.maxFileSize=50MB

log4j.logger.org.apache.spark=WARN

log4j.logger.org.eclipse.jetty=WARN log4j.rootLogger=INFO, rolling #log4j.appender.rolling.file=${spark.yarn.app.container.log.dir}/spark.log

log4j.appender.rolling.file=/var/log/spark/spark.log

${spark.yarn.app.container.log.dir}/spark.log doesn't work for me to write logs in HDFS.