Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Spark logs location

Solved Go to solution

Spark logs location

Hi,

We are running spark jobs and knew that YARN will create logs on hdfs at /app-logs/<running User>/logs/application_1463538185607_99971

To know more details about logs we can run yarn logs -applicationId application_1463538185607_99971

But we are working on Spark Automation process and trying to keep the logs in Custom location. In-order to achieve this we added "log4j.appender.rolling.file" property in "Custom spark-log4j-properties" section through Ambari.

log4j.appender.rolling.file= ${spark.yarn.app.container.log.dir}/spark.log

Here I'm not sure where Spark is going to create logs for sucessfull/Failed jobs.

Can you suggest me where can we check this spark logs?

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Spark logs location

I have added below properties in advanced log4j properties and spark is creating logs in local directory.

log4j.appender.rolling=org.apache.log4j.RollingFileAppender

log4j.appender.rolling.encoding=UTF-8

log4j.appender.rolling.layout=org.apache.log4j.PatternLayout

log4j.appender.rolling.layout.conversionPattern=[%d] %p %m (%c)%n

log4j.appender.rolling.maxBackupIndex=5

log4j.appender.rolling.maxFileSize=50MB

log4j.logger.org.apache.spark=WARN

log4j.logger.org.eclipse.jetty=WARN log4j.rootLogger=INFO, rolling #log4j.appender.rolling.file=${spark.yarn.app.container.log.dir}/spark.log

log4j.appender.rolling.file=/var/log/spark/spark.log

${spark.yarn.app.container.log.dir}/spark.log doesn't work for me to write logs in HDFS.

4 REPLIES 4

Re: Spark logs location

@Divakar Annapureddy

I doubt log4j will work with hdfs.

Try setting the file location on native linux path, something like /var/log/spark/spark.log

Re: Spark logs location

Thanks for this..I tried it earlier but it's not creating any logs here. I'm seeing only .OUT files.

Re: Spark logs location

I looked at "yarn.nodemanager.log-dirs" in YARN but it seems YARN will clear all the logs immediately after completion of the job.

Re: Spark logs location

I have added below properties in advanced log4j properties and spark is creating logs in local directory.

log4j.appender.rolling=org.apache.log4j.RollingFileAppender

log4j.appender.rolling.encoding=UTF-8

log4j.appender.rolling.layout=org.apache.log4j.PatternLayout

log4j.appender.rolling.layout.conversionPattern=[%d] %p %m (%c)%n

log4j.appender.rolling.maxBackupIndex=5

log4j.appender.rolling.maxFileSize=50MB

log4j.logger.org.apache.spark=WARN

log4j.logger.org.eclipse.jetty=WARN log4j.rootLogger=INFO, rolling #log4j.appender.rolling.file=${spark.yarn.app.container.log.dir}/spark.log

log4j.appender.rolling.file=/var/log/spark/spark.log

${spark.yarn.app.container.log.dir}/spark.log doesn't work for me to write logs in HDFS.