Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Logging level in Spark

Highlighted

Logging level in Spark

Explorer

How can i change the logging level of spark to WARN from INFO in spark-shell?

11 REPLIES 11

Re: Logging level in Spark

New Contributor

I tried editing the /etc/spark/conf/log4j.properties file and changed all log levels to WARN. Still when viewing application logs from YARN resource manager I see INFO messages.

Re: Logging level in Spark

Master Collaborator

Are you trying to change logging messages from YARN, or from Spark? this file will not affect what YARN logs.

Re: Logging level in Spark

New Contributor

turns out that we have to change the /etc/hadoop/conf/log4j.properties file

Re: Logging level in Spark

New Contributor

Sowen, kindly inform where in cloudera manager can we set this configuration permenantly since it gets replaced every time we redeploy client configuration.

Re: Logging level in Spark

Master Collaborator

Yes, you should not edit these files manually. Have a look at the YARN configuration in CM. You should quickly find the Logs tab for the various components like ResourceManager. You can configure log levels there.

Re: Logging level in Spark

New Contributor

I opened yarn configuration, searched for logging threshold and changed them all to WARN. Still when I deploy client configuration the log4j.properties does not change

Re: Logging level in Spark

Master Collaborator

YARN config? did you restart the YARN services?

Re: Logging level in Spark

New Contributor

Yes, after loggin in to Cloudera Manager, I clicked on YARN service, then clicked on the Configuration tab. In the search field I entered logging threshold and changed them all to WARN (JobHistory Server Logging Threshold, NodeManager Logging Threshold, NodeManager Logging Threshold,ResourceManager Logging Threshold)

Re: Logging level in Spark

New Contributor

Hello Sowen,

 

Is there a way in Cloudera Manager where we can set the log level specified by /etc/hadoop/conf/log4j.properties?

 

Thank you,