Reply
umb
Explorer
Posts: 8
Registered: ‎11-07-2014

Logging level in Spark

How can i change the logging level of spark to WARN from INFO in spark-shell?

New Contributor
Posts: 17
Registered: ‎02-22-2015

Re: Logging level in Spark

I tried editing the /etc/spark/conf/log4j.properties file and changed all log levels to WARN. Still when viewing application logs from YARN resource manager I see INFO messages.

Cloudera Employee
Posts: 481
Registered: ‎08-11-2014

Re: Logging level in Spark

Are you trying to change logging messages from YARN, or from Spark? this file will not affect what YARN logs.

New Contributor
Posts: 17
Registered: ‎02-22-2015

Re: Logging level in Spark

turns out that we have to change the /etc/hadoop/conf/log4j.properties file

New Contributor
Posts: 17
Registered: ‎02-22-2015

Re: Logging level in Spark

Sowen, kindly inform where in cloudera manager can we set this configuration permenantly since it gets replaced every time we redeploy client configuration.

Cloudera Employee
Posts: 481
Registered: ‎08-11-2014

Re: Logging level in Spark

Yes, you should not edit these files manually. Have a look at the YARN configuration in CM. You should quickly find the Logs tab for the various components like ResourceManager. You can configure log levels there.

New Contributor
Posts: 17
Registered: ‎02-22-2015

Re: Logging level in Spark

I opened yarn configuration, searched for logging threshold and changed them all to WARN. Still when I deploy client configuration the log4j.properties does not change

Cloudera Employee
Posts: 481
Registered: ‎08-11-2014

Re: Logging level in Spark

YARN config? did you restart the YARN services?

New Contributor
Posts: 17
Registered: ‎02-22-2015

Re: Logging level in Spark

Yes, after loggin in to Cloudera Manager, I clicked on YARN service, then clicked on the Configuration tab. In the search field I entered logging threshold and changed them all to WARN (JobHistory Server Logging Threshold, NodeManager Logging Threshold, NodeManager Logging Threshold,ResourceManager Logging Threshold)

New Contributor
Posts: 17
Registered: ‎02-22-2015

Re: Logging level in Spark

Hello Sowen,

 

Is there a way in Cloudera Manager where we can set the log level specified by /etc/hadoop/conf/log4j.properties?

 

Thank you,