Created 11-24-2014 12:57 AM
How can i change the logging level of spark to WARN from INFO in spark-shell?
Created 02-23-2015 07:08 AM
I tried editing the /etc/spark/conf/log4j.properties file and changed all log levels to WARN. Still when viewing application logs from YARN resource manager I see INFO messages.
Created 02-23-2015 07:10 AM
Are you trying to change logging messages from YARN, or from Spark? this file will not affect what YARN logs.
Created 02-23-2015 07:11 AM
turns out that we have to change the /etc/hadoop/conf/log4j.properties file
Created 02-23-2015 07:21 AM
Sowen, kindly inform where in cloudera manager can we set this configuration permenantly since it gets replaced every time we redeploy client configuration.
Created 02-23-2015 07:37 AM
Yes, you should not edit these files manually. Have a look at the YARN configuration in CM. You should quickly find the Logs tab for the various components like ResourceManager. You can configure log levels there.
Created 02-23-2015 08:07 AM
I opened yarn configuration, searched for logging threshold and changed them all to WARN. Still when I deploy client configuration the log4j.properties does not change
Created 02-23-2015 08:22 AM
YARN config? did you restart the YARN services?
Created 02-24-2015 02:44 AM
Yes, after loggin in to Cloudera Manager, I clicked on YARN service, then clicked on the Configuration tab. In the search field I entered logging threshold and changed them all to WARN (JobHistory Server Logging Threshold, NodeManager Logging Threshold, NodeManager Logging Threshold,ResourceManager Logging Threshold)
Created 02-25-2015 03:23 AM
Hello Sowen,
Is there a way in Cloudera Manager where we can set the log level specified by /etc/hadoop/conf/log4j.properties?
Thank you,