I tried editing the /etc/spark/conf/log4j.properties file and changed all log levels to WARN. Still when viewing application logs from YARN resource manager I see INFO messages.
Sowen, kindly inform where in cloudera manager can we set this configuration permenantly since it gets replaced every time we redeploy client configuration.
Yes, you should not edit these files manually. Have a look at the YARN configuration in CM. You should quickly find the Logs tab for the various components like ResourceManager. You can configure log levels there.
I opened yarn configuration, searched for logging threshold and changed them all to WARN. Still when I deploy client configuration the log4j.properties does not change
Yes, after loggin in to Cloudera Manager, I clicked on YARN service, then clicked on the Configuration tab. In the search field I entered logging threshold and changed them all to WARN (JobHistory Server Logging Threshold, NodeManager Logging Threshold, NodeManager Logging Threshold,ResourceManager Logging Threshold)
Is there a way in Cloudera Manager where we can set the log level specified by /etc/hadoop/conf/log4j.properties?