Logging level in Spark

[#1]

How can i change the logging level of spark to WARN from INFO in spark-shell?

[#2]

I tried editing the /etc/spark/conf/log4j.properties file and changed all log levels to WARN. Still when viewing application logs from YARN resource manager I see INFO messages.

[#3]

Are you trying to change logging messages from YARN, or from Spark? this file will not affect what YARN logs.

[#4]

turns out that we have to change the /etc/hadoop/conf/log4j.properties file

[#5]

Sowen, kindly inform where in cloudera manager can we set this configuration permenantly since it gets replaced every time we redeploy client configuration.

[#6]

Yes, you should not edit these files manually. Have a look at the YARN configuration in CM. You should quickly find the Logs tab for the various components like ResourceManager. You can configure log levels there.

[#7]

I opened yarn configuration, searched for logging threshold and changed them all to WARN. Still when I deploy client configuration the log4j.properties does not change

[#8]

YARN config? did you restart the YARN services?

[#9]

Yes, after loggin in to Cloudera Manager, I clicked on YARN service, then clicked on the Configuration tab. In the search field I entered logging threshold and changed them all to WARN (JobHistory Server Logging Threshold, NodeManager Logging Threshold, NodeManager Logging Threshold,ResourceManager Logging Threshold)

[#10]

Hello Sowen,

Is there a way in Cloudera Manager where we can set the log level specified by /etc/hadoop/conf/log4j.properties?

Thank you,