The easiest way to change the log level for Spark on Yarn applications is to copy the existing log4j.properties file in /etc/spark/conf and change the log level to WARN (log4j.rootCategory=WARN, console). After doing this you can start the spark-shell (or use spark-submit) with:
spark-shell --master yarn --files /path/to/new/log4j.properties
The log4j.properties file will be used by Yarn containers and you can verify the result with $yarn logs -applicationId application_<id> from the command line.
Configuring this from CM can be done by following the below steps:
1) Ensure that all nodes have a Spark Gateway role defined (Spark -> Instances -> Add role instance)
2) In Spark Client Advanced Configuration Snippet (Safety Valve) for spark-conf/log4j.properties set:
3) In Spark Client Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-defaults.conf spark.executor.extraJavaOptions=-Dlog4j.configuration=file:///etc/spark/conf/log4j.properties
4) Save configurations and deploy client configs
For further information see the "Debugging your application" section in f.e:
I know it's an old thread. However, I will update it in case someone else comes in here looking for the same thing like I did. This worked for me on 5.7.2.
1) Set Cloudera Manager >> Spark >> Configuration >> Gateway Logging Threshold >> WARN.
2) Deploy Client Configuration.
3) Restart the jobs.