Support Questions
Find answers, ask questions, and share your expertise
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Logging level in Spark

Re: Logging level in Spark

Cloudera Employee

Hi Youssef,


The easiest way to change the log level for Spark on Yarn applications is to copy the existing file in /etc/spark/conf and change the log level to WARN (log4j.rootCategory=WARN, console). After doing this you can start the spark-shell (or use spark-submit) with:


spark-shell --master yarn --files /path/to/new/


The file will be used by Yarn containers and you can verify the result with $yarn logs -applicationId application_<id> from the command line.


Configuring this from CM can be done by following the below steps:


1) Ensure that all nodes have a Spark Gateway role defined (Spark -> Instances -> Add role instance)

2) In Spark Client Advanced Configuration Snippet (Safety Valve) for spark-conf/ set:

log4j.rootCategory=WARN, console

3) In Spark Client Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-defaults.conf spark.executor.extraJavaOptions=-Dlog4j.configuration=file:///etc/spark/conf/

4) Save configurations and deploy client configs


For further information see the "Debugging your application" section in f.e:


Re: Logging level in Spark

New Contributor

I know it's an old thread. However, I will update it in case someone else comes in here looking for the same thing like I did. This worked for me on 5.7.2.


1) Set Cloudera Manager >> Spark >> Configuration >> Gateway Logging Threshold >> WARN.

2) Deploy Client Configuration.

3) Restart the jobs.