Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Logging level in Spark

Re: Logging level in Spark

Cloudera Employee

Hi Youssef,

 

The easiest way to change the log level for Spark on Yarn applications is to copy the existing log4j.properties file in /etc/spark/conf and change the log level to WARN (log4j.rootCategory=WARN, console). After doing this you can start the spark-shell (or use spark-submit) with:

 

spark-shell --master yarn --files /path/to/new/log4j.properties

 

The log4j.properties file will be used by Yarn containers and you can verify the result with $yarn logs -applicationId application_<id> from the command line.

 

Configuring this from CM can be done by following the below steps:

 

1) Ensure that all nodes have a Spark Gateway role defined (Spark -> Instances -> Add role instance)

2) In Spark Client Advanced Configuration Snippet (Safety Valve) for spark-conf/log4j.properties set:

log4j.rootCategory=WARN, console

3) In Spark Client Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-defaults.conf spark.executor.extraJavaOptions=-Dlog4j.configuration=file:///etc/spark/conf/log4j.properties

4) Save configurations and deploy client configs

 

For further information see the "Debugging your application" section in f.e:

http://spark.apache.org/docs/1.2.0/running-on-yarn.html

 

Re: Logging level in Spark

New Contributor

I know it's an old thread. However, I will update it in case someone else comes in here looking for the same thing like I did. This worked for me on 5.7.2.

 

1) Set Cloudera Manager >> Spark >> Configuration >> Gateway Logging Threshold >> WARN.

2) Deploy Client Configuration.

3) Restart the jobs.