Reply
Cloudera Employee
Posts: 26
Registered: ‎07-20-2014

Re: Logging level in Spark

Hi Youssef,

 

The easiest way to change the log level for Spark on Yarn applications is to copy the existing log4j.properties file in /etc/spark/conf and change the log level to WARN (log4j.rootCategory=WARN, console). After doing this you can start the spark-shell (or use spark-submit) with:

 

spark-shell --master yarn --files /path/to/new/log4j.properties

 

The log4j.properties file will be used by Yarn containers and you can verify the result with $yarn logs -applicationId application_<id> from the command line.

 

Configuring this from CM can be done by following the below steps:

 

1) Ensure that all nodes have a Spark Gateway role defined (Spark -> Instances -> Add role instance)

2) In Spark Client Advanced Configuration Snippet (Safety Valve) for spark-conf/log4j.properties set:

log4j.rootCategory=WARN, console

3) In Spark Client Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-defaults.conf spark.executor.extraJavaOptions=-Dlog4j.configuration=file:///etc/spark/conf/log4j.properties

4) Save configurations and deploy client configs

 

For further information see the "Debugging your application" section in f.e:

http://spark.apache.org/docs/1.2.0/running-on-yarn.html

 

Highlighted
New Contributor
Posts: 4
Registered: ‎11-04-2016

Re: Logging level in Spark

I know it's an old thread. However, I will update it in case someone else comes in here looking for the same thing like I did. This worked for me on 5.7.2.

 

1) Set Cloudera Manager >> Spark >> Configuration >> Gateway Logging Threshold >> WARN.

2) Deploy Client Configuration.

3) Restart the jobs.