Hi Jitendra, You are right, it works this way! The relevant lines in my spark-submit run script are: --files external/log4j.properties \ --conf "spark.executor.extraJavaOptions='-Dlog4j.configuration=log4j.properties'" \ --driver-java-options "-Dlog4j.configuration=file:/absolute/path/to/your/project/externall/log4j.properties" \ The result is still confusing because yarn keeps pushing INFO messages even if your project's rootlogger is set to WARN. But this setup does allow to set the logging level for your own project's packages. Thanks for pushing me into this final step! Marc ... View more

Hi Jitendra, I believe I did, but that I stranded in the fact the -Dlog4j.configuration needs an absolute path, while you can merely provide the log4j,properties with --files and provide a relative path as ./log4j.properties on the executor. Best wishes, Marc ... View more