https://community.cloudera.com/t5/Support-Questions/Yarn-Spark-Having-a-hard-time-suppressing-Spark-INFO-logs/m-p/223565#M185430 <P> <A rel="user" href="https://community.cloudera.com/users/83321/robcornell.html" nodeid="83321">@Robert Cornell</A> </P><P>In my experience the verbosity level in spark 2 has greatly been reduced compared to 1.6 - specially in the interactive interpreters like spark-shell. </P><P>.</P><P>Please check default log4j in ambari &gt; spark2 &gt; conf and make sure the global log4j file is not setting any loggers to INFO. </P><P>.</P><P>If you wish to point to a specific log4j file, depending on the master and deployment mode you need to use one or more properties:</P><P>#yarn-client mode</P><PRE>bin/spark-submit --master yarn --deploy-mode client --files /path/to/log4j/log4j.properties --conf "spark.executor.extraJavaOptions='-Dlog4j.configuration=log4j.properties'" --driver-java-options "-Dlog4j.configuration=file:/path/to/log4j/log4j.properties" </PRE><P>#yarn-cluster mode</P><PRE>bin/spark-submit --master yarn --deploy-mode cluster --files /path/to/log4j/log4j.properties --conf "spark.executor.extraJavaOptions='-Dlog4j.configuration=log4j.properties'" --conf "spark.driver.extraJavaOptions='-Dlog4j.configuration=log4j.properties'" </PRE><P>HTH</P><P>*** If you found this answer addressed your question, please take a moment to login and click the "accept" link on the answer.</P> Sat, 16 Jun 2018 20:11:07 GMT falbani 2018-06-16T20:11:07Z