https://community.cloudera.com/t5/Support-Questions/need-to-control-spark2-history-size/m-p/242328#M204131 <P><A rel="user" href="https://community.cloudera.com/users/112829/nikhilbelure99.html" nodeid="112829">@John</A><BR /><BR /></P><P>If you see your "/spark2-history" log folder has many old orphan files then probably these are the files left over due to spark-driver failures / crashes ...etc.</P><P>You can check your Spark configs following parameters.</P><PRE>spark.history.fs.cleaner.enabled=true spark.history.fs.cleaner.interval spark.history.fs.cleaner.maxAge</PRE><P><BR /></P><P><BR /></P><P><STRONG>NOTE: </STRONG>However, there are some issues reported already for older version of Spark where the spark.history.fs.cleaner required some improvements. </P><P>As part of <A href="https://issues.apache.org/jira/browse/SPARK-8617" target="_blank">https://issues.apache.org/jira/browse/SPARK-8617</A> fix Spark 2.2 should function properly. </P><P>Also please check if the ownership of the files present inside the "/spark2-history" is correctly set or not? If not then please set it correctly according to your setup.</P><PRE># hdfs dfs -chown spark:hadoop /spark2-history </PRE><P>.</P> Thu, 20 Jun 2019 14:23:46 GMT jsensharma 2019-06-20T14:23:46Z