I have installed a cluster with spark server and spark client nodes
When trying to access the spark server webi UI
Last updated: 2019-11-21 12:50:57
Client local time zone: Europe/Berlin
Did you specify the correct logging directory? Please verify your setting of spark.history.fs.logDirectory listed above and whether you have the permissions to access it.
It is also possible that your application did not run to completion or did not stop the SparkContext.
One of the reason might be that the "hdfs://spark2-history" directory size has grown too much due to some old applications data might not be cleared.
# su - hdfs -c "hdfs dfs -du -s -h hdfs:///spark2-history/" # su - hdfs -c "hdfs dfs -ls hdfs:///spark2-history/" | wc -l # su - hdfs -c "hdfs dfs -ls hdfs:///spark2-history/"
Do you see too many inprogress logs?
# su - hdfs -c "hdfs dfs -ls hdfs:///spark2-history/" | grep 'inprogress'
Do you see any repeated warning / errors in spark logs?
What is the value set for the following properties set in your spark configs?
spark.history.fs.cleaner.enabled spark.history.fs.cleaner.interval spark.history.fs.cleaner.maxAge
Fro more details on these properties please refer to spark documentation: https://spark.apache.org/docs/latest/monitoring.html