Spark History UI crashing often?

[#1]

We have experienced that the Spark History UI often requires being restarted as we find the service on port 18080 is down, returns Server 500 error. Any reason this might be the case?

HDP 2.2.4.2, Spark 1.2

It was suggested to increase the YARN heap to 8G, which fostered no improvement.

Thanks!

Joshua

[#2]

@Joshua Lickteig are you experiencing This ?

[#3 · Accepted solution]

@Joshua can you increase the SPARK_DAEMON_MEMORY to 4g

SPARK_DAEMON_MEMORY

Memory to allocate to the history server (default: 1g).

http://spark.apache.org/docs/latest/monitoring.html

[#4]

Hi,

Can you share the error message from spark history server logs when the spark history UI is crashing?

Thanks

AKR