question Re: spark history server event log question in Support Questions

spark history server event log question

mokkan — Wed, 21 May 2025 18:18:07 GMT

Spark job is writing event log to hdfs://namenode:8021/spark-history , but our job creates many events within 10 to 12 minutes.

As per the document spark.eventLog.buffer.kb =100kb

Does it mean when event writes to /spark-history/application_xxxxxx_xx/xx , whenever buffer gets 100kb? It means every time it is going to call fsync?

spark.eventLog.buffer.kb

100k

Buffer size to use when writing to output streams, in KiB unless otherwise specified.

Re: spark history server event log question

vafs — Fri, 23 May 2025 23:38:29 GMT

Hello @mokkan,

Yes, that buffer means that it will wait until it reaches the 100kb and will write it to the location under HDFS.
If you need to get the events before, you can decrease the buffer size.

Not all the times the buffer is reached it will perform the fsync,
According to what I found, Spark uses OutputStream.flush(), so not all the times an fsync() will be performed.

Regards,
Andrés Fallas
Cloudera Customer Operations Engineer