spark history server event log question

[#1]

Spark job is writing  event log to  hdfs://namenode:8021/spark-history , but our job creates many events within  10  to 12 minutes.

As per the document spark.eventLog.buffer.kb  =100kb

Does it mean when event writes to /spark-history/application_xxxxxx_xx/xx  ,  whenever buffer gets 100kb?  It means every time it is going to call fsync?

spark.eventLog.buffer.kb

100k

Buffer size to use when writing to output streams, in KiB unless otherwise specified.

[#2]

Hello @mokkan, 

Yes, that buffer means that it will wait until it reaches the 100kb and will write it to the location under HDFS. 
If you need to get the events before, you can decrease the buffer size. 

Not all the times the buffer is reached it will perform the fsync, 
According to what I found, Spark uses OutputStream.flush(), so not all the times an fsync() will be performed. 

Regards, 
Andrés Fallas
Cloudera Customer Operations Engineer