Spark2 History Server UI is not showing completed application when spark.eventLog.compress=true

[#1]

Spark2 History Server UI is not showing completed application when spark.eventLog.compress=true. I could see completed application in spark eventlog dir in lz4 format.

/spark2-history/application_1574201808558_1228909.lz4

Ambari Version : 2.6.0.0
HDP Version : 2.6.3.0-235
Spark Version : 2.x

Thank you.

[#2]

History Server Log

ERROR FsHistoryProvider: Exception encountered when attempting to load application log hdfs:///spark2-history/application_1574201808558_1228909.lz4
java.io.EOFException: Stream ended prematurely
        at org.apache.spark.io.LZ4BlockInputStream.readFully(LZ4BlockInputStream.java:230)
        at org.apache.spark.io.LZ4BlockInputStream.refill(LZ4BlockInputStream.java:203)
        at org.apache.spark.io.LZ4BlockInputStream.read(LZ4BlockInputStream.java:125)
        at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
        at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
        at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
        at java.io.InputStreamReader.read(InputStreamReader.java:184)
        at java.io.BufferedReader.fill(BufferedReader.java:161)
        at java.io.BufferedReader.readLine(BufferedReader.java:324)
        at java.io.BufferedReader.readLine(BufferedReader.java:389)
        at scala.io.BufferedSource$BufferedLineIterator.hasNext(BufferedSource.scala:72)
        at scala.collection.Iterator$$anon$21.hasNext(Iterator.scala:836)
        at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:461)
        at org.apache.spark.scheduler.ReplayListenerBus.replay(ReplayListenerBus.scala:78)
        at org.apache.spark.scheduler.ReplayListenerBus.replay(ReplayListenerBus.scala:58)

[#3 · Accepted solution]

This seems to be related to https://issues.apache.org/jira/browse/SPARK-21447