Spark2 History Server UI is not showing completed application when spark.eventLog.compress=true

AdityaShaw — Mon, 06 Jul 2020 08:06:51 GMT

Spark2 History Server UI is not showing completed application when spark.eventLog.compress=true. I could see completed application in spark eventlog dir in lz4 format.

/spark2-history/application_1574201808558_1228909.lz4

Ambari Version : 2.6.0.0 HDP Version : 2.6.3.0-235 Spark Version : 2.x

Thank you.

Re: Spark2 History Server UI is not showing completed application when spark.eventLog.compress=true

AdityaShaw — Mon, 06 Jul 2020 08:28:49 GMT

History Server Log

ERROR FsHistoryProvider: Exception encountered when attempting to load application log hdfs:///spark2-history/application_1574201808558_1228909.lz4 java.io.EOFException: Stream ended prematurely at org.apache.spark.io.LZ4BlockInputStream.readFully(LZ4BlockInputStream.java:230) at org.apache.spark.io.LZ4BlockInputStream.refill(LZ4BlockInputStream.java:203) at org.apache.spark.io.LZ4BlockInputStream.read(LZ4BlockInputStream.java:125) at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284) at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326) at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178) at java.io.InputStreamReader.read(InputStreamReader.java:184) at java.io.BufferedReader.fill(BufferedReader.java:161) at java.io.BufferedReader.readLine(BufferedReader.java:324) at java.io.BufferedReader.readLine(BufferedReader.java:389) at scala.io.BufferedSource$BufferedLineIterator.hasNext(BufferedSource.scala:72) at scala.collection.Iterator$$anon$21.hasNext(Iterator.scala:836) at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:461) at org.apache.spark.scheduler.ReplayListenerBus.replay(ReplayListenerBus.scala:78) at org.apache.spark.scheduler.ReplayListenerBus.replay(ReplayListenerBus.scala:58)

Re: Spark2 History Server UI is not showing completed application when spark.eventLog.compress=true

AdityaShaw — Mon, 06 Jul 2020 08:51:06 GMT

This seems to be related to https://issues.apache.org/jira/browse/SPARK-21447

question Re: Spark2 History Server UI is not showing completed application when spark.eventLog.compress=true in Support Questions

Spark2 History Server UI is not showing completed application when spark.eventLog.compress=true

Re: Spark2 History Server UI is not showing completed application when spark.eventLog.compress=true

Re: Spark2 History Server UI is not showing completed application when spark.eventLog.compress=true