Support Questions
Find answers, ask questions, and share your expertise

Spark2 History Server UI is not showing completed application when spark.eventLog.compress=true

Solved Go to solution
Highlighted

Spark2 History Server UI is not showing completed application when spark.eventLog.compress=true

Explorer

Spark2 History Server UI is not showing completed application when spark.eventLog.compress=true. I could see completed application in spark eventlog dir in lz4 format.

/spark2-history/application_1574201808558_1228909.lz4
Ambari Version : 2.6.0.0
HDP Version : 2.6.3.0-235
Spark Version : 2.x

Thank you.

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Spark2 History Server UI is not showing completed application when spark.eventLog.compress=true

Explorer
2 REPLIES 2

Re: Spark2 History Server UI is not showing completed application when spark.eventLog.compress=true

Explorer

History Server Log

ERROR FsHistoryProvider: Exception encountered when attempting to load application log hdfs:///spark2-history/application_1574201808558_1228909.lz4
java.io.EOFException: Stream ended prematurely
        at org.apache.spark.io.LZ4BlockInputStream.readFully(LZ4BlockInputStream.java:230)
        at org.apache.spark.io.LZ4BlockInputStream.refill(LZ4BlockInputStream.java:203)
        at org.apache.spark.io.LZ4BlockInputStream.read(LZ4BlockInputStream.java:125)
        at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
        at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
        at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
        at java.io.InputStreamReader.read(InputStreamReader.java:184)
        at java.io.BufferedReader.fill(BufferedReader.java:161)
        at java.io.BufferedReader.readLine(BufferedReader.java:324)
        at java.io.BufferedReader.readLine(BufferedReader.java:389)
        at scala.io.BufferedSource$BufferedLineIterator.hasNext(BufferedSource.scala:72)
        at scala.collection.Iterator$$anon$21.hasNext(Iterator.scala:836)
        at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:461)
        at org.apache.spark.scheduler.ReplayListenerBus.replay(ReplayListenerBus.scala:78)
        at org.apache.spark.scheduler.ReplayListenerBus.replay(ReplayListenerBus.scala:58)

 

Highlighted

Re: Spark2 History Server UI is not showing completed application when spark.eventLog.compress=true

Explorer
Don't have an account?