Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Spark2 History Server UI is not showing completed application when spark.eventLog.compress=true

Solved Go to solution

Spark2 History Server UI is not showing completed application when spark.eventLog.compress=true

Explorer

Spark2 History Server UI is not showing completed application when spark.eventLog.compress=true. I could see completed application in spark eventlog dir in lz4 format.

/spark2-history/application_1574201808558_1228909.lz4
Ambari Version : 2.6.0.0
HDP Version : 2.6.3.0-235
Spark Version : 2.x

Thank you.

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Spark2 History Server UI is not showing completed application when spark.eventLog.compress=true

Explorer
2 REPLIES 2
Highlighted

Re: Spark2 History Server UI is not showing completed application when spark.eventLog.compress=true

Explorer

History Server Log

ERROR FsHistoryProvider: Exception encountered when attempting to load application log hdfs:///spark2-history/application_1574201808558_1228909.lz4
java.io.EOFException: Stream ended prematurely
        at org.apache.spark.io.LZ4BlockInputStream.readFully(LZ4BlockInputStream.java:230)
        at org.apache.spark.io.LZ4BlockInputStream.refill(LZ4BlockInputStream.java:203)
        at org.apache.spark.io.LZ4BlockInputStream.read(LZ4BlockInputStream.java:125)
        at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
        at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
        at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
        at java.io.InputStreamReader.read(InputStreamReader.java:184)
        at java.io.BufferedReader.fill(BufferedReader.java:161)
        at java.io.BufferedReader.readLine(BufferedReader.java:324)
        at java.io.BufferedReader.readLine(BufferedReader.java:389)
        at scala.io.BufferedSource$BufferedLineIterator.hasNext(BufferedSource.scala:72)
        at scala.collection.Iterator$$anon$21.hasNext(Iterator.scala:836)
        at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:461)
        at org.apache.spark.scheduler.ReplayListenerBus.replay(ReplayListenerBus.scala:78)
        at org.apache.spark.scheduler.ReplayListenerBus.replay(ReplayListenerBus.scala:58)

 

Highlighted

Re: Spark2 History Server UI is not showing completed application when spark.eventLog.compress=true

Explorer
Don't have an account?
Coming from Hortonworks? Activate your account here