Seeing consider setting spark.io.compression.codec=lzf

[#1]

Guys,

When I am trying to start Spark History Server component, I see an error IllegalArgumentException not sure why it is popping up!

lease use the new key 'spark.yarn.am.waitTime' instead.
16/06/08 15:04:32 ERROR FsHistoryProvider: Exception encountered when attempting to load application log hdfs://hdfsfederation/spark-history/.a5555e556-3301-433e-44de-23311665ed
java.lang.IllegalArgumentException: Codec [a5555e556-3301-433e-44de-23311665ed] is not available. Consider setting spark.io.compression.codec=lzf
        at org.apache.spark.io.CompressionCodec$$anonfun$createCodec$1.apply(CompressionCodec.scala:77)
        at org.apache.spark.io.CompressionCodec$$anonfun$createCodec$1.apply(CompressionCodec.scala:77)
        at scala.Option.getOrElse(Option.scala:120)
        at org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:77)

I also tried setting spark.io.compression.codec = lzf ( This error didn't go) Also tried setting spark.io.compression.codec = org.apache.spark.io.LZFCompressionCodec . This also does not fix the error.

Any one faced this situation?

This is on HDP 2.4, Spark 1.6.0

[#2 · Accepted solution]

@Smart Solutions

I have seen application_* files inside /spark-history directory but don't know from where you got ".a5555e556-3301-433e-44de-23311665ed. Can you check the content of this file/dir? Also what if you move this file/dir from other location and restart the spark history server?

[#3]

Hi @Smart Solutions Did you tried the suggested way?

[#4]

HI @Jitendra Yadav , I restarted couple of times and it worked smoothly so did not do anything else.

Thank you 🙂