question Re: Seeing consider setting spark.io.compression.codec=lzf in Support Questions

Seeing consider setting spark.io.compression.codec=lzf

smartninja723 — Wed, 08 Jun 2016 22:15:05 GMT

Guys,

When I am trying to start Spark History Server component, I see an error IllegalArgumentException not sure why it is popping up!

lease use the new key 'spark.yarn.am.waitTime' instead.
16/06/08 15:04:32 ERROR FsHistoryProvider: Exception encountered when attempting to load application log hdfs://hdfsfederation/spark-history/.a5555e556-3301-433e-44de-23311665ed
java.lang.IllegalArgumentException: Codec [a5555e556-3301-433e-44de-23311665ed] is not available. Consider setting spark.io.compression.codec=lzf
        at org.apache.spark.io.CompressionCodec$$anonfun$createCodec$1.apply(CompressionCodec.scala:77)
        at org.apache.spark.io.CompressionCodec$$anonfun$createCodec$1.apply(CompressionCodec.scala:77)
        at scala.Option.getOrElse(Option.scala:120)
        at org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:77)

I also tried setting spark.io.compression.codec = lzf ( This error didn't go) Also tried setting spark.io.compression.codec = org.apache.spark.io.LZFCompressionCodec . This also does not fix the error.

Any one faced this situation?

This is on HDP 2.4, Spark 1.6.0

Re: Seeing consider setting spark.io.compression.codec=lzf

jyadav — Wed, 08 Jun 2016 23:38:09 GMT

@Smart Solutions

I have seen application_* files inside /spark-history directory but don't know from where you got ".a5555e556-3301-433e-44de-23311665ed. Can you check the content of this file/dir? Also what if you move this file/dir from other location and restart the spark history server?

Re: Seeing consider setting spark.io.compression.codec=lzf

jyadav — Thu, 09 Jun 2016 17:32:28 GMT

Hi @Smart Solutions Did you tried the suggested way?

Re: Seeing consider setting spark.io.compression.codec=lzf

smartninja723 — Thu, 09 Jun 2016 23:33:59 GMT

HI @Jitendra Yadav , I restarted couple of times and it worked smoothly so did not do anything else.

Thank you 🙂