Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Seeing consider setting spark.io.compression.codec=lzf

avatar
Expert Contributor

Guys,

When I am trying to start Spark History Server component, I see an error IllegalArgumentException not sure why it is popping up!

lease use the new key 'spark.yarn.am.waitTime' instead.
16/06/08 15:04:32 ERROR FsHistoryProvider: Exception encountered when attempting to load application log hdfs://hdfsfederation/spark-history/.a5555e556-3301-433e-44de-23311665ed
java.lang.IllegalArgumentException: Codec [a5555e556-3301-433e-44de-23311665ed] is not available. Consider setting spark.io.compression.codec=lzf
        at org.apache.spark.io.CompressionCodec$$anonfun$createCodec$1.apply(CompressionCodec.scala:77)
        at org.apache.spark.io.CompressionCodec$$anonfun$createCodec$1.apply(CompressionCodec.scala:77)
        at scala.Option.getOrElse(Option.scala:120)
        at org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:77)

I also tried setting spark.io.compression.codec = lzf ( This error didn't go) Also tried setting spark.io.compression.codec = org.apache.spark.io.LZFCompressionCodec . This also does not fix the error.

Any one faced this situation?

This is on HDP 2.4, Spark 1.6.0

1 ACCEPTED SOLUTION

avatar
Super Guru
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login
3 REPLIES 3

avatar
Super Guru
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login

avatar
Super Guru

Hi @Smart Solutions Did you tried the suggested way?

avatar
Expert Contributor

HI @Jitendra Yadav , I restarted couple of times and it worked smoothly so did not do anything else.

Thank you 🙂