Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

this version of libhadoop was built without snappy support.

avatar
Rising Star

Hi,

I hope it's the right place to ask the following question 🙂

I try to put in hdfs a file with snappy compression. I write a Java code for that and when I try to run it on my cluster I got the following exception:

Exception in thread "main" java.lang.RuntimeException: native snappy library not available: this version of libhadoop was built without snappy support.

at org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:65)

at org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:134)

at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:150)

at org.apache.hadoop.io.compress.CompressionCodec$Util.createOutputStreamWithCodecPool(CompressionCodec.java:131)

at org.apache.hadoop.io.compress.SnappyCodec.createOutputStream(SnappyCodec.java:99)

Apparently the snappy library is not available... I check on the os with the following cmd "rpm -qa | less | grep snappy" and snappy and snappy-devel is present.

In the configuration of hdfs (core-site.xml) org.apache.hadoop.io.compress.SnappyCodec is present in the field io.compression.codecs.

Does anyone has a idea why it's not working?

Thanks in advance

1 ACCEPTED SOLUTION

avatar
Rising Star

The problem is solve by making the following change in the spark config:

2840-cparkconfig.jpg

Thanks for the help guys!

View solution in original post

18 REPLIES 18

avatar
Rising Star

@Neeraj Sabharwal

Thanks for the reply, In my case it's not a solution because when I'm doing

hadoop checknative -a

I see that the snappy lib is true located at / usr/hdp/2.3.4.0-3485/hadoop/lib/native/libsnappy.so.1.

avatar
New Contributor

We have the same problem.

> hadoop checknative -a

snappy: true /usr/hdp/2.3.4.0-3485/hadoop/lib/native/libsnappy.so.1

> rpm -qa snappy

snappy-1.1.0-3.el7.x86_64

What else can I check?

avatar
Master Mentor

Please confirm that you have the following property set correctly in hadoop-env.sh

avatar
Rising Star

@Artem Ervits which property?

avatar
Master Mentor

avatar
New Contributor

I have compiled the hadoop again with snappy :

svn checkout http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.5.0

mvn package -Drequire.snappy -Pdist,native,src -DskipTests -Dtar

but got the same exception again...

I have also checked the hadoop-env.sh:

export JAVA_LIBRARY_PATH=${JAVA_LIBRARY_PATH}

avatar
Rising Star

The problem is solve by making the following change in the spark config:

2840-cparkconfig.jpg

Thanks for the help guys!

avatar
Super Collaborator

just want to add that it seems the spark.driver.extraClassPath is not necessary, at least in my case when I write file in snappy in spark using:

rdd.saveAsTextFile(path, SnappyCodec.class)

avatar
Contributor

For me adding the line below to spark-defaults.conf helped based on packages installed on my test cluster.

spark.executor.extraLibraryPath /usr/hdp/current/hadoop-client/lib/native/:/usr/hdp/current/share/lzo/0.6.0/lib/native/Linux-amd64-64/