this version of libhadoop was built without snappy support.

[#1]

Hi,

I hope it's the right place to ask the following question 🙂

I try to put in hdfs a file with snappy compression. I write a Java code for that and when I try to run it on my cluster I got the following exception:

Exception in thread "main" java.lang.RuntimeException: native snappy library not available: this version of libhadoop was built without snappy support.

at org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:65)

at org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:134)

at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:150)

at org.apache.hadoop.io.compress.CompressionCodec$Util.createOutputStreamWithCodecPool(CompressionCodec.java:131)

at org.apache.hadoop.io.compress.SnappyCodec.createOutputStream(SnappyCodec.java:99)

Apparently the snappy library is not available... I check on the os with the following cmd "rpm -qa | less | grep snappy" and snappy and snappy-devel is present.

In the configuration of hdfs (core-site.xml) org.apache.hadoop.io.compress.SnappyCodec is present in the field io.compression.codecs.

Does anyone has a idea why it's not working?

Thanks in advance

[#2]

@Neeraj Sabharwal

Thanks for the reply, In my case it's not a solution because when I'm doing

hadoop checknative -a

I see that the snappy lib is true located at / usr/hdp/2.3.4.0-3485/hadoop/lib/native/libsnappy.so.1.

[#3]

We have the same problem.

> hadoop checknative -a

snappy: true /usr/hdp/2.3.4.0-3485/hadoop/lib/native/libsnappy.so.1

> rpm -qa snappy

snappy-1.1.0-3.el7.x86_64

What else can I check?

[#4]

Please confirm that you have the following property set correctly in hadoop-env.sh

[#5]

@Artem Ervits which property?

[#6]

Sorry, forgot to link https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.4/bk_upgrading_hdp_manually/content/configure...

[#7]

I have compiled the hadoop again with snappy :

svn checkout http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.5.0

mvn package -Drequire.snappy -Pdist,native,src -DskipTests -Dtar

but got the same exception again...

I have also checked the hadoop-env.sh:

export JAVA_LIBRARY_PATH=${JAVA_LIBRARY_PATH}

[#8 · Accepted solution]

The problem is solve by making the following change in the spark config:

Thanks for the help guys!

[#9]

just want to add that it seems the spark.driver.extraClassPath is not necessary, at least in my case when I write file in snappy in spark using:

rdd.saveAsTextFile(path, SnappyCodec.class)

[#10]

For me adding the line below to spark-defaults.conf helped based on packages installed on my test cluster.

spark.executor.extraLibraryPath /usr/hdp/current/hadoop-client/lib/native/:/usr/hdp/current/share/lzo/0.6.0/lib/native/Linux-amd64-64/