question Re: this version of libhadoop was built without snappy support. in Archives of Support Questions (Read Only)

this version of libhadoop was built without snappy support.

michelsumbul — Fri, 16 Sep 2022 10:05:19 GMT

Hi,

I hope it's the right place to ask the following question 🙂

I try to put in hdfs a file with snappy compression. I write a Java code for that and when I try to run it on my cluster I got the following exception:

Exception in thread "main" java.lang.RuntimeException: native snappy library not available: this version of libhadoop was built without snappy support.

at org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:65)

at org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:134)

at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:150)

at org.apache.hadoop.io.compress.CompressionCodec$Util.createOutputStreamWithCodecPool(CompressionCodec.java:131)

at org.apache.hadoop.io.compress.SnappyCodec.createOutputStream(SnappyCodec.java:99)

Apparently the snappy library is not available... I check on the os with the following cmd "rpm -qa | less | grep snappy" and snappy and snappy-devel is present.

In the configuration of hdfs (core-site.xml) org.apache.hadoop.io.compress.SnappyCodec is present in the field io.compression.codecs.

Does anyone has a idea why it's not working?

Thanks in advance

Re: this version of libhadoop was built without snappy support.

aervits — Tue, 23 Feb 2016 22:37:07 GMT

@Michel Sumbul

please post your code.

Re: this version of libhadoop was built without snappy support.

michelsumbul — Tue, 23 Feb 2016 22:44:35 GMT

here's the piece of code:

Path outFile = new Path(destPathFolder.toString() + "/" + listFolder[i].getName() + "_" + listFiles[b].getName() + ".txt");
FSDataOutputStream fin = dfs.create(outFile);
Configuration conf = new Configuration();
conf.setBoolean("mapreduce.map.output.compress", true);
conf.set("mapreduce.map.output.compress.codec", "org.apache.hadoop.io.compress.SnappyCodec");
CompressionCodecFactory codecFactory = new CompressionCodecFactory(conf);
CompressionCodec codec = codecFactory.getCodecByName("SnappyCodec");
CompressionOutputStream compressedOutput = codec.createOutputStream(fin);
FileReader input = new FileReader(listFiles[b]);
BufferedReader bufRead = new BufferedReader(input);
String myLine = null;
while ((myLine = bufRead.readLine()) != null) {
if (!myLine.isEmpty()) {
compressedOutput.write(myLine.getBytes());
compressedOutput.write('\n'); } }
compressedOutput.flush();
compressedOutput.close();

Re: this version of libhadoop was built without snappy support.

aervits — Tue, 23 Feb 2016 23:13:54 GMT

@Michel Sumbul

I don't remember setting CompressionCodecFactory explicitly. Just let configuration do its magic so remove the following

CompressionCodecFactory codecFactory = new CompressionCodecFactory(conf);
CompressionCodec codec = codecFactory.getCodecByName("SnappyCodec");
CompressionOutputStream compressedOutput = codec.createOutputStream(fin);

and

compressedOutput.write(myLine.getBytes());
compressedOutput.write('\n'); } }
compressedOutput.flush();
compressedOutput.close();

Re: this version of libhadoop was built without snappy support.

michelsumbul — Tue, 23 Feb 2016 23:22:20 GMT

Hi Artem,

Thanks for the fast reply. I don't really understand how it will work without the

compressedOutput.write(myLine.getBytes());
compressedOutput.write('\n'); } }
compressedOutput.flush();
compressedOutput.close();

How it will write to hdfs? Also if I remove the first part, when the configuration will be use?

Can you give me an example because, I don't see how it works without the part that you specify :s

Thanks in advance

Re: this version of libhadoop was built without snappy support.

aervits — Tue, 23 Feb 2016 23:30:25 GMT

@Michel Sumbul

it's been awhile, here's an example from definitive guide book

public class StreamCompressor {

  public static void main(String[] args) throws Exception {
    String codecClassname = args[0];
    Class<?> codecClass = Class.forName(codecClassname);
    Configuration conf = new Configuration();
    CompressionCodec codec = (CompressionCodec)
      ReflectionUtils.newInstance(codecClass, conf);
    
    CompressionOutputStream out = codec.createOutputStream(System.out);
    IOUtils.copyBytes(System.in, out, 4096, false);
    out.finish();
  }
}

Re: this version of libhadoop was built without snappy support.

nsabharwal — Wed, 24 Feb 2016 00:06:52 GMT

@Michel Sumbul See this thread. The same solution was used in the past by a client.

Re: this version of libhadoop was built without snappy support.

michelsumbul — Wed, 24 Feb 2016 16:52:44 GMT

@Artem Ervits I just made the test with the example of the definitve guide and I still have exactly the same error:

Exception in thread "main" java.lang.RuntimeException: native snappy library not available: this version of libhadoop was built without snappy support.

Any idea?

Re: this version of libhadoop was built without snappy support.

michelsumbul — Wed, 24 Feb 2016 16:59:47 GMT

@Neeraj Sabharwal

Thanks for the reply, In my case it's not a solution because when I'm doing

hadoop checknative -a

I see that the snappy lib is true located at / usr/hdp/2.3.4.0-3485/hadoop/lib/native/libsnappy.so.1.

Re: this version of libhadoop was built without snappy support.

aervits — Wed, 24 Feb 2016 19:56:25 GMT

Please go through these steps and confirm everything is in place https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_installing_manually_book/content/install_compression_libraries.html and just double-check snappy versions according to this doc http://docs.hortonworks.com/HDPDocuments/Ambari-2.1.1.0/bk_releasenotes_ambari_2.1.1.0/content/ambari_relnotes-2.1.1.0-known-issues.html

Re: this version of libhadoop was built without snappy support.

olga_katzenelso — Thu, 03 Mar 2016 21:05:18 GMT

We have the same problem.

> hadoop checknative -a

snappy: true /usr/hdp/2.3.4.0-3485/hadoop/lib/native/libsnappy.so.1

> rpm -qa snappy

snappy-1.1.0-3.el7.x86_64

What else can I check?

Re: this version of libhadoop was built without snappy support.

aervits — Sat, 05 Mar 2016 23:03:29 GMT

Can you post your maven pom?

Re: this version of libhadoop was built without snappy support.

aervits — Tue, 08 Mar 2016 19:09:26 GMT

Please confirm that you have the following property set correctly in hadoop-env.sh

Re: this version of libhadoop was built without snappy support.

michelsumbul — Mon, 14 Mar 2016 17:31:48 GMT

@Artem Ervits which property?

Re: this version of libhadoop was built without snappy support.

aervits — Mon, 14 Mar 2016 18:22:24 GMT

Sorry, forgot to link https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.4/bk_upgrading_hdp_manually/content/configure-hadoop-22.html

Re: this version of libhadoop was built without snappy support.

michelsumbul — Sun, 18 Aug 2019 12:34:10 GMT

The problem is solve by making the following change in the spark config:

Thanks for the help guys!

Re: this version of libhadoop was built without snappy support.

olga_katzenelso — Thu, 17 Mar 2016 16:18:49 GMT

I have compiled the hadoop again with snappy :

svn checkout http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.5.0

mvn package -Drequire.snappy -Pdist,native,src -DskipTests -Dtar

but got the same exception again...

I have also checked the hadoop-env.sh:

export JAVA_LIBRARY_PATH=${JAVA_LIBRARY_PATH}

Re: this version of libhadoop was built without snappy support.

David_Tam — Thu, 09 Jun 2016 20:15:52 GMT

just want to add that it seems the spark.driver.extraClassPath is not necessary, at least in my case when I write file in snappy in spark using:

rdd.saveAsTextFile(path, SnappyCodec.class)

Re: this version of libhadoop was built without snappy support.

kushalbohra — Mon, 12 Mar 2018 03:32:07 GMT

For me adding the line below to spark-defaults.conf helped based on packages installed on my test cluster.

spark.executor.extraLibraryPath /usr/hdp/current/hadoop-client/lib/native/:/usr/hdp/current/share/lzo/0.6.0/lib/native/Linux-amd64-64/