using snappy and other compressions with Nifi hdfs components

[#1]

A usual query that comes up is on using snappy and other compression codes when loading data from hdfs using the nifi puthdfs component. A common error that users come across is "java.lang.UnsatisfiedLinkError". This error occurs because snappy and other compression codecs, which are part of the native linux binaries and which the java libraries for these codecs utilize to do the actual compression, are not in the path of the JVM. Since the JVM cannot find them the bindings fails and you get a java.lang.UnsatisfiedLinkError.

Follow the following steps to resolve this issue.

1. copy the native folder containing the compression libraries, from one of your hadoop nodes.

cd /usr/hdp/x.x.x.x-xx/hadoop/lib/

tar -cf ~/native.tar native/

2. scp the native.tar from your hadoop node to your NiFi node, untar to a location of your choice. in my case i use /home/myuser/hadoop/

cd ~

mkdir hadoop

cd hadoop

tar -cf /path/to/native.tar

3. go to your nifi folder , and open conf/bootstrap.conf and add the following jvn argument for java.library.path pointing to your folder containing the native hadoop binaries. (/home/myuser/hadoop/native in my case )

java.arg.15=-Djava.library.path=/home/myuser/hadoop/native/

i used 15 because that was the number for the last jvm argument in my bootstrap.conf.

Alternately, you can edit bin/nifi-env.sh and add

export LD_LIBRARY_PATH=/home/mysuser/hadoop/native/

4. restart nifi.