Community Articles
Find and share helpful community-sourced technical articles
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.
Labels (2)
Expert Contributor

A usual query that comes up is on using snappy and other compression codes when loading data from hdfs using the nifi puthdfs component. A common error that users come across is "java.lang.UnsatisfiedLinkError". This error occurs because snappy and other compression codecs, which are part of the native linux binaries and which the java libraries for these codecs utilize to do the actual compression, are not in the path of the JVM. Since the JVM cannot find them the bindings fails and you get a java.lang.UnsatisfiedLinkError.

Follow the following steps to resolve this issue.

1. copy the native folder containing the compression libraries, from one of your hadoop nodes.

cd /usr/hdp/x.x.x.x-xx/hadoop/lib/

tar -cf ~/native.tar native/

2. scp the native.tar from your hadoop node to your NiFi node, untar to a location of your choice. in my case i use /home/myuser/hadoop/

cd ~

mkdir hadoop

cd hadoop

tar -cf /path/to/native.tar

3. go to your nifi folder , and open conf/bootstrap.conf and add the following jvn argument for java.library.path pointing to your folder containing the native hadoop binaries. (/home/myuser/hadoop/native in my case )

java.arg.15=-Djava.library.path=/home/myuser/hadoop/native/

i used 15 because that was the number for the last jvm argument in my bootstrap.conf.

Alternately, you can edit bin/nifi-env.sh and add

export LD_LIBRARY_PATH=/home/mysuser/hadoop/native/

4. restart nifi.

1,080 Views
Don't have an account?
Coming from Hortonworks? Activate your account here
Version history
Revision #:
1 of 1
Last update:
‎12-15-2016 03:11 PM
Updated by:
 
Contributors
Top Kudoed Authors