Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

lzo compression

lzo compression

New Contributor

Hi Team,

We have setup a 3 node test cluster running on top of Centos 6.3 64 bit OS with yarn configuration enabled.We are able to run mapreduce application on the cluster.When we tried to add LZO compression and load data to the cluster, we are getting compression error.

Following properties are added in the core-site.xml


<property>
<name>io.compression.codecs</name>
<value>org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.BZip2Codec,com.hadoop.compression.lzo.LzoCodec,com.hadoop.compression.lzo.LzopCodec</value>
</property>

<property>
<name>io.compression.codec.lzo.class</name>
<value>com.hadoop.compression.lzo.LzoCodec</value>


The error details are as below.

ERROR [main] 2014-03-31 18:33:50,407 GPLNativeCodeLoader.java (line 35) Could not load native gpl library
java.lang.UnsatisfiedLinkError: no gplcompression in java.library.path
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1860)
at java.lang.Runtime.loadLibrary0(Runtime.java:845)
at java.lang.System.loadLibrary(System.java:1084)
at com.hadoop.compression.lzo.GPLNativeCodeLoader.<clinit>(GPLNativeCodeLoader.java:31)
at com.hadoop.compression.lzo.LzoCodec.isNativeLzoLoaded(LzoCodec.java:69)
at com.hadoop.compression.lzo.LzoCodec.createCompressor(LzoCodec.java:154)
at com.hadoop.compression.lzo.LzoCodec.createOutputStream(LzoCodec.java:99)
at com.acunu.loader.HDFSLoader.validateAndSplitContents(HDFSLoader.java:333)
at com.acunu.loader.HDFSLoader.loadFile(HDFSLoader.java:201)
at com.acunu.loader.FileLoader.loadFiles(FileLoader.java:81)
at com.acunu.loader.HDFSLoader.main(HDFSLoader.java:383)
ERROR [main] 2014-03-31 18:33:50,408 LzoCodec.java (line 79) Cannot load native-lzo without native-hadoop
ERROR [main] 2014-03-31 18:33:50,408 FileLoader.java (line 46) Fatal exception in thread Thread[main,5,main]
java.lang.RuntimeException: native-lzo library not available
at com.hadoop.compression.lzo.LzoCodec.createCompressor(LzoCodec.java:155)
at com.hadoop.compression.lzo.LzoCodec.createOutputStream(LzoCodec.java:99)
at com.acunu.loader.HDFSLoader.validateAndSplitContents(HDFSLoader.java:333)
at com.acunu.loader.HDFSLoader.loadFile(HDFSLoader.java:201)
at com.acunu.loader.FileLoader.loadFiles(FileLoader.java:81)
at com.acunu.loader.HDFSLoader.main(HDFSLoader.java:383)

Thank you.
Addy
1 REPLY 1

Re: lzo compression

Master Collaborator

I'm not sure what version of CDH you're using, but it looks like you don't have your native hadoop libs installed (eg. libhadoop.so.1.0.0).

 

Did you try building the LZO jar yourself via these instructions?  It looks like your problem is that it's trying to find the GPL compression libraries and can't.  I think Todd's version of LZO will work for you.

Don't have an account?
Coming from Hortonworks? Activate your account here