Reply
New Contributor
Posts: 1
Registered: ‎09-19-2018

replacing zlib

Hi, I am new and not sure if this is the right board to ask following questions.

 

We are planning to use our own implementation of zlib for default compression in Hadoop ( for performance reason ), and our host OS is Linux ( say Ubuntu ).

 

1. Which default libraries does Hadoop use? On our OS, /lib/x86_64-linux-gnu/libz.so.1.2.8 is installed? Does this mean  Hadoop uses this one as default DEFLATE library? 

 

2. If so, which Hadoop configuration files specify parameters for the DEFLATE algorithm ( and libz.so.1.28 )?

 

3. How does Hadoop uses zlib, i.e., which source files contain lines related to using zlib, or which Hadoop source files define AIPs to call zlib?

 

Thanks for any help.

Highlighted
Cloudera Employee
Posts: 44
Registered: ‎08-13-2014

Re: replacing zlib

Hi,

You can use the command `hadoop checknative` to see which native libraries are being loaded. In the example below I check the native libraries and then change LD_LIBRARY_PATH and run the command again. In the second example you can see I'm loading libz from a different location.

 

[cloudera@quickstart ~]$ hadoop checknative
18/09/27 14:06:47 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native
18/09/27 14:06:47 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
Native library checking:
hadoop:  true /usr/lib/hadoop/lib/native/libhadoop.so.1.0.0
zlib:    true /lib64/libz.so.1
snappy:  true /usr/lib/hadoop/lib/native/libsnappy.so.1
lz4:     true revision:10301
bzip2:   true /lib64/libbz2.so.1
openssl: true /usr/lib64/libcrypto.so
[cloudera@quickstart ~]$ cp /lib64/libz.so.1 lib/
[cloudera@quickstart ~]$ export LD_LIBRARY_PATH=$PWD/lib
[cloudera@quickstart ~]$ hadoop checknative
18/09/27 14:07:29 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native
18/09/27 14:07:29 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
Native library checking:
hadoop:  true /usr/lib/hadoop/lib/native/libhadoop.so.1.0.0
zlib:    true /home/cloudera/lib/libz.so.1
snappy:  true /usr/lib/hadoop/lib/native/libsnappy.so.1
lz4:     true revision:10301
bzip2:   true /lib64/libbz2.so.1
openssl: true /usr/lib64/libcrypto.so

 

Hope this helps,

Jim

Announcements