Created 10-30-2017 06:13 AM
I have a requirement to compress the HBase tabes in a HDP cluster using snappy. However, I have some clarifications over Snappy installation on HDP.
I executed the Compression Test even without installing snappy and I got a success output. I tried both the commands below.
hbase org.apache.hadoop.hbase.util.CompressionTest file:///tmp/test.txt snappy hbase org.apache.hadoop.hbase.util.CompressionTest hdfs://hbase.primary.namenode:8020/tmp/test1.txt snappy
On both the occasions the output was:
2017-10-30 11:25:18,454 INFO [main] hfile.CacheConfig: CacheConfig:disabled 2017-10-30 11:25:18,671 INFO [main] compress.CodecPool: Got brand-new compressor [.snappy] 2017-10-30 11:25:18,679 INFO [main] compress.CodecPool: Got brand-new compressor [.snappy] 2017-10-30 11:25:21,560 INFO [main] hfile.CacheConfig: CacheConfig:disabled 2017-10-30 11:25:22,366 INFO [main] compress.CodecPool: Got brand-new decompressor [.snappy] SUCCESS
I see below libraries in the path /usr/hdp/2.6.1.0-129/hadoop/lib/native/ as well.
libhadoop.a libhadooppipes.a libhadoop.so libhadoop.so.1.0.0 libhadooputils.a libhdfs.a libsnappy.so libsnappy.so.1 libsnappy.so.1.1.4
So my questions are:
Does HDP support snappy compression by default? Don't we need to install Snappy?
Please verify this. The HDP version I'm using is 2.6.1.
Created 10-30-2017 06:17 AM
As per the doc of HDP you should install snappy-devel packages on all hosts.
----For RHEL/CentOS/Oracle Linux: yum install snappy snappy-devel ----Execute the following command at all the nodes in your cluster: RHEL/CentOS/Oracle Linux: yum install lzo lzo-devel hadooplzo hadooplzo-native
.
Created 09-04-2018 11:16 AM
Got this answer from stackoverflow.
https://stackoverflow.com/questions/47029697/installing-snappy-on-a-hdp-cluster