Support Questions

Find answers, ask questions, and share your expertise
Announcements
Check out our newest addition to the community, the Cloudera Data Analytics (CDA) group hub.

Snappy Installation on HDP

Explorer

I have a requirement to compress the HBase tabes in a HDP cluster using snappy. However, I have some clarifications over Snappy installation on HDP.

I executed the Compression Test even without installing snappy and I got a success output. I tried both the commands below.

hbase org.apache.hadoop.hbase.util.CompressionTest file:///tmp/test.txt snappy
hbase org.apache.hadoop.hbase.util.CompressionTest hdfs://hbase.primary.namenode:8020/tmp/test1.txt snappy

On both the occasions the output was:

2017-10-30 11:25:18,454 INFO  [main] hfile.CacheConfig: CacheConfig:disabled
2017-10-30 11:25:18,671 INFO  [main] compress.CodecPool: Got brand-new compressor [.snappy]
2017-10-30 11:25:18,679 INFO  [main] compress.CodecPool: Got brand-new compressor [.snappy]
2017-10-30 11:25:21,560 INFO  [main] hfile.CacheConfig: CacheConfig:disabled
2017-10-30 11:25:22,366 INFO  [main] compress.CodecPool: Got brand-new decompressor [.snappy]
SUCCESS

I see below libraries in the path /usr/hdp/2.6.1.0-129/hadoop/lib/native/ as well.

libhadoop.a 
libhadooppipes.a 
libhadoop.so 
libhadoop.so.1.0.0 
libhadooputils.a 
libhdfs.a 
libsnappy.so 
libsnappy.so.1 
libsnappy.so.1.1.4

So my questions are:

Does HDP support snappy compression by default? Don't we need to install Snappy?

Please verify this. The HDP version I'm using is 2.6.1.

2 REPLIES 2

Super Mentor

@Thanuja Kularathna

As per the doc of HDP you should install snappy-devel packages on all hosts.

----For RHEL/CentOS/Oracle Linux:
yum install snappy snappy-devel

----Execute the following command at all the nodes in your cluster:
 RHEL/CentOS/Oracle Linux:
yum install lzo lzo-devel hadooplzo hadooplzo-native

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.2/bk_command-line-installation/content/instal...

.

Explorer
Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.