- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
How to confirm my files are snappy compresses in hive?
- Labels:
-
Apache Hadoop
-
Apache Hive
Created ‎02-27-2018 07:57 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
So I compressed my table in hive using snappy compression and it did get compress. The size was reduced.
But when i run hadoop fs -lsr /hive/user.db/table_name, I see no file extensions with .snappy.
I want to know if they really were snappy compressed or not?
Created ‎03-02-2018 01:26 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
If you describe the table in hive, you should be able to see the compression algorithm applied to the table:
hive> describe formatted <hive_table>;
I don't believe there is a quick way to see if it is compressed via HDFS.
Created ‎03-02-2018 05:38 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Yup @Michael Young.
Another way I found was through
hadoop fs -text <file-location>
On the top of results, INFO compress.CodecPool: Got brand-new decompressor [.snappy] is written which I think is a confirmation that snappy compression is applied.
Created ‎03-02-2018 05:42 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
That is handy, thank you for sharing! If you think my response was helpful, please accept the answer to make it easier for others to find answers.
