question Re: How to reduce HDFS output file size? in Archives of Support Questions (Read Only)

How to reduce HDFS output file size?

connectchen — Fri, 11 Dec 2015 19:41:59 GMT

Sometimes I insert data to hive table using two ways:Hive and Hive on Tez.The HDFS output file size is twice when using hive on Tez. It take up more hdfs space.Is there any configurations to reduce the size?

Re: How to reduce HDFS output file size?

landon_t_robins — Fri, 11 Dec 2015 22:50:02 GMT

Have you looked into CompressedStorage features on Hive?

You should be able to use this (for Snappy at least):

SET hive.exec.compress.output=true;
SET mapred.output.compression.codec=org.apache.hadoop.io.compress.SnappyCodec;
SET mapred.output.compression.type=BLOCK;

Re: How to reduce HDFS output file size?

aervits — Wed, 03 Feb 2016 23:46:03 GMT

@Jun Chen are you still having issues with this? Can you accept best answer or provide your own solution?