question Re: Distcp compression not working in Archives of Support Questions (Read Only)

Distcp compression not working

arun_bigdata_de — Wed, 05 Oct 2016 12:02:22 GMT

Hi, I am trying to compress the output of distcp but it doesn't compress the output . pls help me to compress it . below is the command using

hadoop distcp -D mapreduce.output.fileoutputformat.compress=true -D mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.BZip2Codec inputdir outputdir

Re: Distcp compression not working

SK1 — Wed, 05 Oct 2016 20:30:46 GMT

@Arun Reddy Which version of hdp you are using ?

Re: Distcp compression not working

arun_bigdata_de — Thu, 06 Oct 2016 01:46:10 GMT

Apache raw version Hadoop 2.7.1

Re: Distcp compression not working

mqureshi — Thu, 06 Oct 2016 02:59:42 GMT

@Arun Reddy

This feature is still not available in Hadoop by default. You can add a patch but distcp doesn't compress data. following JIRA will give you all the details including the patch you want to download.

https://issues.apache.org/jira/browse/HADOOP-8065

Following is the new JIRA

https://issues.apache.org/jira/browse/HADOOP-13114 --> use this one if you decide to apply the patch.

Re: Distcp compression not working

arun_bigdata_de — Thu, 06 Oct 2016 13:07:49 GMT

Thank you @mqureshi . how about HDP 2.4 .does that patch included in HDP 2.4 and above ?

Re: Distcp compression not working

mqureshi — Thu, 06 Oct 2016 23:21:18 GMT

Negative. If you check the Jira's, they are unresolved. We don't ship unresolved issues in our product. So, your only option right now is to download the patch and apply to your installation. That will affect support if you have that because you are applying a non hortonworks patch.

I would suggest that you simply distcp the file and then compress it. You are only saving a step. It's not saving you any time or giving better performance.

Re: Distcp compression not working

arun_bigdata_de — Fri, 07 Oct 2016 13:20:21 GMT

Thanks for your time . I am bringing the dir to local and applying compression