Support Questions
Find answers, ask questions, and share your expertise

where can i get a copy of DISTCP2

New Contributor

I've got HDP 2.6 installed and i see a bit of documentation on DISTCP2 being able to copy inter-cluster.

I hear Falcon's falling out of favor and I want to use DISTCP2.

Where can I download the JAR/Package/Source/whatever for it?


Super Collaborator

@James Liu - distcp2 is the default with Mapreduce2. It is not called distcp2, but just distcp.

Rising Star

Distcp2 is the default and only distcp available in MR2. The command 'hadoop distcp' in MR2 runs on distcp2 automatically.

Super Guru

@James Liu

I believe HDP 2.X onwards, we have distcp2 by default in backend for hadoop distcp command.

Tagging HDFS engineers here just to confirm this thing.

@Arpit Agarwal @aengineer

Please correct my answer if needed or feel free to add anything which is missing.

Expert Contributor

I'm running CDH 6.3 ( Hadoop 3.0.0 and MR2 ).   If what you are saying is true, then shouldn't I be able to pass the "-direct" option to my hadoop distcp commands?   Currently it fails to recognize this command line option, which tells me distcp2 isn't being used.

Expert Contributor

Answered my own question.  I mistakingly thought distcp2 supported the -direct copy option but it does not.