question Re: Can we dist cp multiple directories from one cluster to other server in one go? in Support Questions

Can we dist cp multiple directories from one cluster to other server in one go?

nilesh_shrimant — Thu, 05 Oct 2017 16:39:53 GMT

I need to distcp hdfs://nn1:port/foo/bar/2016/* and

hdfs://nn1:port/foo/bar/2017/* into

hdfs://nn2:port/foo/bar/2016/* and

hdfs://nn2:port/foo/bar/2017/* in one go..How can we achieve that? If I use distcp

hdfs://nn1:port/foo/bar/2016/

hdfs://nn1:port/foo/bar/2017

hdfs://nn2:port/foo/bar/ then underlying directoris i.e. 2016 and 2017 are not getting created. I need distcp to create these directories and copy data.

Re: Can we dist cp multiple directories from one cluster to other server in one go?

nshetty — Wed, 18 Oct 2017 01:56:52 GMT

1. Below command works, i tried it. It also creates folders 2016 and 2017

hadoop distcp hdfs://nn1:port/foo/bar/2016 hdfs://nn1:port/foo/bar/2017 hdfs://nn2:port/foo/bar

2. If not working can you try this (Actually both are same)

hadoop distcp -f hdfs://nn1:port/srclist hdfs://nn2:port/foo/bar

Where srclist has

hdfs://nn1:port/foo/bar/2016
hdfs://nn1:port/foo/bar/2017