Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Can we dist cp multiple directories from one cluster to other server in one go?

avatar
Contributor

I need to distcp hdfs://nn1:port/foo/bar/2016/* and

hdfs://nn1:port/foo/bar/2017/* into

hdfs://nn2:port/foo/bar/2016/* and

hdfs://nn2:port/foo/bar/2017/* in one go..How can we achieve that? If I use distcp

hdfs://nn1:port/foo/bar/2016/

\

hdfs://nn1:port/foo/bar/2017

hdfs://nn2:port/foo/bar/ then underlying directoris i.e. 2016 and 2017 are not getting created. I need distcp to create these directories and copy data.

1 ACCEPTED SOLUTION

avatar
Rising Star

1. Below command works, i tried it. It also creates folders 2016 and 2017

hadoop distcp hdfs://nn1:port/foo/bar/2016 hdfs://nn1:port/foo/bar/2017 hdfs://nn2:port/foo/bar

2. If not working can you try this (Actually both are same)

hadoop distcp -f hdfs://nn1:port/srclist hdfs://nn2:port/foo/bar

Where srclist has

hdfs://nn1:port/foo/bar/2016
hdfs://nn1:port/foo/bar/2017

View solution in original post

1 REPLY 1

avatar
Rising Star

1. Below command works, i tried it. It also creates folders 2016 and 2017

hadoop distcp hdfs://nn1:port/foo/bar/2016 hdfs://nn1:port/foo/bar/2017 hdfs://nn2:port/foo/bar

2. If not working can you try this (Actually both are same)

hadoop distcp -f hdfs://nn1:port/srclist hdfs://nn2:port/foo/bar

Where srclist has

hdfs://nn1:port/foo/bar/2016
hdfs://nn1:port/foo/bar/2017