question Re: Migrating data off an old CDH cluster to a new HDP cluster in Archives of Support Questions (Read Only)

Migrating data off an old CDH cluster to a new HDP cluster

efusaro — Fri, 23 Dec 2016 04:14:36 GMT

I have an old CDH (unsecured) cluster (5.4.3) that I am trying to migrate a directory of data from to a new HDP (2.4.x) cluster. Webhdfs has been disabled on the source (unsecured) cluster. So far I have tried distcp, however due to the mismatched versions, and the secure/unsecure issues, hdfs:// > hdfs:// does not work, hftp > hdfs does not work and webhdfs:// > webhdfs:// cannot work. (HDFS-7037 HDFS-6776)

I also tried sqooping the files from Hive(unsecured cluster) to HDFS (secured cluster) but version differences again prevent that avenue. (HIVE-6050)

Outside of writing a script to execute on the unsecured cluster, and pushing data through the rest api on the secured cluster's version of webhdfs I can't think of a faster way. Before I do that I figured I'd ask if anyone else had a better idea?

Re: Migrating data off an old CDH cluster to a new HDP cluster

ssoldatov — Fri, 23 Dec 2016 09:12:17 GMT

Could you please explain more why distcp hftp -> hdfs didn't work? This is the recommended way to copy between different versions of hadoop. And it should be running on destination cluster.

Re: Migrating data off an old CDH cluster to a new HDP cluster

efusaro — Fri, 23 Dec 2016 21:52:55 GMT

If you check out the Jira tickets the issue revolves around needing to execute from the secure cluster and read from the insecure cluster. https://issues.apache.org/jira/browse/HDFS-7037

Re: Migrating data off an old CDH cluster to a new HDP cluster

myoung — Sat, 24 Dec 2016 02:11:35 GMT

@Erin Fusaro

Have you considered using NiFi to transfer the data? The FetchHDFS process would be configured to communicate to the unsecure cluster and the PutHDFS processor would be configured to communicate with the secure cluster.

Re: Migrating data off an old CDH cluster to a new HDP cluster

efusaro — Sat, 24 Dec 2016 07:39:24 GMT

Great suggestion! I may try that first.