I have an old CDH (unsecured) cluster (5.4.3) that I am trying to migrate a directory of data from to a new HDP (2.4.x) cluster. Webhdfs has been disabled on the source (unsecured) cluster.
So far I have tried distcp, however due to the mismatched versions, and the secure/unsecure issues, hdfs:// > hdfs:// does not work, hftp > hdfs does not work and webhdfs:// > webhdfs:// cannot work. (HDFS-7037 HDFS-6776)
I also tried sqooping the files from Hive(unsecured cluster) to HDFS (secured cluster) but version differences again prevent that avenue. (HIVE-6050) Outside of writing a script to execute on the unsecured cluster, and pushing data through the rest api on the secured cluster's version of webhdfs I can't think of a faster way.
Before I do that I figured I'd ask if anyone else had a better idea?
... View more
