Move file from one HDFS directoy to another using scala/java

[#1]

I've files in one hdfs folder and after checking few things i wanted to move that file to another directory on hdfs.

Currently i am using filesystem object with rename it is doing the job but it is actually renaming the file with complete path.

Do have any other way to do it?

Appriciate your help.

Thanks,

[#2]

@gnovak, I am still wondering why it has created the directory on my local machine? Kind of wired...

Related to this i have another issue, i am also reading files from hdfs directory using wholeTextFile() my hdfs input directory has text files and sub directories in it. On my local development machine i was able to read the files where wholeTextFile() was not considering sub directories, however whenever i deployed the same code cluster, it started to consider sub directories as well. Do you have any idea on this? Appreciate your help on this

[#3]

@RAUI wholeTextFile() is not part of the HDFS API, I'm assuming you're using Spark, with which I'm not too familiar. I suggest you to post another question for this to HCC.

[#4]

@RAUI

Yes there is another way of achieving this. You can use the method copy() from the FileUtil class and pass your FileSystem object to it to effectively copy your files from the source HDFS location to the target. As with using rename() you will need to ensure you target directory is created before calling copy. FileUtil.copy() has a signature where you provide a source and destination FS and in this case you would provide the same FS object since you are looking to copy files to a different location on the same HDFS. There is also a boolean option to delete the source file after the copy if that fits your use case.

Here is a link to the FileUtil API: http://hadoop.apache.org/docs/r2.8.0/api/org/apache/hadoop/fs/FileUtil.html