question Re: How to copy files from HDFS recursive to the local file system in Support Questions

How to copy files from HDFS recursive to the local file system

mike_bronson7 — Fri, 08 Mar 2019 00:42:08 GMT

How we can copy recursive jar files from HDFS ( jar files are under sub folders ) to local folder?

Example

export hdfs_folder=/app/lib export local_folder=/home/work_app

while under /app/lib we have the following sub folder with the jar files

/app/lib/folder_jar1
/app/lib/folder_jar2

Under each of above folder we have jar files

The following command , will copy only the jar files under /app/lib

but not the Jar files under the sub folders as /app/lib/folder_jar1 , /app/lib/folder_jar2 , Etc

hadoop fs -copyToLocal $hdfs_folder/*.jar $local_folder

Re: How to copy files from HDFS recursive to the local file system

JonathanSneep — Fri, 08 Mar 2019 16:46:22 GMT

Hi @Michael Bronson

You are specifying /folder/*.jar. If you want the .jar files from one level deeper, you would specify /folder/*/*.jar.

Or, here is an alternative example.

[hdfs@c2175-node4 stuff]$ hdfs dfs -find /tmp -name *.jar
/tmp/somefolder/y.jar
/tmp/x.jar
[hdfs@c2175-node4 stuff]$ for result in `hdfs dfs -find /tmp -name *.jar` ; do hdfs dfs -copyToLocal $result; done
[hdfs@c2175-node4 stuff]$ ls -al
-rw-r--r-- 1 hdfs hadoop  0 Mar  8 08:43 x.jar
-rw-r--r-- 1 hdfs hadoop  0 Mar  8 08:43 y.j