Support Questions
Find answers, ask questions, and share your expertise

How to copy files from HDFS recursive to the local file system

Solved Go to solution

How to copy files from HDFS recursive to the local file system

How we can copy recursive jar files from HDFS ( jar files are under sub folders ) to local folder?

Example

export hdfs_folder=/app/lib export local_folder=/home/work_app

while under /app/lib we have the following sub folder with the jar files

as

/app/lib/folder_jar1
/app/lib/folder_jar2

Under each of above folder we have jar files

The following command , will copy only the jar files under /app/lib

but not the Jar files under the sub folders as /app/lib/folder_jar1 , /app/lib/folder_jar2 , Etc

hadoop fs -copyToLocal $hdfs_folder/*.jar $local_folder
Michael-Bronson
1 ACCEPTED SOLUTION

Accepted Solutions

Re: How to copy files from HDFS recursive to the local file system

Hi @Michael Bronson

You are specifying /folder/*.jar. If you want the .jar files from one level deeper, you would specify /folder/*/*.jar.

Or, here is an alternative example.


[hdfs@c2175-node4 stuff]$ hdfs dfs -find /tmp -name *.jar
/tmp/somefolder/y.jar
/tmp/x.jar
[hdfs@c2175-node4 stuff]$ for result in `hdfs dfs -find /tmp -name *.jar` ; do hdfs dfs -copyToLocal $result; done
[hdfs@c2175-node4 stuff]$ ls -al
-rw-r--r-- 1 hdfs hadoop  0 Mar  8 08:43 x.jar
-rw-r--r-- 1 hdfs hadoop  0 Mar  8 08:43 y.j

View solution in original post

1 REPLY 1

Re: How to copy files from HDFS recursive to the local file system

Hi @Michael Bronson

You are specifying /folder/*.jar. If you want the .jar files from one level deeper, you would specify /folder/*/*.jar.

Or, here is an alternative example.


[hdfs@c2175-node4 stuff]$ hdfs dfs -find /tmp -name *.jar
/tmp/somefolder/y.jar
/tmp/x.jar
[hdfs@c2175-node4 stuff]$ for result in `hdfs dfs -find /tmp -name *.jar` ; do hdfs dfs -copyToLocal $result; done
[hdfs@c2175-node4 stuff]$ ls -al
-rw-r--r-- 1 hdfs hadoop  0 Mar  8 08:43 x.jar
-rw-r--r-- 1 hdfs hadoop  0 Mar  8 08:43 y.j

View solution in original post