Created 10-19-2017 04:14 PM
Hello,
I would like to move lot of files of a hdfs directory but not the files with size at 0 and name like ".*"
For example, move only files "file3" "file4" file5" but not the files "file1" and "file2". These ones have'ont been not entirely written in hdfs directory when I execute the "hdfs dfs -mv" command.
hdfs@host:~> hadoop dfs -ls /mydirectory
Found 1942 items
-rw-r----- 3 xagcla02 hdfs 0 2017-10-19 18:07 /mydirectory/.file1
-rw-r----- 3 xagcla02 hdfs 0 2017-10-19 18:07 /mydirectory/.file2
-rw-r----- 3 xagcla02 hdfs 2540 2017-10-19 18:07 /mydirectory/file3
-rw-r----- 3 xagcla02 hdfs 2540 2017-10-19 18:07 /mydirectory/file4
-rw-r----- 3 xagcla02 hdfs 5252 2017-10-19 18:07 /mydirectory/file5
…
Thanks for your feedbacks
Created 10-23-2017 10:56 AM
you can try below script:
for f in $(hdfs dfs -ls /tmp/files | awk '$1 !~ /^d/ && $5 == "0" { print $8 }'); do hdfs dfs -mv "$f" /tmp/files/exclude-files; done
Created 10-20-2017 09:09 PM
There isn't an exclusion filter per se (there isn't one for the native UNIX mv command either) but if the files you want to move are named with a pattern you could use that pattern to select only those files to move. The wildcard operator (*) will come in handy in this situation as well.
Created 10-23-2017 06:40 AM
Except "." character and timestamp, all the files have the same name. So It's impossible to use pattern.
Created 10-23-2017 10:56 AM
you can try below script:
for f in $(hdfs dfs -ls /tmp/files | awk '$1 !~ /^d/ && $5 == "0" { print $8 }'); do hdfs dfs -mv "$f" /tmp/files/exclude-files; done
Created 10-23-2017 11:35 AM
Thanks a lot Pavan., I've just modified ... $5 == "0" ... by ... $5 != "0" ... because I don't want to move files with "0"size.
for f in $(hdfs dfs -ls /tmp/files | awk '$1 !~ /^d/ && $5 != "0" { print $8 }');do hdfs dfs -mv "$f"/tmp/files/exclude-files;done