how to exclude .* files while executing "hdfs dfs -mv" command ?

[#1]

Hello,

I would like to move lot of files of a hdfs directory but not the files with size at 0 and name like ".*"

For example, move only files "file3" "file4" file5" but not the files "file1" and "file2". These ones have'ont been not entirely written in hdfs directory when I execute the "hdfs dfs -mv" command.

hdfs@host:~> hadoop dfs -ls /mydirectory

Found 1942 items

-rw-r----- 3 xagcla02 hdfs 0 2017-10-19 18:07 /mydirectory/.file1

-rw-r----- 3 xagcla02 hdfs 0 2017-10-19 18:07 /mydirectory/.file2

-rw-r----- 3 xagcla02 hdfs 2540 2017-10-19 18:07 /mydirectory/file3

-rw-r----- 3 xagcla02 hdfs 2540 2017-10-19 18:07 /mydirectory/file4

-rw-r----- 3 xagcla02 hdfs 5252 2017-10-19 18:07 /mydirectory/file5

…

Thanks for your feedbacks

[#2]

There isn't an exclusion filter per se (there isn't one for the native UNIX mv command either) but if the files you want to move are named with a pattern you could use that pattern to select only those files to move. The wildcard operator (*) will come in handy in this situation as well.

[#3]

Except "." character and timestamp, all the files have the same name. So It's impossible to use pattern.

[#4 · Accepted solution]

you can try below script:

for f in $(hdfs dfs -ls /tmp/files | awk '$1 !~ /^d/ && $5 == "0" { print $8 }'); do hdfs dfs -mv "$f" /tmp/files/exclude-files; done

[#5]

Thanks a lot Pavan., I've just modified ... $5 == "0" ... by ... $5 != "0" ... because I don't want to move files with "0"size.

for f in $(hdfs dfs -ls /tmp/files | awk '$1 !~ /^d/ && $5 != "0" { print $8 }');do hdfs dfs -mv "$f"/tmp/files/exclude-files;done