question Re: how to exclude .* files while executing "hdfs dfs -mv" command ? in Support Questions

how to exclude .* files while executing "hdfs dfs -mv" command ?

thierry_vernhet — Thu, 19 Oct 2017 23:14:40 GMT

Hello,

I would like to move lot of files of a hdfs directory but not the files with size at 0 and name like ".*"

For example, move only files "file3" "file4" file5" but not the files "file1" and "file2". These ones have'ont been not entirely written in hdfs directory when I execute the "hdfs dfs -mv" command.

hdfs@host:~> hadoop dfs -ls /mydirectory

Found 1942 items

-rw-r----- 3 xagcla02 hdfs 0 2017-10-19 18:07 /mydirectory/.file1

-rw-r----- 3 xagcla02 hdfs 0 2017-10-19 18:07 /mydirectory/.file2

-rw-r----- 3 xagcla02 hdfs 2540 2017-10-19 18:07 /mydirectory/file3

-rw-r----- 3 xagcla02 hdfs 2540 2017-10-19 18:07 /mydirectory/file4

-rw-r----- 3 xagcla02 hdfs 5252 2017-10-19 18:07 /mydirectory/file5

…

Thanks for your feedbacks

Re: how to exclude .* files while executing "hdfs dfs -mv" command ?

anarasimham — Sat, 21 Oct 2017 04:09:37 GMT

There isn't an exclusion filter per se (there isn't one for the native UNIX mv command either) but if the files you want to move are named with a pattern you could use that pattern to select only those files to move. The wildcard operator (*) will come in handy in this situation as well.

Re: how to exclude .* files while executing "hdfs dfs -mv" command ?

thierry_vernhet — Mon, 23 Oct 2017 13:40:56 GMT

Except "." character and timestamp, all the files have the same name. So It's impossible to use pattern.

Re: how to exclude .* files while executing "hdfs dfs -mv" command ?

mcapavan — Mon, 23 Oct 2017 17:56:48 GMT

you can try below script:

for f in $(hdfs dfs -ls /tmp/files | awk '$1 !~ /^d/ && $5 == "0" { print $8 }'); do hdfs dfs -mv "$f" /tmp/files/exclude-files; done

Re: how to exclude .* files while executing "hdfs dfs -mv" command ?

thierry_vernhet — Mon, 23 Oct 2017 18:35:49 GMT

Thanks a lot Pavan., I've just modified ... $5 == "0" ... by ... $5 != "0" ... because I don't want to move files with "0"size.

for f in $(hdfs dfs -ls /tmp/files | awk '$1 !~ /^d/ && $5 != "0" { print $8 }');do hdfs dfs -mv "$f"/tmp/files/exclude-files;done