Support Questions

Find answers, ask questions, and share your expertise

how to exclude .* files while executing "hdfs dfs -mv" command ?

avatar
Rising Star

Hello,

I would like to move lot of files of a hdfs directory but not the files with size at 0 and name like ".*"

For example, move only files "file3" "file4" file5" but not the files "file1" and "file2". These ones have'ont been not entirely written in hdfs directory when I execute the "hdfs dfs -mv" command.

hdfs@host:~> hadoop dfs -ls /mydirectory

Found 1942 items

-rw-r----- 3 xagcla02 hdfs 0 2017-10-19 18:07 /mydirectory/.file1

-rw-r----- 3 xagcla02 hdfs 0 2017-10-19 18:07 /mydirectory/.file2

-rw-r----- 3 xagcla02 hdfs 2540 2017-10-19 18:07 /mydirectory/file3

-rw-r----- 3 xagcla02 hdfs 2540 2017-10-19 18:07 /mydirectory/file4

-rw-r----- 3 xagcla02 hdfs 5252 2017-10-19 18:07 /mydirectory/file5

Thanks for your feedbacks

1 ACCEPTED SOLUTION

avatar
Contributor

you can try below script:

for f in $(hdfs dfs -ls /tmp/files | awk '$1 !~ /^d/ && $5 == "0" { print $8 }'); do hdfs dfs -mv "$f" /tmp/files/exclude-files; done

View solution in original post

4 REPLIES 4

avatar
Super Collaborator

There isn't an exclusion filter per se (there isn't one for the native UNIX mv command either) but if the files you want to move are named with a pattern you could use that pattern to select only those files to move. The wildcard operator (*) will come in handy in this situation as well.

avatar
Rising Star

Except "." character and timestamp, all the files have the same name. So It's impossible to use pattern.

avatar
Contributor

you can try below script:

for f in $(hdfs dfs -ls /tmp/files | awk '$1 !~ /^d/ && $5 == "0" { print $8 }'); do hdfs dfs -mv "$f" /tmp/files/exclude-files; done

avatar
Rising Star

Thanks a lot Pavan., I've just modified ... $5 == "0" ... by ... $5 != "0" ... because I don't want to move files with "0"size.

for f in $(hdfs dfs -ls /tmp/files | awk '$1 !~ /^d/ && $5 != "0" { print $8 }');do hdfs dfs -mv "$f"/tmp/files/exclude-files;done