Support Questions
Find answers, ask questions, and share your expertise

how to exclude .* files while executing "hdfs dfs -mv" command ?

Hello,

I would like to move lot of files of a hdfs directory but not the files with size at 0 and name like ".*"

For example, move only files "file3" "file4" file5" but not the files "file1" and "file2". These ones have'ont been not entirely written in hdfs directory when I execute the "hdfs dfs -mv" command.

hdfs@host:~> hadoop dfs -ls /mydirectory

Found 1942 items

-rw-r----- 3 xagcla02 hdfs 0 2017-10-19 18:07 /mydirectory/.file1

-rw-r----- 3 xagcla02 hdfs 0 2017-10-19 18:07 /mydirectory/.file2

-rw-r----- 3 xagcla02 hdfs 2540 2017-10-19 18:07 /mydirectory/file3

-rw-r----- 3 xagcla02 hdfs 2540 2017-10-19 18:07 /mydirectory/file4

-rw-r----- 3 xagcla02 hdfs 5252 2017-10-19 18:07 /mydirectory/file5

Thanks for your feedbacks

1 ACCEPTED SOLUTION

Cloudera Employee

you can try below script:

for f in $(hdfs dfs -ls /tmp/files | awk '$1 !~ /^d/ && $5 == "0" { print $8 }'); do hdfs dfs -mv "$f" /tmp/files/exclude-files; done

View solution in original post

4 REPLIES 4

Expert Contributor

There isn't an exclusion filter per se (there isn't one for the native UNIX mv command either) but if the files you want to move are named with a pattern you could use that pattern to select only those files to move. The wildcard operator (*) will come in handy in this situation as well.

Except "." character and timestamp, all the files have the same name. So It's impossible to use pattern.

Cloudera Employee

you can try below script:

for f in $(hdfs dfs -ls /tmp/files | awk '$1 !~ /^d/ && $5 == "0" { print $8 }'); do hdfs dfs -mv "$f" /tmp/files/exclude-files; done

Thanks a lot Pavan., I've just modified ... $5 == "0" ... by ... $5 != "0" ... because I don't want to move files with "0"size.

for f in $(hdfs dfs -ls /tmp/files | awk '$1 !~ /^d/ && $5 != "0" { print $8 }');do hdfs dfs -mv "$f"/tmp/files/exclude-files;done

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.