Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

how to exclude .* files while executing "hdfs dfs -mv" command ?

Solved Go to solution

how to exclude .* files while executing "hdfs dfs -mv" command ?

Hello,

I would like to move lot of files of a hdfs directory but not the files with size at 0 and name like ".*"

For example, move only files "file3" "file4" file5" but not the files "file1" and "file2". These ones have'ont been not entirely written in hdfs directory when I execute the "hdfs dfs -mv" command.

hdfs@host:~> hadoop dfs -ls /mydirectory

Found 1942 items

-rw-r----- 3 xagcla02 hdfs 0 2017-10-19 18:07 /mydirectory/.file1

-rw-r----- 3 xagcla02 hdfs 0 2017-10-19 18:07 /mydirectory/.file2

-rw-r----- 3 xagcla02 hdfs 2540 2017-10-19 18:07 /mydirectory/file3

-rw-r----- 3 xagcla02 hdfs 2540 2017-10-19 18:07 /mydirectory/file4

-rw-r----- 3 xagcla02 hdfs 5252 2017-10-19 18:07 /mydirectory/file5

Thanks for your feedbacks

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: how to exclude .* files while executing "hdfs dfs -mv" command ?

Cloudera Employee

you can try below script:

for f in $(hdfs dfs -ls /tmp/files | awk '$1 !~ /^d/ && $5 == "0" { print $8 }'); do hdfs dfs -mv "$f" /tmp/files/exclude-files; done

View solution in original post

4 REPLIES 4
Highlighted

Re: how to exclude .* files while executing "hdfs dfs -mv" command ?

Expert Contributor

There isn't an exclusion filter per se (there isn't one for the native UNIX mv command either) but if the files you want to move are named with a pattern you could use that pattern to select only those files to move. The wildcard operator (*) will come in handy in this situation as well.

Highlighted

Re: how to exclude .* files while executing "hdfs dfs -mv" command ?

Except "." character and timestamp, all the files have the same name. So It's impossible to use pattern.

Highlighted

Re: how to exclude .* files while executing "hdfs dfs -mv" command ?

Cloudera Employee

you can try below script:

for f in $(hdfs dfs -ls /tmp/files | awk '$1 !~ /^d/ && $5 == "0" { print $8 }'); do hdfs dfs -mv "$f" /tmp/files/exclude-files; done

View solution in original post

Highlighted

Re: how to exclude .* files while executing "hdfs dfs -mv" command ?

Thanks a lot Pavan., I've just modified ... $5 == "0" ... by ... $5 != "0" ... because I don't want to move files with "0"size.

for f in $(hdfs dfs -ls /tmp/files | awk '$1 !~ /^d/ && $5 != "0" { print $8 }');do hdfs dfs -mv "$f"/tmp/files/exclude-files;done

Don't have an account?
Coming from Hortonworks? Activate your account here