question Re: Can I delete multiple files from hdfs? in Support Questions

Can I delete multiple files from hdfs?

madhura2688 — Fri, 22 Mar 2019 08:56:22 GMT

Currently we are seeing our HDFS DFS directory is getting filled up and we have to remove the data at faster rate.

We currently have 12 datanodes and 4 masternodes 1 edgenode. Can I delete the files from HDFS from masternodes and edgenodes at once? I have created a script on edgenode which deletes the HDFS files but speed is really slow.

How can I delete multiple files at a time ? Can I place that script on multiple server and delete the files?

Re: Can I delete multiple files from hdfs?

lomacka_oleg — Sat, 23 Mar 2019 07:55:03 GMT

Hello, Madhura Mhatre!

You can try to use tHDFSList to iterate each file that you want to delete.

For example:

tHDFSList--iterate--tHDFSDelete

There is a global variable that stores the current file path on tHDFSList:

(String)globalMap.get("tHDFSList_1_CURRENT_FILEPATH")

Set the file path of tHDFSdDelete with this variable.

Also please check this article on StackOverflow

This video

I hope it helps!

Re: Can I delete multiple files from hdfs?

ppatel — Wed, 12 Feb 2020 21:18:38 GMT

Hi,

I have similar scenario as you had. Did you run script to all node to delete files parallel with different parameters ?
I have created script that first collect all file name and add into array then loop through deleting those files, since it is deleting one by one is there any way to delete all at once ?

Re: Can I delete multiple files from hdfs?

ask_bill_brooks — Thu, 13 Feb 2020 23:37:43 GMT

Hi @ppatel,

As this thread is older and was marked 'Solved' in March of 2019 you would have a better chance of receiving a resolution by starting a new thread. This will also provide the opportunity to provide details about your script that could aid others in providing a more relevant answer to your question.