question Re: hadoop -du retrun du: java.util.ConcurrentModificationException in Archives of Support Questions (Read Only)

hadoop -du retrun du: java.util.ConcurrentModificationException

AlonEldi — Mon, 12 Jun 2017 10:54:31 GMT

Hi ,

I started to get today this message when trying to check folder size in HDFS .

hadoop fs -du -s -h /data/
du: java.util.ConcurrentModificationException

using cloudera 5.11 .

The only thing that happen yesterday was deleting huge ammount of data from the HDFS .

Note - I'm using hadoop snapshots to backup the HDFS may it be related ?

Did not happen on CDH 5.4.3 with snapshots as well .

Any help will be appreciated.

Thanks

Alon

Re: hadoop -du retrun du: java.util.ConcurrentModificationException

Borg — Tue, 13 Jun 2017 19:13:50 GMT

Hey Alon,

You are correct in assuming that the issue is related to the snapshots, specifically in /data. This is a bug that will be resolved in the coming 5.11.1 maintenance release.

It was originally identified by HDFS-10797, there was an attempt to fix it in HDFS-11515, however in the meantime HDFS-11661 revealed that the original fix introduced some memory pressure that can get pretty high in case the filesystem has tens or hundreds of millions of file while du is running.

To resolve you can try to find the error ./snapshot files and delete, however this may prove to be difficult. I would suggest that you install the maintenance release that shouldbe available in the coming weeks (*not guaranteed as we do no give specific timelines to maintenance releases dates*) . You could also use an API call like (http://HOST.DOMAIN.COM:7180/api/v16/clusters/Cluster%201/services/HDFS-1/reports/hdfsUsageReport) to pull the statistics or view in the HDFS usage reports.

Re: hadoop -du retrun du: java.util.ConcurrentModificationException

AlonEldi — Wed, 14 Jun 2017 07:24:03 GMT

Thanks Borg ,

It happaned after deleteing huge ammount of data from the HDFS .

To reslove for the meatim I deleted all snapshots and created new one .

Its working OK .

Waiting for 5.11.1 🙂

Many Thanks

Alon