question What triggers du in Impala? in Archives of Support Questions (Read Only)

What triggers du in Impala?

ZL — Fri, 16 Sep 2022 10:15:13 GMT

Hi there,

We are doing some testing on a very small cluster and we were experiencing some extra load by du command. It is affecting our testing results significantly and we are bypassing it by creating a symbolic link of du to a df command.

Our testing steps:

1. on all nodes: echo 1 > /proc/sys/vm/drop_caches

2. run scripts

Can anyone has a detailed explaination of how the du command gets triggered by impala( we assume it's something related with vfs caching). Is there a config or a better way to make it not doing du?

Thanks a lot! Let me know if you need more information. 🙂

Re: What triggers du in Impala?

ZL — Thu, 21 Apr 2016 23:10:09 GMT

Also we didn't have fs.du.interval setting in our config so by default it should be 600000 ms but we are seeing it much more often than that.

Re: What triggers du in Impala?

Tim Armstrong — Fri, 22 Apr 2016 16:21:43 GMT

Are you sure it's Impala that's triggering it? I don't think Impala would use du for anything.

HDFS apparently does and Cloudera Manager might use it.

Have you tried tracing back what is running 'du'? E.g. run "ps auxf" to get a tree-view of processes.

Re: What triggers du in Impala?

ZL — Fri, 22 Apr 2016 16:40:55 GMT

Actually it is datanode doing it. I guess I'll ask more about it as an HDFS topic. Thanks!