Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

What triggers du in Impala?

Solved Go to solution
Highlighted

What triggers du in Impala?

New Contributor

Hi there,

 

We are doing some testing on a very small cluster and we were experiencing some extra load by du command. It is affecting our testing results significantly and we are bypassing it by creating a symbolic link of du to a df command. 

 

Our testing steps:

1. on all nodes: echo 1 > /proc/sys/vm/drop_caches

2. run scripts

 

Can anyone has a detailed explaination of how the du command gets triggered by impala( we assume it's something related with vfs caching). Is there a config or a better way to make it not doing du?

 

Thanks a lot! Let me know if you need more information. :)

 

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: What triggers du in Impala?

Are you sure it's Impala that's triggering it? I don't think Impala would use du for anything.

 

HDFS apparently does and Cloudera Manager might use it.

 

Have you tried tracing back what is running 'du'? E.g. run "ps auxf" to get a tree-view of processes.

View solution in original post

3 REPLIES 3
Highlighted

Re: What triggers du in Impala?

New Contributor
Also we didn't have fs.du.interval setting in our config so by default it should be 600000 ms but we are seeing it much more often than that.
Highlighted

Re: What triggers du in Impala?

Are you sure it's Impala that's triggering it? I don't think Impala would use du for anything.

 

HDFS apparently does and Cloudera Manager might use it.

 

Have you tried tracing back what is running 'du'? E.g. run "ps auxf" to get a tree-view of processes.

View solution in original post

Re: What triggers du in Impala?

New Contributor
Actually it is datanode doing it. I guess I'll ask more about it as an HDFS topic. Thanks!
Don't have an account?
Coming from Hortonworks? Activate your account here