Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Can 'hdfs dfsadmin' and 'hdfs dsfs -du' be taxing on my cluster if I run them every 5 minutes?

Can 'hdfs dfsadmin' and 'hdfs dsfs -du' be taxing on my cluster if I run them every 5 minutes?

New Contributor

I have set up a job that runs several hadoop commands including 'hdfs dfsadmin' and 'hdfs dsfs -du' and I was wondering if these could be taxing on my cluster at all if I run them every 5 minutes or would it be harmless? The reason I am running several of these commands every 5 minutes is I have set up a job to run these commands, parse out the output to a structured format for a hive table so we can create historical reports about our system

1 REPLY 1

Re: Can 'hdfs dfsadmin' and 'hdfs dsfs -du' be taxing on my cluster if I run them every 5 minutes?

@Chad Woodhedad, yes 'fs -du` is expensive compared to other read operations. Running it every 5 minutes is probably overkill. You can run it less frequently e.g. once an hour.

`hdfs dfsadmin -report` is also expensive compared to typical read operations.

We've occasionally seen these calls affect NameNode performance when buggy monitoring scripts invoke them many times per second. Barring that, you should be fine.

Don't have an account?
Coming from Hortonworks? Activate your account here