question Re: Rate of growth HDFS Filesystem in Archives of Support Questions (Read Only)

Rate of growth HDFS Filesystem

kums — Fri, 31 Mar 2017 18:43:25 GMT

I need to know the rate of growth of my HDFS file system for past 6 months. I did browse Ambari and I do not see in Name Node UI too.

Is there any other UI I need to check ?

Or Is there any hdfs admin commnds to get this information ?

Re: Rate of growth HDFS Filesystem

balavignesh_nag — Fri, 31 Mar 2017 20:32:59 GMT

@Kumar Veerapan Im not sure whether I got your question right. But If you are asking for UI to check the rate of file growth then do check this https://www.datadoghq.com/product/

Also grafana https://grafana.com/grafana

Helped me a lot in terms of time series metrics.

Re: Rate of growth HDFS Filesystem

jsensharma — Fri, 31 Mar 2017 20:51:06 GMT

@Kumar Veerappan

Ambari provides some inbuilt alerts to findout the Weekly/Daily growth in the HDFS usage.

Ambari UI --> Alerts (Tab) --> "Alert Definition Filter" Search for "HDFS Storage Capacity Usage"

This service-level alert is triggered if the increase in storage capacity usage deviation has grown beyond the specified threshold within a given period. This alert will monitor Daily and Weekly periods.

Please see: https://docs.hortonworks.com/HDPDocuments/Ambari-2.4.2.0/bk_ambari-user-guide/content/hdfs_service_alerts.html

However if you want to get this data for 6 months then you might have to write your own custom alert script.

\Sometime back i have written a basic example of how we can have our own custom ambari alert. https://community.hortonworks.com/articles/38149/how-to-create-and-register-custom-ambari-alerts.html

Grafana basically fetches data from AMS (Ambari Metrics Collector) using APIs so the data need to be available on AMS first.