question Method for recovering from a full HDD due to Ambari Metrics Collector? in Archives of Support Questions (Read Only)

Method for recovering from a full HDD due to Ambari Metrics Collector?

slm — Tue, 29 Dec 2015 00:41:17 GMT

I recently setup HDP (HBase) on a single VM which had ~15GB of space. The installation went fine but after ~2 months the system ran out of HDD space. I'd like to come up with a method for clearing out the metrics or truncating them. While researching this I drilled down to this directory where the bulk of the space is being used:

$ du -sh /var/lib/ambari-metrics-collector/hbase/data/default/*  | sort -rh | head -5
7.1G   /var/lib/ambari-metrics-collector/hbase/data/default/METRIC_AGGREGATE
403M   /var/lib/ambari-metrics-collector/hbase/data/default/METRIC_RECORD_MINUTE
209M   /var/lib/ambari-metrics-collector/hbase/data/default/METRIC_RECORD
76M    /var/lib/ambari-metrics-collector/hbase/data/default/METRIC_AGGREGATE_HOURLY
45M    /var/lib/ambari-metrics-collector/hbase/data/default/METRIC_RECORD_HOURLY

I've toyed with several methods of truncating these files using the `truncate -s 0 <file>` command but this trashes the files so that they're no longer usable by AMS.

Questions

Is there a simple way to reset the metrics?
Is there a safe way to delete the data collected periodically, from say a cron job?

NOTE: This is a small installation and I don't have the ability to throw more HDD space at the problem. I'd like to keep AMS enabled if possible.

Re: Method for recovering from a full HDD due to Ambari Metrics Collector?

SQLShaw — Tue, 29 Dec 2015 02:53:32 GMT

@Sam Mingolelli Which version of Amabri are you using? Ambari 2.1 does allow you to truncate. It would be easier to remove AMS and reinstall. We also recommend a dedicated minimum of 10 GB for AMS. See: https://cwiki.apache.org/confluence/display/AMBAR...

You may also want to edit your TTL settings - https://cwiki.apache.org/confluence/display/AMBARI/Known+Issues and here - https://cwiki.apache.org/confluence/display/AMBARI/Configuration

Re: Method for recovering from a full HDD due to Ambari Metrics Collector?

slm — Tue, 29 Dec 2015 03:36:47 GMT

@Scott Shaw - I'm using 2.0.1 of Ambari. I never thought of that, so I can remove AMS and then re-install it to get it to recreate it when I happen upon the out of HDD space issue? I'll look thru the links to see how to dial down the TTLs for AMS. Thanks for the info!

Re: Method for recovering from a full HDD due to Ambari Metrics Collector?

slm — Tue, 29 Dec 2015 03:39:26 GMT

BTW - this is a single node of HortonWorks so it seems odd that it would require so much space? I'm going w/ the default options when I do a server install too.

Re: Method for recovering from a full HDD due to Ambari Metrics Collector?

SQLShaw — Tue, 29 Dec 2015 05:14:48 GMT

@Sam Mingolelli Agree, but its only because those are the set minimum TTL requirements. Change the TTL values and you should be able to get by with less space. It took 2 months to fill up for you. I've seen it fill up in days with multi-node clusters.

Re: Method for recovering from a full HDD due to Ambari Metrics Collector?

aervits — Sat, 06 Feb 2016 04:15:35 GMT

@Sam Mingolelli has this been addressed? Please accept the best answer or provide your own solution.