Method for recovering from a full HDD due to Ambari Metrics Collector?

[#1]

I recently setup HDP (HBase) on a single VM which had ~15GB of space. The installation went fine but after ~2 months the system ran out of HDD space. I'd like to come up with a method for clearing out the metrics or truncating them. While researching this I drilled down to this directory where the bulk of the space is being used:

$ du -sh /var/lib/ambari-metrics-collector/hbase/data/default/*  | sort -rh | head -5
7.1G   /var/lib/ambari-metrics-collector/hbase/data/default/METRIC_AGGREGATE
403M   /var/lib/ambari-metrics-collector/hbase/data/default/METRIC_RECORD_MINUTE
209M   /var/lib/ambari-metrics-collector/hbase/data/default/METRIC_RECORD
76M    /var/lib/ambari-metrics-collector/hbase/data/default/METRIC_AGGREGATE_HOURLY
45M    /var/lib/ambari-metrics-collector/hbase/data/default/METRIC_RECORD_HOURLY

I've toyed with several methods of truncating these files using the `truncate -s 0 <file>` command but this trashes the files so that they're no longer usable by AMS.

Questions

• Is there a simple way to reset the metrics?
• Is there a safe way to delete the data collected periodically, from say a cron job?

NOTE: This is a small installation and I don't have the ability to throw more HDD space at the problem. I'd like to keep AMS enabled if possible.

[#2 · Accepted solution]

@Sam Mingolelli Which version of Amabri are you using? Ambari 2.1 does allow you to truncate. It would be easier to remove AMS and reinstall. We also recommend a dedicated minimum of 10 GB for AMS. See: https://cwiki.apache.org/confluence/display/AMBAR...

You may also want to edit your TTL settings - https://cwiki.apache.org/confluence/display/AMBARI/Known+Issues and here - https://cwiki.apache.org/confluence/display/AMBARI/Configuration

[#3]

@Scott Shaw - I'm using 2.0.1 of Ambari. I never thought of that, so I can remove AMS and then re-install it to get it to recreate it when I happen upon the out of HDD space issue? I'll look thru the links to see how to dial down the TTLs for AMS. Thanks for the info!

[#4]

BTW - this is a single node of HortonWorks so it seems odd that it would require so much space? I'm going w/ the default options when I do a server install too.

[#5]

@Sam Mingolelli Agree, but its only because those are the set minimum TTL requirements. Change the TTL values and you should be able to get by with less space. It took 2 months to fill up for you. I've seen it fill up in days with multi-node clusters.

[#6]

@Sam Mingolelli has this been addressed? Please accept the best answer or provide your own solution.