Ambari hbase metrices filling up ambari-server space

[#1]

Hi All,

My Ambari root partition is running out of space due to Hbase ambari-metrices[PATH is /var/lib/ambari-metrics-collector/hbase/data/default]

Below folders are taking space.

METRIC_AGGREGATE

METRIC_RECORD_MINUTE

METRIC_RECORD_HOURLY

METRIC_RECORD

Can I delete older files directly here ?

What can be the other option ?

Can I add big volume and redirect logs to new volume. what all parameter do i need to change in UI / commandLine

OR

Can I shift from embedded mode to distributed mode. what all steps/precaution needed to shift to this mode.

Any help is appreciated. Thanks in advance.

[#2]

@Harshal Joshi You may want to check your Ambari Metric TTL settings and adjust accordingly. https://docs.hortonworks.com/HDPDocuments/Ambari-2.2.1.1/bk_ambari_reference_guide/content/ams_aggre....

Also, this issue was addressed in a previous post. https://community.hortonworks.com/questions/13419/how-can-we-delete-old-data-of-ambari-metric-server...

[#3]

Hi @Scott Shaw Thanks for your quick reply.

[#4]

hbase.rootdir in ambari-metrics through ambari UI has the local hbase base directory where metrics are stored. This by default is file:///var/lib/ambari-metrics-collector/hbase

[#5]

Hi @Ravi Mutyala Thanks for your reply.

[#6 · Accepted solution]

@Harshal JoshiThis is a usual problem and what I would recommend is to have a larger volume mounted for this different than root partition i.e. /opt and then configure Ambari Metrics Service to use /opt i.e. change directories from /var/lib/ambari-metrics-collector to /opt/ambari-metrics-collector. This can be done easily from Ambari UI --> Ambari metrics --> Config .

In terms of deleting older files, delete those as soon as you have configured AMS directories to use different volume other than root i.e. /opt and restarted the service.

Other option is to use distributed mode following below instructions and then remove old files but still you will have AMS logs filling up root partition. So, you should anyways move to /opt for log and data directories for AMS.

http://docs.hortonworks.com/HDPDocuments/Ambari-2.1.0.0/bk_ambari_reference_guide/content/_configuri...

[#7]

Thanks @Pradeep kumar. I have implemented the solution of adding non root volume. Do i need to verify anything post change. Let me know.