Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Ambari hbase metrices filling up ambari-server space

avatar
Contributor

Hi All,

My Ambari root partition is running out of space due to Hbase ambari-metrices[PATH is /var/lib/ambari-metrics-collector/hbase/data/default]

Below folders are taking space.

METRIC_AGGREGATE

METRIC_RECORD_MINUTE

METRIC_RECORD_HOURLY

METRIC_RECORD

Can I delete older files directly here ?

What can be the other option ?

Can I add big volume and redirect logs to new volume. what all parameter do i need to change in UI / commandLine

OR

Can I shift from embedded mode to distributed mode. what all steps/precaution needed to shift to this mode.

Any help is appreciated. Thanks in advance.

1 ACCEPTED SOLUTION

avatar

@Harshal JoshiThis is a usual problem and what I would recommend is to have a larger volume mounted for this different than root partition i.e. /opt and then configure Ambari Metrics Service to use /opt i.e. change directories from /var/lib/ambari-metrics-collector to /opt/ambari-metrics-collector. This can be done easily from Ambari UI --> Ambari metrics --> Config .

In terms of deleting older files, delete those as soon as you have configured AMS directories to use different volume other than root i.e. /opt and restarted the service.

Other option is to use distributed mode following below instructions and then remove old files but still you will have AMS logs filling up root partition. So, you should anyways move to /opt for log and data directories for AMS.

http://docs.hortonworks.com/HDPDocuments/Ambari-2.1.0.0/bk_ambari_reference_guide/content/_configuri...

View solution in original post

6 REPLIES 6

avatar

avatar
Contributor

Hi @Scott Shaw Thanks for your quick reply.

avatar
Guru

hbase.rootdir in ambari-metrics through ambari UI has the local hbase base directory where metrics are stored. This by default is file:///var/lib/ambari-metrics-collector/hbase

avatar
Contributor

Hi @Ravi Mutyala Thanks for your reply.

avatar

@Harshal JoshiThis is a usual problem and what I would recommend is to have a larger volume mounted for this different than root partition i.e. /opt and then configure Ambari Metrics Service to use /opt i.e. change directories from /var/lib/ambari-metrics-collector to /opt/ambari-metrics-collector. This can be done easily from Ambari UI --> Ambari metrics --> Config .

In terms of deleting older files, delete those as soon as you have configured AMS directories to use different volume other than root i.e. /opt and restarted the service.

Other option is to use distributed mode following below instructions and then remove old files but still you will have AMS logs filling up root partition. So, you should anyways move to /opt for log and data directories for AMS.

http://docs.hortonworks.com/HDPDocuments/Ambari-2.1.0.0/bk_ambari_reference_guide/content/_configuri...

avatar
Contributor

Thanks @Pradeep kumar. I have implemented the solution of adding non root volume. Do i need to verify anything post change. Let me know.