Cloudera Manager Space Constraints

[#1]

Hello All,

We are facing issue with one of our clients cluster, we see some files over 2GB, and would like to know the purpose of these files, and if these could be deleted to make way for some space.

1. Eventserver
[root@TestBed ~]# du -sh /var/lib/cloudera-scm-eventserver/v3/* |grep G
5.4G /var/lib/cloudera-scm-eventserver/v3/_1bx6l.fdt
1.2G /var/lib/cloudera-scm-eventserver/v3/_1hgcy.fdt
1.4G /var/lib/cloudera-scm-eventserver/v3/_1mz17.fdt

2. Cloudera-Scm-Headlamp
[root@TestBed ~]# du -sh /var/lib/cloudera-scm-headlamp/hdfs/nameservice1/index/* |grep G
2.3G /var/lib/cloudera-scm-headlamp/hdfs/nameservice1/index/_2t.fdt
2.2G /var/lib/cloudera-scm-headlamp/hdfs/nameservice1/index/_5m.fdt
2.1G /var/lib/cloudera-scm-headlamp/hdfs/nameservice1/index/_8f.fdt

Also is there a way to limit TS data in SMON & HMON, I understand that 10GB is the minimum requirement for these roles but what could be done if a clients has limited resources.

Any help / guidance is appreciated

Thanks

[#2 · Accepted solution]

Hi @Amn_468 ,

Please check this documentation on how to plan storage space for Cloudera Management Services:

https://docs.cloudera.com/documentation/enterprise/latest/topics/cm_ig_reqs_space.html

And documentation on SMON and HMON:

https://docs.cloudera.com/documentation/enterprise/latest/topics/cm_ig_storage.html#data_storage

The minimum size is a sum of 12GB for /var/lib/cloudera-service-monitor/ and an additional 10GB for /var/lib/cloudera-host-monitor/, it's not supported to set up less space than that. A metrics storage below 10 GB is not tested, and could cause issues with monitoring and alerts.

Here are some details information:

1. Eventserver (/var/lib/cloudera-scm-eventserver):
The amount of storage space the Event Server uses is influenced by configuring how many discrete events it can retain.

To configure data retention in Cloudera Manager Administration Console,
Go the Cloudera Management Service.
Click the Configuration tab.
Select Scope > Event Server or Cloudera Management Service (Service-Wide).
Select Category > Main.
Edit the following property:
Maximum Number of Events in the Event Server Store
The maximum size of the Event Server store, in events. When this size is exceeded, events are deleted starting with the oldest first until the size of the store is below this threshold

2. Reports Manager (/var/lib/cloudera-scm-headlamp):
Reports Manager downloads the fsimage from the NameNode (every 60 minutes by default) and stores it locally to perform operations against, including indexing the HDFS filesystem structure. More files and directories results in a larger fsimage, which consumes more disk space.

Reports Manager has no control over the size of the fsimage. If your total HDFS usage trends upward notably or you add excessively long paths in HDFS, it might be necessary to revisit and adjust the amount of local storage allocated to the Reports Manager. Periodically monitor, review, and adjust the local storage allocation.

3. Service Monitor:
10GiB Services Time Series Storage +
1GiB Impala Query Storage +
1GiB YARN Application Storage
===================
~12GiB Minimum / No Maximum

The amount of storage space the Service Monitor uses is influenced by configuring how many hours|days worth of data it is configured retain.

Find these properties at Cloudera Manager UI Home > Cloudera Management Service > Configuration > Scope > Service Monitor:

- Time-Series Storage (firehose_time_series_storage_bytes)
- Impala Storage (firehose_impala_storage_bytes)
- YARN Storage (firehose_yarn_storage_bytes)

4. Host Monitor:
Host Monitor data growth is controlled by configuring a maximum amount of storage space it may use.
Find these properties at Cloudera Manager UI Home > Cloudera Management Service > Configuration > Scope > Host Monitor:
Time-Series Storage (firehose_time_series_storage_bytes)
(Default + Minimum: 10GiB Host Time Series Storage)

Thanks and hope this helps,

Li

[#3]

Hi Li,

Thanks for all your help and assistance 🙂