Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Cloudera Manager Space Constraints

avatar
Rising Star

Hello All,

We are facing issue with one of our clients cluster, we see some files over 2GB, and would like to know the purpose of these files, and if these could be deleted to make way for some space.

1. Eventserver
[root@TestBed ~]# du -sh /var/lib/cloudera-scm-eventserver/v3/* |grep G
5.4G /var/lib/cloudera-scm-eventserver/v3/_1bx6l.fdt
1.2G /var/lib/cloudera-scm-eventserver/v3/_1hgcy.fdt
1.4G /var/lib/cloudera-scm-eventserver/v3/_1mz17.fdt

 

2. Cloudera-Scm-Headlamp
[root@TestBed ~]# du -sh /var/lib/cloudera-scm-headlamp/hdfs/nameservice1/index/* |grep G
2.3G /var/lib/cloudera-scm-headlamp/hdfs/nameservice1/index/_2t.fdt
2.2G /var/lib/cloudera-scm-headlamp/hdfs/nameservice1/index/_5m.fdt
2.1G /var/lib/cloudera-scm-headlamp/hdfs/nameservice1/index/_8f.fdt

 

Also is there a way to limit TS data in SMON & HMON, I understand that 10GB is the minimum requirement for these roles but what could be done if a clients has limited resources.

 

Any help / guidance is appreciated

 

Thanks

1 ACCEPTED SOLUTION

avatar
Guru

Hi @Amn_468 ,

 

Please check this documentation on how to plan storage space for Cloudera Management Services:

https://docs.cloudera.com/documentation/enterprise/latest/topics/cm_ig_reqs_space.html

 

And documentation on SMON and HMON:

https://docs.cloudera.com/documentation/enterprise/latest/topics/cm_ig_storage.html#data_storage

The minimum size is a sum of 12GB for /var/lib/cloudera-service-monitor/ and an additional 10GB for /var/lib/cloudera-host-monitor/, it's not supported to set up less space than that. A metrics storage below 10 GB is not tested, and could cause issues with monitoring and alerts.

 

Here are some details information:

1. Eventserver (/var/lib/cloudera-scm-eventserver):
The amount of storage space the Event Server uses is influenced by configuring how many discrete events it can retain.

To configure data retention in Cloudera Manager Administration Console,
Go the Cloudera Management Service.
Click the Configuration tab.
Select Scope > Event Server or Cloudera Management Service (Service-Wide).
Select Category > Main.
Edit the following property:
Maximum Number of Events in the Event Server Store
The maximum size of the Event Server store, in events. When this size is exceeded, events are deleted starting with the oldest first until the size of the store is below this threshold

 

2. Reports Manager (/var/lib/cloudera-scm-headlamp):
Reports Manager downloads the fsimage from the NameNode (every 60 minutes by default) and stores it locally to perform operations against, including indexing the HDFS filesystem structure. More files and directories results in a larger fsimage, which consumes more disk space.

Reports Manager has no control over the size of the fsimage. If your total HDFS usage trends upward notably or you add excessively long paths in HDFS, it might be necessary to revisit and adjust the amount of local storage allocated to the Reports Manager. Periodically monitor, review, and adjust the local storage allocation.


3. Service Monitor:
10GiB Services Time Series Storage +
1GiB Impala Query Storage +
1GiB YARN Application Storage
===================
~12GiB Minimum / No Maximum

The amount of storage space the Service Monitor uses is influenced by configuring how many hours|days worth of data it is configured retain.

Find these properties at Cloudera Manager UI Home > Cloudera Management Service > Configuration > Scope > Service Monitor:

- Time-Series Storage (firehose_time_series_storage_bytes)
- Impala Storage (firehose_impala_storage_bytes)
- YARN Storage (firehose_yarn_storage_bytes)

4. Host Monitor:
Host Monitor data growth is controlled by configuring a maximum amount of storage space it may use.
Find these properties at Cloudera Manager UI Home > Cloudera Management Service > Configuration > Scope > Host Monitor:
Time-Series Storage (firehose_time_series_storage_bytes)
(Default + Minimum: 10GiB Host Time Series Storage)

 

Thanks and hope this helps,

Li

Li Wang, Technical Solution Manager


Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Learn more about the Cloudera Community:

Terms of Service

Community Guidelines

How to use the forum

View solution in original post

2 REPLIES 2

avatar
Guru

Hi @Amn_468 ,

 

Please check this documentation on how to plan storage space for Cloudera Management Services:

https://docs.cloudera.com/documentation/enterprise/latest/topics/cm_ig_reqs_space.html

 

And documentation on SMON and HMON:

https://docs.cloudera.com/documentation/enterprise/latest/topics/cm_ig_storage.html#data_storage

The minimum size is a sum of 12GB for /var/lib/cloudera-service-monitor/ and an additional 10GB for /var/lib/cloudera-host-monitor/, it's not supported to set up less space than that. A metrics storage below 10 GB is not tested, and could cause issues with monitoring and alerts.

 

Here are some details information:

1. Eventserver (/var/lib/cloudera-scm-eventserver):
The amount of storage space the Event Server uses is influenced by configuring how many discrete events it can retain.

To configure data retention in Cloudera Manager Administration Console,
Go the Cloudera Management Service.
Click the Configuration tab.
Select Scope > Event Server or Cloudera Management Service (Service-Wide).
Select Category > Main.
Edit the following property:
Maximum Number of Events in the Event Server Store
The maximum size of the Event Server store, in events. When this size is exceeded, events are deleted starting with the oldest first until the size of the store is below this threshold

 

2. Reports Manager (/var/lib/cloudera-scm-headlamp):
Reports Manager downloads the fsimage from the NameNode (every 60 minutes by default) and stores it locally to perform operations against, including indexing the HDFS filesystem structure. More files and directories results in a larger fsimage, which consumes more disk space.

Reports Manager has no control over the size of the fsimage. If your total HDFS usage trends upward notably or you add excessively long paths in HDFS, it might be necessary to revisit and adjust the amount of local storage allocated to the Reports Manager. Periodically monitor, review, and adjust the local storage allocation.


3. Service Monitor:
10GiB Services Time Series Storage +
1GiB Impala Query Storage +
1GiB YARN Application Storage
===================
~12GiB Minimum / No Maximum

The amount of storage space the Service Monitor uses is influenced by configuring how many hours|days worth of data it is configured retain.

Find these properties at Cloudera Manager UI Home > Cloudera Management Service > Configuration > Scope > Service Monitor:

- Time-Series Storage (firehose_time_series_storage_bytes)
- Impala Storage (firehose_impala_storage_bytes)
- YARN Storage (firehose_yarn_storage_bytes)

4. Host Monitor:
Host Monitor data growth is controlled by configuring a maximum amount of storage space it may use.
Find these properties at Cloudera Manager UI Home > Cloudera Management Service > Configuration > Scope > Host Monitor:
Time-Series Storage (firehose_time_series_storage_bytes)
(Default + Minimum: 10GiB Host Time Series Storage)

 

Thanks and hope this helps,

Li

Li Wang, Technical Solution Manager


Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Learn more about the Cloudera Community:

Terms of Service

Community Guidelines

How to use the forum

avatar
Rising Star

Hi Li,

 

Thanks for all your help and assistance 🙂