02-10-2019 07:29 PM
How do I change data (logs) retention policy for all Cloudera Management Services like Service Monitor, Host Monitor and Event Server. My target is to retain data (logs) only last 7 days.
Cloudera Enterprise 5.11.1
02-11-2019 05:31 AM
With "firehose_time_series_storage_bytes" parameter in Cloudera Manager.
We can control the approximate amount of disk space dedicated to storing time series and health data. Once the store has reached its maximum size, older data is deleted to make room for newer data. The disk usage is approximate because data is deleted only when the limit is reached.
But configuring the log retention based on time seems unlikely. However, you can write a shell script to remove the data every 7 days from the "Service Monitor Storage Directory".
By default, the data is stored in /var/lib/cloudera-service-monitor/ on the Service Monitor host. You can change this by modifying the Service Monitor Storage Directory configuration (firehose.storage.base.directory). But this step is not recommended by Cloudera.
02-11-2019 06:49 AM
02-11-2019 07:30 PM
Thank you for your reply, since there is no built-in mechanism to delete logs after 7 days I guess the best option for us, would be to adjust "firehose_time_series_storage_bytes" property. have some questions around the same here:
02-12-2019 12:54 AM
To answer your questions:
Please note that this is all about time series data which will be used to populate the charts in Cloudera Manager with data. This will not affect the log file size for the roles, e.g. in /var/log/cloudera-scm-firehose/ directory, as you mention /logs again in your recent post.
02-12-2019 09:55 PM
Thanks for your reply,my issue is that health status of Host Monitor Health, Event Server Health & Alert Publisher Health shows as Bad (Red) due to directory free space issue, is there anything I can check or change to fix this ? Will deleting the logs from the location you have mentioned in the previous location help us in reducing the alerts and bring us in Green zone.
Appreciate any help in this regard.
02-13-2019 12:30 AM
02-13-2019 07:39 PM - edited 02-13-2019 09:53 PM
Following is the details requested, unfortunately I am unablle to run (du -hs /var/*) seems I do not have required permissions.
Host Monitor Error
Host Monitor Storage Directory Free Space:
This role's Host Monitor Storage Directory is on a filesystem with less than 5.0 GiB of its space free. /var/lib/cloudera-host-monitor (free: 3.2 GiB (16.09%), capacity: 20.0 GiB)
Event Server Error
Log Directory Free Space :
This role's Log Directory is on a filesystem with less than 5.0 GiB of its space free. /var/log/cloudera-scm-eventserver (free: 4.9 GiB (35.25%), capacity: 14.0 GiB)
Alert Publisher Error
Log Directory Free Space :
This role's Log Directory is on a filesystem with less than 10.0 GiB of its space free. /var/log/cloudera-scm-alertpublisher (free: 5.0 GiB (35.81%), capacity: 14.0 GiB
02-13-2019 09:17 PM
As per the information provided, you have mentioned free space available.
16.09%(free: 3.2 GiB) of free space in /var/lib/cloudera-host-monitor.
35.25%(free: 4.9 GiB) of free space in /var/log/cloudera-scm-eventserver.
35.81%(free: 5.0 GiB) of free space in /var/log/cloudera-scm-alertpublisher.
Thus explaining the alert for low disk space.
The data in "/var/lib/cloudera-[host|service]-monitor" is the sum total of the working data for these respective services. Time-series metrics and health data - Time-Series Storage (firehose_time_series_storage_bytes - 10 GB default, 10 GB minimum)
1.) Change the default directory("/var/lib/cloudera-[host|service]-monitor") to some other location in your environment with enough space.
>> Stop the Service(Service Monitor or Host Monitor).
>> Save your old data and then copy the current directory to the new directory(optional)(Only if you need the old data).
>> Update the Storage Directory configuration option (firehose.storage.base.directory) on the corresponding role configuration page.
>> Start the Service Monitor or Host Monitor.
2.) If the data available in "/var/lib/cloudera-host-monitor" is not of much importance you can remove the data manually. But it's not a recommended step.
Your Health statuses will be Unknown or Bad for a short time and you will lose all Charts in the UI while the timeseries store is rebuilt and repopulated (due to the fact that you are deleting ALL the historical metrics). But this shouldn't have an impact on any service.
3.) Either add more disk to the cluster or remove unused/unnecessary files available on the disk to free up some disk space.