Cleaning up CM logs

[#1]

My disks are filling up from files in /var/lib/cloudera-service-monitor/ts

1) What it safe to delete and is this done manually or is there a CM way to clean up?

2) What CM config controlls the amount of data produced here?

3) What does "ts" stand for?

[#2]

"ts" stands for Time Series. This is where the Service Monitor role in the Management Service stores data for Time Series information. You can configure retention policies in the configuraiton of your Service Monitor role. There's also a page to help you understand the disk usage in detail, as described below:

Time-Series Storage  firehose_time_series_storage_bytes

10 GiB

The approximate amount of disk space dedicated to storing time series and health data. Once the store has reached its maximum size older data will be deleted to make room for newer data. The disk usage is approximate because we only begin deleting data once we've reached the limit. Note that Cloudera Manager stores time-series data at a number of different data granularities, and these granularities have different effective retention periods. Specifically, Cloudera Manager stores metric data as both raw data points and ten-minutely, hourly, six-hourly, daily, and weekly summary data points. Raw data consumes the bulk of the allocated storage space, weekly summaries the least. As such, raw data is retained for the shortest amount of time, while weekly summary points are unlikely to ever be deleted. See the "Disk Usage" tab on the Service Monitor page for more information on how space is consumed within the Service Monitor. This tab also shows information about the amount of data retained and time window covered by each data granularity.

[#3]

When I try to make it less than 2GB I get this error in the CM GUI. Seems like 10GB is minimum? Maybe on a production cluster this make sense but I'm testing in a bunch of VMs and don't really care about the depth of stats so why can't I just reduce this? Why should I need to dedicate 10GB per VM just for stats and also there is a config entry for both Host Monitor and Service Monitor so does that mean it requires a minimum of 20GB per node??

2147483648 is less than the minimum allowed value 10737418240

[#4 · Accepted solution]

Yes, 10 GB is the minimum value here, and yes both the Host Monitor and Service Monitor have a setting like this.

So yes, the VM hosting your Service Monitor needs a volume to hold this store which will grow to approximately 10 GB over time. So does the VM hosting your Host Monitor. If it's the same VM, then you'd need 20 GB on that node. Note that things will perform much better if the Host Monitor and Service Monitor storage directories are on different volumes.

[#5]

hi, 

came across this post as i am having the same issue with the /var partition being at 100% full.

is it possible to delete older *.ts files from command prompt? aside from cleaning up /var/log, what else can i do?

thanks in advance, terry

[#6]

I am facing a similair issue, is there a definitive way of clearing up old ts files?

thanks

[#7]

has anyone tried deleting older ts directories?

[#8]

You should not do this yourself. You can control the amount of space
that CM consumes in these directories via the Time-Series Storage
configuration of HMON and SMON. There's a 10 GB minimum for both,
meaning you can expect 20 GB to be consumed by these two roles
eventually.

It is possible, though discouraged to set a lower limit, though you
have to use the cmon.conf safety valve. You can enter a number of
bytes to use as a maximum -- something like below where XXX is the
number of bytes. This requires an SMON or HMON restart, but it should
very quickly cleanup to the limit specified. Again, this is something
we don't really test with outside of some dev clusters, so proceed
with caution.


firehose_time_series_storage_bytes
XXX


chris

[#9]

Thanks Chris. I'll have a look.

[#10]

This worked for me, the snipped below is an example for limiting the diskspace to 5gb

<property>
   <name>firehose_time_series_storage_bytes</name>
   <value>5368709120</value>
</property>