/run/cloudera-scm-agent/process fills up disk

[#1]

Hello,

We are using Cloudera Manager 5.7.0 with cloudera parcels 5.7.0. We have found that some files in /run/cloudera-scm-agent/process/* are filling up disk space. Here's some examples :

[root@cdh02:/run/cloudera-scm-agent/process]# ls -larth | grep collect-host-statistics
drwxr-x--x  4 root      root      120 juin  29 16:01 968-collect-host-statistics
drwxr-x--x  4 root      root      120 juil.  6 16:01 981-collect-host-statistics
drwxr-x--x  4 root      root      120 juil. 13 16:01 996-collect-host-statistics
drwxr-x--x  4 root      root      120 juil. 20 16:01 1010-collect-host-statistics
drwxr-x--x  4 root      root      120 juil. 27 16:02 1037-collect-host-statistics
drwxr-x--x  4 root      root      120 août   3 16:02 1051-collect-host-statistics
drwxr-x--x  4 root      root      120 août  10 16:02 1065-collect-host-statistics
drwxr-x--x  4 root      root      120 août  17 16:02 1086-collect-host-statistics
drwxr-x--x  4 root      root      100 août  23 16:00 1100-collect-host-statistics
3:41 [root@cdh02:/run/cloudera-scm-agent/process]# ls -larth | grep host-inspector
drwxr-x--x  3 root      root      120 juin  29 16:11 975-host-inspector
drwxr-x--x  3 root      root      120 juil.  6 16:11 988-host-inspector
drwxr-x--x  3 root      root      120 juil. 13 16:01 1003-host-inspector
drwxr-x--x  3 root      root      120 juil. 20 16:01 1023-host-inspector
drwxr-x--x  3 root      root      120 juil. 27 16:02 1044-host-inspector
drwxr-x--x  3 root      root      120 août   3 16:02 1058-host-inspector
drwxr-x--x  3 root      root      120 août  10 16:12 1072-host-inspector
drwxr-x--x  3 root      root      120 août  17 16:02 1093-host-inspector
drwxr-x--x  3 root      root      100 août  23 16:02 1107-host-inspector

du -sh * | grep collect-host-statistics
36M    1010-collect-host-statistics
36M    1037-collect-host-statistics
38M    1051-collect-host-statistics
38M    1065-collect-host-statistics
39M    1086-collect-host-statistics
41M    1100-collect-host-statistics
38M    968-collect-host-statistics
36M    981-collect-host-statistics
38M    996-collect-host-statistics

du -sh * | grep M
214M    1037-collect-host-statistics
223M    1051-collect-host-statistics
226M    1086-collect-host-statistics
227M    968-collect-host-statistics
214M    981-collect-host-statistics
223M    996-collect-host-statistics

Is there a Cloudera manager configuration to manage this ? My colleague found that it seems to rotate the file every 7 days (like defined in one of the retention policy). But shouldn't it clean the old files afterwards ?

Is this the normal behavior ? Are these files still useful ? Can we delete them safely ?

Regards,

Nicolas PHUNG

[#2 · Accepted solution]

https://community.cloudera.com/t5/Cloudera-Manager-Installation/Filled-up-cm-processes/td-p/20430

This works but it's weird to have to restart the agent every now and then.

[#3]

Its related to statistics data. i was check whole of file and find it related to killed process. you can empty statictics data safty by below command:

find /run -name messages_stdout -exec bash -c 'echo >"$1"' _ {} \;