Automatic deletion of rotated gc.log files

[#1]

we have hadoop cluster version 2.6.4 while metrics collector installed on one of our Linux servers

    root@master02 ambari-metrics-collector]#  rpm -qa | grep metrics-collector
    ambari-metrics-collector-2.6.1.0-143.x86_64

we have file of gc.log from Jul last year , and we want to automatic deletion of rotated gc.log files

under /var/log/ambari-metrics-collector

[root@master02 ambari-metrics-collector]# ls -ltr | grep gc.log | grep -v collector| more
    -rw-r--r--. 1 ams  hadoop      1430 Jul 13  2017 gc.log-201707131628
    -rw-r--r--  1 ams  hadoop 178518767 Aug 28  2017 gc.log-201707160637
    -rw-r--r--  1 ams  hadoop 128748829 Sep 17  2017 gc.log-201708280731
    -rw-r--r--  1 ams  hadoop     60849 Sep 18  2017 gc.log-201709181421
    -rw-r--r--  1 ams  hadoop     82562 Sep 18  2017 gc.log-201709181424
    -rw-r--r--  1 ams  hadoop 164381669 Oct 19  2017 gc.log-201709181508
    -rw-r--r--  1 ams  hadoop     46149 Oct 19  2017 gc.log-201710191738
    -rw-r--r--  1 ams  hadoop  13763796 Oct 23  2017 gc.log-201710200633
    -rw-r--r--  1 ams  hadoop   1048289 Oct 23  2017 gc.log-201710231021
    -rw-r--r--  1 ams  hadoop   3473874 Oct 24  2017 gc.log-201710231649
    -rw-r--r--  1 ams  hadoop 137289030 Nov 15  2017 gc.log-201710241352
    -rw-r--r--  1 ams  hadoop   8037548 Nov 16  2017 gc.log-201711151355
    -rw-r--r--  1 ams  hadoop  23919164 Nov 19  2017 gc.log-201711161246
    -rw-r--r--  1 ams  hadoop 139759594 Dec  6 16:35 gc.log-201711191241
    -rw-r--r--  1 ams  hadoop 184118442 Dec 25 16:13 gc.log-201712061635
    -rw-r--r--  1 ams  hadoop      1440 Jan  1 04:38 gc.log-201712251613
    -rw-r--r--  1 ams  hadoop      1437 Jan 10 00:03 gc.log-201801021532
    -rw-r--r--  1 ams  hadoop      1439 Jan 10 01:04 gc.log-201801100004
    -rw-r--r--  1 ams  hadoop      3637 Jan 10 01:11 gc.log-201801100105
    -rw-r--r--  1 ams  hadoop      1437 Jan 10 01:16 gc.log-201801100113
    -rw-r--r--  1 ams  hadoop      3636 Jan 10 01:24 gc.log-201801100118
    -rw-r--r--  1 ams  hadoop      1438 Jan 10 01:35 gc.log-201801100126
    -rw-r--r--  1 ams  hadoop      1438 Jan 10 01:42 gc.log-201801100137
    -rw-r--r--  1 ams  hadoop      1436 Jan 10 02:14 gc.log-201801100144
    -rw-r--r--  1 ams  hadoop      8353 Jan 10 02:19 gc.log-201801100216
    -rw-r--r--  1 ams  hadoop      1437 Jan 11 09:08 gc.log-201801110730
    -rw-r--r--  1 ams  hadoop      1436 Jan 11 09:19 gc.log-201801110909
    -rw-r--r--  1 ams  hadoop       766 Jan 11 09:20 gc.log-201801110920
    -rw-r--r--  1 ams  hadoop       765 Jan 11 09:26 gc.log-201801110926 

from Advanced ams-hbase-env in ambari gui we see the following :

{% if java_version < 8 %}
    export HBASE_MASTER_OPTS=" -XX:PermSize=64m -XX:MaxPermSize={{hbase_master_maxperm_size}} -Xms{{hbase_heapsize}} -Xmx{{hbase_heapsize}} -Xmn{{hbase_master_xmn_size}} -XX:CMSInitiatingOccupancyFraction=70 -XX:+UseCMSInitiatingOccupancyOnly"
    export HBASE_REGIONSERVER_OPTS="-XX:MaxPermSize=128m -Xmn{{regionserver_xmn_size}} -XX:CMSInitiatingOccupancyFraction=70 -XX:+UseCMSInitiatingOccupancyOnly -Xms{{regionserver_heapsize}} -Xmx{{regionserver_heapsize}}"
    {% else %}
    export HBASE_MASTER_OPTS=" -Xms{{hbase_heapsize}} -Xmx{{hbase_heapsize}} -Xmn{{hbase_master_xmn_size}} -XX:CMSInitiatingOccupancyFraction=70 -XX:+UseCMSInitiatingOccupancyOnly"
    export HBASE_REGIONSERVER_OPTS=" -Xmn{{regionserver_xmn_size}} -XX:CMSInitiatingOccupancyFraction=70 -XX:+UseCMSInitiatingOccupancyOnly -Xms{{regionserver_heapsize}} -Xmx{{regionserver_heapsize}}"
    {% endif %} 

and we try to set in HBASE_MASTER_OPTS or HBASE_REGIONSERVER_OPTS the **-XX:NumberOfGClogFiles=10** in order to get only 10 gc.log files , but metrics collector failed to start , seems because incorrect syntax export

HBASE_MASTER_OPTS=" -XX:PermSize=64m -XX:MaxPermSize={{hbase_master_maxperm_size}} -Xms{{hbase_heapsize}} -Xmx{{hbase_heapsize}} -Xmn{{hbase_master_xmn_size}} -XX:CMSInitiatingOccupancyFraction=70 -XX:+UseCMSInitiatingOccupancyOnly -XX:NumberOfGClogFiles=1" 

please advice how to perform automatic deletion of rotated gc.log files

[#2 · Accepted solution]

@Michael Bronson

When we want to do GC log rotation then in that case we will have to define few properties together to make it work appropriately like following along with the "-Xloggc" option.

-XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=2M

Just defining NumberOfGClogFiles option will lead to inconsistent JVM options and JVM will not boot like:

So please try this:

HBASE_MASTER_OPTS=" -XX:PermSize=64m -XX:MaxPermSize={{hbase_master_maxperm_size}} -Xms{{hbase_heapsize}} -Xmx{{hbase_heapsize}} -Xmn{{hbase_master_xmn_size}} -XX:CMSInitiatingOccupancyFraction=70 -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=3 -XX:GCLogFileSize=20M" 

.

NOTE: Also in your case the JVM was not booting because there is a Typo in your case "-XX:NumberOfGClogFiles" (here you used lowercase log) where as the correct option has Uppercase L in the option like "-XX:NumberOfGCLogFiles"