Hbase getting gc.log error as ([GC (Allocation Failure) 2017-06-05T09:05:30.275+0800: 57301.047: )

[#1]

1) what this log for /var/log/hbase/gc.log-201706041710 ?

2) Error as below on Hbase log GC ?

centos:/var/log/hbase # tailf gc.log-201706041710
2017-06-05T09:02:36.768+0800: 57127.541: [GC (Allocation Failure) 2017-06-05T09:02:36.769+0800: 57127.541: [ParNew: 600799K->49774K(629120K), 0.0073147 secs] 1457924K->907076K(2027264K), 0.0075249 secs] [Times: user=0.12 sys=0.00, real=0.01 secs]
2017-06-05T09:02:40.990+0800: 57131.762: [GC (Allocation Failure) 2017-06-05T09:02:40.990+0800: 57131.762: [ParNew: 609006K->44417K(629120K), 0.0074685 secs] 1466308K->907097K(2027264K), 0.0076719 secs] [Times: user=0.12 sys=0.00, real=0.01 secs]
2017-06-05T09:03:33.493+0800: 57184.265: [GC (Allocation Failure) 2017-06-05T09:03:33.493+0800: 57184.265: [ParNew: 603649K->40908K(629120K), 0.0089747 secs] 1466329K->903627K(2027264K), 0.0091741 secs] [Times: user=0.09 sys=0.01, real=0.01 secs]
2017-06-05T09:03:37.142+0800: 57187.915: [GC (Allocation Failure) 2017-06-05T09:03:37.142+0800: 57187.915: [ParNew: 600140K->46821K(629120K), 0.0105207 secs] 1462859K->915053K(2027264K), 0.0107302 secs] [Times: user=0.13 sys=0.00, real=0.01 secs]
2017-06-05T09:03:53.208+0800: 57203.980: [GC (Allocation Failure) 2017-06-05T09:03:53.208+0800: 57203.980: [ParNew: 606053K->46967K(629120K), 0.0062205 secs] 1474285K->915216K(2027264K), 0.0064273 secs] [Times: user=0.09 sys=0.01, real=0.01 secs]
2017-06-05T09:04:34.979+0800: 57245.752: [GC (Allocation Failure) 2017-06-05T09:04:34.979+0800: 57245.752: [ParNew: 606199K->43336K(629120K), 0.0098484 secs] 1474448K->919955K(2027264K), 0.0100675 secs] [Times: user=0.13 sys=0.00, real=0.01 secs]
2017-06-05T09:04:38.378+0800: 57249.150: [GC (Allocation Failure) 2017-06-05T09:04:38.378+0800: 57249.150: [ParNew: 602568K->40051K(629120K), 0.0076083 secs] 1479187K->922158K(2027264K), 0.0078408 secs] [Times: user=0.10 sys=0.00, real=0.01 secs]
2017-06-05T09:05:30.275+0800: 57301.047: [GC (Allocation Failure) 2017-06-05T09:05:30.275+0800: 57301.047: [ParNew: 599283K->32614K(629120K), 0.0057252 secs] 1481390K->914735K(2027264K), 0.0512496 secs] [Times: user=0.08 sys=0.00, real=0.05 secs]
2017-06-05T09:05:38.363+0800: 57309.135: [GC (Allocation Failure) 2017-06-05T09:05:38.363+0800: 57309.135: [ParNew: 591846K->35399K(629120K), 0.0083321 secs] 1473967K->922741K(2027264K), 0.0085566 secs] [Times: user=0.10 sys=0.01, real=0.00 secs]

[#2]

@zkfs

What's your heap size? This is a Java Garbage collection issue. Usually happens in Eden generation. What's the size of young generation? JVM parameter "NewSize"?

In fact you can see it's the parallel new garbage collection that is unable to allocate memory. Parallel new garbage collector works on young generation. You probably have to allocate more memory.

[#3]

HBase Master Maximum Memory : 32 GB

HBase RegionServer Maximum Memory :20 GB

[#4]

Allocation failures happen when the JVM heap fills up. This is what triggers a garbage collection. It is not a problem, it's how Java GC works.

The issue you would need to worry about is if it happens too often or takes too much time.

[#5]

This issue occurring once in 3-4 days ays, below four log created in hbase log directory.

Jun  3 00:16 gc.log-201705120309
Jun  3 00:19 gc.log-201706030016
Jun  3 00:45 gc.log-201706030019
Jun  4 13:43 gc.log-201706030045
Jun  4 17:10 gc.log-201706041343
Jun  7 12:18 gc.log-201706041710