phoenix due to regionserver OutOfMemory

[#1]

CDH 6.3.2

PHOENIX-5.0.0-cdh6.2.0.p0.1308267

add service phoenix

<property>
<name>hbase.regionserver.wal.codec</name>
<value>org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec</value>
</property> 

<property>
 <name>phoenix.functions.allowUserDefinedFunctions</name>
 <value>true</value>
 <description>enable UDF functions</description>
</property>

adv service and adv client

set phoenix.schema.mapSystemTablesToNamespace=true

set phoenix.schema.isNamespaceMappingEnabled=true

when I run phoenix-sqlline

hbase region is OutOfMemory

2021-09-09T11:43:01.362+0800: [GC (CMS Initial Mark) [1 CMS-initial-mark: 34175K(34176K)] 48788K(49536K), 0.0309811 secs] [Times: user=0.04 sys=0.01, real=0.03 secs] 
2021-09-09T11:43:01.393+0800: [CMS-concurrent-mark-start]
2021-09-09T11:43:01.398+0800: [Full GC (Allocation Failure) 2021-09-09T11:43:01.398+0800: [CMS2021-09-09T11:43:01.414+0800: [CMS-concurrent-mark: 0.018/0.020 secs] [Times: user=0.18 sys=0.00, real=0.02 secs] 
 (concurrent mode failure): 34175K->34175K(34176K), 0.2619832 secs] 49203K->48771K(49536K), [Metaspace: 68808K->68808K(1112064K)], 0.2621194 secs] [Times: user=0.38 sys=0.00, real=0.26 secs] 
2021-09-09T11:43:01.663+0800: [Full GC (Allocation Failure) 2021-09-09T11:43:01.663+0800: [CMS: 34175K->34175K(34176K), 0.2232891 secs] 49535K->48937K(49536K), [Metaspace: 68811K->68811K(1112064K)], 0.2234088 secs] [Times: user=0.22 sys=0.00, real=0.22 secs] 
2021-09-09T11:43:01.887+0800: [GC (CMS Initial Mark) [1 CMS-initial-mark: 34175K(34176K)] 48937K(49536K), 0.0326199 secs] [Times: user=0.04 sys=0.00, real=0.03 secs] 
2021-09-09T11:43:01.920+0800: [CMS-concurrent-mark-start]
2021-09-09T11:43:01.927+0800: [Full GC (Allocation Failure) 2021-09-09T11:43:01.927+0800: [CMS2021-09-09T11:43:01.944+0800: [CMS-concurrent-mark: 0.023/0.025 secs] [Times: user=0.26 sys=0.01, real=0.03 secs] 
 (concurrent mode failure): 34175K->34175K(34176K), 0.2658440 secs] 49535K->48922K(49536K), [Metaspace: 68819K->68819K(1112064K)], 0.2659855 secs] [Times: user=0.40 sys=0.00, real=0.27 secs] 
2021-09-09T11:43:02.201+0800: [Full GC (Allocation Failure) 2021-09-09T11:43:02.201+0800: [CMS: 34175K->34175K(34176K), 0.2263622 secs] 49535K->48954K(49536K), [Metaspace: 68830K->68830K(1112064K)], 0.2265067 secs] [Times: user=0.23 sys=0.00, real=0.22 secs] 
2021-09-09T11:43:02.428+0800: [GC (CMS Initial Mark) [1 CMS-initial-mark: 34175K(34176K)] 48955K(49536K), 0.0318545 secs] [Times: user=0.04 sys=0.01, real=0.03 secs] 
2021-09-09T11:43:02.460+0800: [CMS-concurrent-mark-start]
2021-09-09T11:43:02.467+0800: [Full GC (Allocation Failure) 2021-09-09T11:43:02.467+0800: [CMS2021-09-09T11:43:02.484+0800: [CMS-concurrent-mark: 0.022/0.023 secs] [Times: user=0.23 sys=0.00, real=0.03 secs] 
 (concurrent mode failure): 34175K->34175K(34176K), 0.2410237 secs] 49432K->49067K(49536K), [Metaspace: 68830K->68830K(1112064K)], 0.2411725 secs] [Times: user=0.36 sys=0.00, real=0.24 secs] 
2021-09-09T11:43:02.709+0800: [Full GC (Allocation Failure) 2021-09-09T11:43:02.709+0800: [CMS: 34175K->34175K(34176K), 0.1938236 secs] 49067K->49059K(49536K), [Metaspace: 68830K->68830K(1112064K)], 0.1939479 secs] [Times: user=0.20 sys=0.00, real=0.20 secs] 
java.lang.OutOfMemoryError: Java heap space
Dumping heap to /tmp/hbase_hbase-REGIONSERVER-da84ee7c6c5955f30c908d6d3c80fc67_pid19562.hprof ...
Heap dump file created [89278215 bytes in 0.692 secs]
#
# java.lang.OutOfMemoryError: Java heap space
# -XX:OnOutOfMemoryError="kill -9 %p
/opt/cloudera/cm-agent/service/common/killparent.sh"
#   Executing /bin/sh -c "kill -9 19562
/opt/cloudera/cm-agent/service/common/killparent.sh"...

[#2 · Accepted solution]

Hi @ighack,

Please search keywords "JvmPauseMonitor" in that RegionServer log to see if there are GC pause and determine is it GC or Non-GC.

- For GC pause general step is to increase heap size. 

Please go through the setting of "Java Heap Size of HBase RegionServer in Bytes" in CM > HBase > Configuration. If the current setting is small please increase it.

Please check this KB for the heap size tuning concept.

https://community.cloudera.com/t5/Community-Articles/Tuning-Hbase-for-optimized-performance-Part-1/t...

- For Non-GC pause you will need to check if:

Kernel blocking the process due to:
hardware issues
blocking I/O
page allocation failures due to heavy memory utilization
etc…
Look into kernel messages for clues
dmsg
/var/log/messages

Regards,

Will

If the answer helps, please accept as solution and click thumbs up.

[#3]

I increase heap size

Java Heap Size of HBase RegionServer from 50m to 80m

but I aslo use jdbc to connect

Class.forName("org.apache.phoenix.jdbc.PhoenixDriver");
Connection connection = DriverManager.getConnection("jdbc:phoenix:bigdser5:/hbase");

getConnection is block

[#4]

Hi @ighack,

If you mean the current RS heap is 50 megabytes or 80 megabytes, it's usually not enough.

A good number is 16GB ~ 31 GB for most cases. If you indeed don't have enough resource in RS nodes, at lease keep RS heap 4GB as default, if still see many long GC pauses you have to increase it.

Refer to below link to install phoenix and validate installation.

https://docs.cloudera.com/documentation/enterprise/latest/topics/phoenix_installation.html#concept_o...

If you installed as above steps, then in any of the CDH node find the JDBC jar:

find / -name "phoenix-*client.jar"

and follow this guide:

https://docs.cloudera.com/runtime/7.2.10/phoenix-access-data/topics/phoenix-orchestrating-sql.html

Check your JDBC URL syntax should looks like:

jdbc:phoenix:zookeeper_quorum:zookeeper_port:zookeeper_hbase_path

Regards,

Will