Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Both Hbase Master server and Hbase RegionServer are frequently going down. Possible fix?

Highlighted

Both Hbase Master server and Hbase RegionServer are frequently going down. Possible fix?

New Contributor

Both Hbase Master server and Hbase RegionServer are frequently going down. Please find below error I have extracted for both server. Can you please look at log and suggest what changes I need to make in configuration to fix it.

Hbase Master /var/log/hbase/ gc.log-201810221025 [logfile]

[centos@ip-172-31-46-233 hbase]$ cat gc.log-201810221025

Java HotSpot(TM) 64-Bit Server VM (25.181-b13) for linux-amd64 JRE (1.8.0_181-b13), built on Jul 7 2018 00:56:38 by "java_re" with gcc 4.3.0 20080428 (Red Hat 4.3.0-8)

Memory: 4k page, physical 16265880k(227160k free), swap 0k(0k free)

CommandLine flags: -XX:ErrorFile=/var/log/hbase/hs_err_pid%p.log -XX:InitialHeapSize=260254080 -XX:MaxHeapSize=1073741824 -XX:MaxNewSize=348966912 -XX:MaxTenuringThreshold=6 -XX:OldPLABSize=16 -XX:OnOutOfMemoryError=kill -9 %p -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC

Heap

par new generation total 76800K, used 10973K [0x00000000c0000000, 0x00000000c5350000, 0x00000000d4cc0000)

eden space 68288K, 16% used [0x00000000c0000000, 0x00000000c0ab75a0, 0x00000000c42b0000)

from space 8512K, 0% used [0x00000000c42b0000, 0x00000000c42b0000, 0x00000000c4b00000)

to space 8512K, 0% used [0x00000000c4b00000, 0x00000000c4b00000, 0x00000000c5350000)

concurrent mark-sweep generation total 170688K, used 0K [0x00000000d4cc0000, 0x00000000df370000, 0x0000000100000000)

Metaspace used 2977K, capacity 4480K, committed 4480K, reserved 1056768K

class space used 313K, capacity 384K, committed 384K, reserved 1048576K

Hbase RegionServer /var/log/hbase/gc.log-201810221025 [logfile]

[centos@ip-172-31-47-215 hbase]$ cat gc.log-201810221025

Java HotSpot(TM) 64-Bit Server VM (25.181-b13) for linux-amd64 JRE (1.8.0_181-b13), built on Jul 7 2018 00:56:38 by "java_re" with gcc 4.3.0 20080428 (Red Hat 4.3.0-8)

Memory: 4k page, physical 16265880k(380008k free), swap 0k(0k free)

CommandLine flags: -XX:CMSInitiatingOccupancyFraction=50 -XX:ErrorFile=/var/log/hbase/hs_err_pid%p.log -XX:InitialHeapSize=2147483648 -XX:MaxHeapSize=2147483648 -XX:MaxNewSize=427819008 -XX:MaxTenuringThreshold=6 -XX:NewSize=427819008 -XX:OldPLABSize=16 -XX:OnOutOfMemoryError=kill -9 %p -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC

Heap

par new generation total 376064K, used 20060K [0x0000000080000000, 0x0000000099800000, 0x0000000099800000)

eden space 334336K, 6% used [0x0000000080000000, 0x00000000813970c8, 0x0000000094680000)

from space 41728K, 0% used [0x0000000094680000, 0x0000000094680000, 0x0000000096f40000)

to space 41728K, 0% used [0x0000000096f40000, 0x0000000096f40000, 0x0000000099800000)

concurrent mark-sweep generation total 1679360K, used 0K [0x0000000099800000, 0x0000000100000000, 0x0000000100000000)

Metaspace used 2981K, capacity 4480K, committed 4480K, reserved 1056768K

class space used 313K, capacity 384K, committed 384K, reserved 1048576K

[centos@ip-172-31-47-215 hbase]$

Also I am getting below output while submitting hadoop ls command

[centos@ip-172-31-46-233 hbase]$ hadoop fs -ls /user

% Total % Received % Xferd Average Speed Time Time Time Current

Dload Upload Total Spent Left Speed

100 1344 100 1344 0 0 3468 0 --:--:-- --:--:-- --:--:-- 3463

Cron exists

Regards,

Mustafa Akmal