Both Hbase Master server and Hbase RegionServer are frequently going down. Please find below error I have extracted for both server. Can you please look at log and suggest what changes I need to make in configuration to fix it.
Hbase Master /var/log/hbase/ gc.log-201810221025 [logfile]
[centos@ip-172-31-46-233 hbase]$ cat gc.log-201810221025
Java HotSpot(TM) 64-Bit Server VM (25.181-b13) for linux-amd64 JRE (1.8.0_181-b13), built on Jul 7 2018 00:56:38 by "java_re" with gcc 4.3.0 20080428 (Red Hat 4.3.0-8)
Memory: 4k page, physical 16265880k(227160k free), swap 0k(0k free)
CommandLine flags: -XX:ErrorFile=/var/log/hbase/hs_err_pid%p.log -XX:InitialHeapSize=260254080 -XX:MaxHeapSize=1073741824 -XX:MaxNewSize=348966912 -XX:MaxTenuringThreshold=6 -XX:OldPLABSize=16 -XX:OnOutOfMemoryError=kill -9 %p -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC
Heap
par new generation total 76800K, used 10973K [0x00000000c0000000, 0x00000000c5350000, 0x00000000d4cc0000)
eden space 68288K, 16% used [0x00000000c0000000, 0x00000000c0ab75a0, 0x00000000c42b0000)
from space 8512K, 0% used [0x00000000c42b0000, 0x00000000c42b0000, 0x00000000c4b00000)
to space 8512K, 0% used [0x00000000c4b00000, 0x00000000c4b00000, 0x00000000c5350000)
concurrent mark-sweep generation total 170688K, used 0K [0x00000000d4cc0000, 0x00000000df370000, 0x0000000100000000)
Metaspace used 2977K, capacity 4480K, committed 4480K, reserved 1056768K
class space used 313K, capacity 384K, committed 384K, reserved 1048576K
Hbase RegionServer /var/log/hbase/gc.log-201810221025 [logfile]
[centos@ip-172-31-47-215 hbase]$ cat gc.log-201810221025
Java HotSpot(TM) 64-Bit Server VM (25.181-b13) for linux-amd64 JRE (1.8.0_181-b13), built on Jul 7 2018 00:56:38 by "java_re" with gcc 4.3.0 20080428 (Red Hat 4.3.0-8)
Memory: 4k page, physical 16265880k(380008k free), swap 0k(0k free)
CommandLine flags: -XX:CMSInitiatingOccupancyFraction=50 -XX:ErrorFile=/var/log/hbase/hs_err_pid%p.log -XX:InitialHeapSize=2147483648 -XX:MaxHeapSize=2147483648 -XX:MaxNewSize=427819008 -XX:MaxTenuringThreshold=6 -XX:NewSize=427819008 -XX:OldPLABSize=16 -XX:OnOutOfMemoryError=kill -9 %p -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC
Heap
par new generation total 376064K, used 20060K [0x0000000080000000, 0x0000000099800000, 0x0000000099800000)
eden space 334336K, 6% used [0x0000000080000000, 0x00000000813970c8, 0x0000000094680000)
from space 41728K, 0% used [0x0000000094680000, 0x0000000094680000, 0x0000000096f40000)
to space 41728K, 0% used [0x0000000096f40000, 0x0000000096f40000, 0x0000000099800000)
concurrent mark-sweep generation total 1679360K, used 0K [0x0000000099800000, 0x0000000100000000, 0x0000000100000000)
Metaspace used 2981K, capacity 4480K, committed 4480K, reserved 1056768K
class space used 313K, capacity 384K, committed 384K, reserved 1048576K
[centos@ip-172-31-47-215 hbase]$
Also I am getting below output while submitting hadoop ls command
[centos@ip-172-31-46-233 hbase]$ hadoop fs -ls /user
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1344 100 1344 0 0 3468 0 --:--:-- --:--:-- --:--:-- 3463
Cron exists
Regards,
Mustafa Akmal
