Support Questions

Find answers, ask questions, and share your expertise
Celebrating as our community reaches 100,000 members! Thank you!

Hbase region server is getting stopped frequently without any error log

Expert Contributor

We are having five nodes hadoop cluster powered by HDP (Version 2.1), Ambari (Version 1.6). We have 1 hbase(Version 0.98) master. 3 are data nodes which are having 3 region servers. We are having hbase application running on this.

For last couple of weeks, region server of data node2 was getting stopped arbitrarily without any error logs. What we observed is that region server was going down on weekly basis. But from last couple of days, region server of data node3 is also going down without any error logs.

Region server logs are as follows-

Log list-

-rw-r--r-- 1 hbase hadoop 191 Dec 29 18:17

-rw-r--r-- 1 hbase hadoop 814M Dec 29 18:17 gc.log-201511240826 -rw-r--r-- 1 hbase hadoop 191 Jan 4 18:27 -rw-r--r-- 1 hbase hadoop 186M Jan 4 18:27 gc.log-201512300433

[root@fsdata2c hbase]# more /usr/lib/hbase/bin/ line 197: 19217 Killed nice -n $HBASE_NICENESS "$HBASE_HOME"/bin/hbase --config "${HBASE_CONF_DIR }" $command "$@" start >> "$logout" 2>&1 [root@fsdata2c hbase]#

As of now, we are starting region server manually which is solving the problem on temporary basis until region server stops again.

We require permanent solutions. Can anyone please help on this issue.


Master Mentor

@Raja Ray are all standard requirements set, i.e. ulimit, swappiness? Also, can you check the disk health? Also, what OS are you running, in case of RPM based, do you have Transparent Huge Pages off?

View solution in original post


Master Mentor

Excellent can you please accept the answer to close it out @Raja Ray