question Re: Hbase Master and Reg Server shutting down on HDP 2.5 TP in Archives of Support Questions (Read Only)

Hbase Master and Reg Server shutting down on HDP 2.5 TP

ganne — Sat, 27 Aug 2016 01:19:41 GMT

On HDP 2.5 TP sandbox, HBase 's both master and Region servers are shutting down continuously. Attempting to restart the service from ambari also resulted in service going down after few seconds. Attaching the logs for reference. Any inputs appreciated. Thanks.

Re: Hbase Master and Reg Server shutting down on HDP 2.5 TP

rpednekar — Sat, 27 Aug 2016 01:22:14 GMT

Please check and attach .out file from /var/log/hbase for HBase master as well as region server.

Re: Hbase Master and Reg Server shutting down on HDP 2.5 TP

ganne — Sat, 27 Aug 2016 01:31:57 GMT

Thanks. Added the .out files. Please let me know your thoughts

Re: Hbase Master and Reg Server shutting down on HDP 2.5 TP

tyu — Sat, 27 Aug 2016 05:12:24 GMT

Snippet from master log:

http://pastebin.com/4iv5YKEQ

Here is related code:

String plugins = conf.get(HBASE_MASTER_LOGCLEANER_PLUGINS);

Looks like the value for "hbase.master.logcleaner.plugins" was null - default should be

org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner

Can you attach hbase-site.xml ?

Thanks

Re: Hbase Master and Reg Server shutting down on HDP 2.5 TP

ganne — Mon, 29 Aug 2016 07:17:40 GMT

@Ted Yu Thanks. I see that my hbase-default.xml which is packaged in the "hbase-common-1.1.2.2.5.0.0-817.jar" has the 'hbase.master.logcleaner.plugins' value set to 'org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner'.

Attaching my hbase-site.xml and hbase-default.xml files. Please let me know if you can find the issue.

I also keep getting the below error in a storm topology:

'java.lang.RuntimeException: hbase-default.xml file seems to be for an older version of HBase (1.1.2), this version is 1.1.2.2.5.0.0-817'

Re: Hbase Master and Reg Server shutting down on HDP 2.5 TP

tyu — Mon, 29 Aug 2016 08:53:34 GMT

'hbase.master.logcleaner.plugins' is not overridden in hbase-site.xml

Need to investigate further.