question Re: Hbase RegionServer dies and shortly after Hbase Master dies. How do you fix this? in Archives of Support Questions (Read Only)

Hbase RegionServer dies and shortly after Hbase Master dies. How do you fix this?

JohnsonF — Tue, 15 Aug 2017 23:01:03 GMT

Ambari shows green for approximately 1 minute and I have followed the fine tuning guide for HBASE for Big Insights that utilizes HBASE here. All server start green and the RegionServers begin to die one-by-one and then Hbase Master finally dies. I even tried disabling RANGER and disabling the HBASE plugin to see if that had something to with the following error:

Connection failed: [Errno 111] Connection refused to x.x.x:16030 (region servers)

Connection failed: [Errno 111] Connection refused to x.x.x:16000 (master)

But that had no effect. My Hbase logs are located here.

Re: Hbase RegionServer dies and shortly after Hbase Master dies. How do you fix this?

rbuckley — Wed, 16 Aug 2017 01:32:07 GMT

The error I see in those logs is "Caused by: java.lang.IllegalArgumentException: Path must start with / character" , which relates to this previous log message: baseZNode=hdfs://bi-node-1.novalocal:8020/amshbase

In Ambari->Ambari Metrics->Advanced ams-hbase-site, what is your setting for zookeeper.znode.parent ?

Is it set to 'hdfs://bi-node-1.novalocal:8020/amshbase'? Try setting it as /amshbase or /ams-hbase-unsecure

Re: Hbase RegionServer dies and shortly after Hbase Master dies. How do you fix this?

JohnsonF — Wed, 16 Aug 2017 02:41:47 GMT

@Ron Buckley

Thank you very much Ron! That worked! Should I start another thread on why Ambari-Metrics is not updating? All of my values are "n/a" and "no data available" and I have all green on the Ambari console.

Re: Hbase RegionServer dies and shortly after Hbase Master dies. How do you fix this?

rbuckley — Wed, 16 Aug 2017 07:38:29 GMT

Assuming you bounced all of Ambari Metrics and possibly the services, and the metrics still aren't appearing, that could be a separate question.

Also, it looks like you're trying to use BigInsights 4.2, that's been deprecated by IBM in favor of HDP. You may want to look at upgrading instead of working through these issues on BI.

Re: Hbase RegionServer dies and shortly after Hbase Master dies. How do you fix this?

JohnsonF — Wed, 16 Aug 2017 19:44:24 GMT

@Ron Buckley

IBM has been using IOP (IBM Open Platform) all the way through version 4.2.5. BI version 4.3 is not available for free at least as the others. IOP as you mostly know is HDP or their fork of it at least that they maintain in their own repository. Lastly our sponsor is unfortunately using version 4.1 and 4.2 which is why I am using this. I would much rather run the latest vanilla version of HDP with all of the bug fixes and support.

I will start a new thread for the Ambari-Metrics in the interim. If you know something that refutes my statement above that I do not please let me know. Otherwise thank you very much for at least getting me through the Hbase hurdle.