Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Hbase RegionServer dies and shortly after Hbase Master dies. How do you fix this?

avatar
Explorer

@Jay SenSharma

@Geoffrey Shelton Okot

Ambari shows green for approximately 1 minute and I have followed the fine tuning guide for HBASE for Big Insights that utilizes HBASE here. All server start green and the RegionServers begin to die one-by-one and then Hbase Master finally dies. I even tried disabling RANGER and disabling the HBASE plugin to see if that had something to with the following error:

Connection failed: [Errno 111] Connection refused to x.x.x:16030 (region servers)

Connection failed: [Errno 111] Connection refused to x.x.x:16000 (master)

But that had no effect. My Hbase logs are located here.

1 ACCEPTED SOLUTION

avatar
Contributor

The error I see in those logs is "Caused by: java.lang.IllegalArgumentException: Path must start with / character" , which relates to this previous log message: baseZNode=hdfs://bi-node-1.novalocal:8020/amshbase

In Ambari->Ambari Metrics->Advanced ams-hbase-site, what is your setting for zookeeper.znode.parent ?

Is it set to 'hdfs://bi-node-1.novalocal:8020/amshbase'? Try setting it as /amshbase or /ams-hbase-unsecure

View solution in original post

4 REPLIES 4

avatar
Contributor

The error I see in those logs is "Caused by: java.lang.IllegalArgumentException: Path must start with / character" , which relates to this previous log message: baseZNode=hdfs://bi-node-1.novalocal:8020/amshbase

In Ambari->Ambari Metrics->Advanced ams-hbase-site, what is your setting for zookeeper.znode.parent ?

Is it set to 'hdfs://bi-node-1.novalocal:8020/amshbase'? Try setting it as /amshbase or /ams-hbase-unsecure

avatar
Explorer

@Ron Buckley

Thank you very much Ron! That worked! Should I start another thread on why Ambari-Metrics is not updating? All of my values are "n/a" and "no data available" and I have all green on the Ambari console.

avatar
Contributor

Assuming you bounced all of Ambari Metrics and possibly the services, and the metrics still aren't appearing, that could be a separate question.

Also, it looks like you're trying to use BigInsights 4.2, that's been deprecated by IBM in favor of HDP. You may want to look at upgrading instead of working through these issues on BI.

avatar
Explorer

@Ron Buckley

IBM has been using IOP (IBM Open Platform) all the way through version 4.2.5. BI version 4.3 is not available for free at least as the others. IOP as you mostly know is HDP or their fork of it at least that they maintain in their own repository. Lastly our sponsor is unfortunately using version 4.1 and 4.2 which is why I am using this. I would much rather run the latest vanilla version of HDP with all of the bug fixes and support.

I will start a new thread for the Ambari-Metrics in the interim. If you know something that refutes my statement above that I do not please let me know. Otherwise thank you very much for at least getting me through the Hbase hurdle.