Cloudera Community
Chandresh
user rank
Explorer

Re: Apache Phoenix / Hbase - Stops connecting If o...

by Explorer in Support Questions
‎03-15-2021 01:06 AM
‎03-15-2021 01:06 AM
@smdas Many thanks for your reply, we are looking at the inputs shared by you. Also we are going through the link you have shared regarding GC tuning.   While we are doing that, I wanted to bring up a point that we have a heavy read / write cluster. We do lots of selects and based on the result we either upsert / insert into the Hbase via Phoenix queries.   Do you think that could be a bottleneck too. We came across this link https://docs.cloudera.com/runtime/7.2.2/configuring-hbase/topics/hbase-advanced-configuration-write-heavy-workloads.html while we are trying to resolve the issue.    Are any of this configuration we need to check and configure that could also help along with the recommendations you have provided.   Thanks Chandresh ... View more

Apache Phoenix / Hbase - Stops connecting If one o...

by Explorer in Support Questions
‎03-14-2021 12:30 AM
‎03-14-2021 12:30 AM
Hi we are seeing an issue with our Cloudera setup. We have multiple data nodes and on each node we have phoenix set up to query on HBase tables.   What we have observed is that when one of the region server is going through Garbage collection (GC cycle) it becomes unavailable. Due to this unavailability of 1 region server, our entire cluster is becoming unavailable and phoenix is not able to connect to HBase and query the tables. We have around 40 region servers like that.   When we restart that region server, things comes back to normal and the phoenix can query the tables. Are we missing any specific configuration that will avoid blocking of all the region servers if one of them is not available. ... View more
Labels:

Re: HBase Master Running but not Active

by Explorer in Support Questions
‎05-19-2019 06:05 AM
‎05-19-2019 06:05 AM
Hi,   More update on this :   We upgraded our Cloudera Manager & CDH Version to 6.2 after following all the required steps.   Now, we can see that Master is active and Backup Master is also running and identified and assumed responsibility.   Problem :   When we go to hbase shell & run 'list' command we see all the tables that should be there.   When we run 'count' or 'scan' command we get below error in hbase shell :   "ERROR: No server address listed in hbase:meta for region volume,,1547112088868.09539931eb256ed2dfb5a43c116e0112. containing row"     Below is the error on HBase Master Log :   ABORTING master HOST,16000,1558270209713: Unhandled exception. Starting shutdown. ***** java.lang.IllegalStateException: Expected the service ClusterSchemaServiceImpl [FAILED] to be RUNNING, but the service has FAILED at org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.checkCurrentState(AbstractService.java:345) at org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.awaitRunning(AbstractService.java:291) at org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1260) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:1038) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2271) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:581) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Timedout 300000ms waiting for namespace table to be assigned and enabled: tableName=hbase:namespace, state=ENABLED at org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:108) at org.apache.hadoop.hbase.master.ClusterSchemaServiceImpl.doStart(ClusterSchemaServiceImpl.java:63) at org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.startAsync(AbstractService.java:226) at org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1258) ... 4 more ... View more

HBase Master Running but not Active

by Explorer in Support Questions
‎05-17-2019 10:11 PM
‎05-17-2019 10:11 PM
Hi -   We are running an issue with HBase where the Master is running but not active.   When I go to HBase shell and run 'List' command I get below error :   hbase(main):001:0> list TABLE ERROR: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2371) at org.apache.hadoop.hbase.master.MasterRpcServices.isMasterRunning(MasterRpcServices.java:938) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:55654) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2191) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:183) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:163) List all tables in hbase. Optional regular expression parameter could be used to filter the output. Examples: hbase> list hbase> list 'abc.*' hbase> list 'ns:abc.*' hbase> list 'ns:.*'   We have configured the 'Master' role in Cloudera Manager. The service starts properly and shows 'green' icon which says the service started succesfully.   But when when we go to hbase master log this was one of the error it showed :   ERROR org.apache.hadoop.hbase.master.HMaster: Master failed to complete initialization after 900000ms. Please consider submitting a bug report including a thread dump of this process.   Moreover, the HBase service page which shows all the instances list, shows Master instance in green color. But it does not shows Master (Active) state. ... View more
Contact Me
Online
Offline
Last Visited
‎04-07-2021 10:18 AM
Community Statistics
Member Since ‎05-17-2019 10:06 PM
Last Visited ‎04-07-2021 10:18 AM
Posts 4