Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

org.apache.hadoop.hbase.PleaseHoldException: Master is initializing error

org.apache.hadoop.hbase.PleaseHoldException: Master is initializing error

New Contributor
hello,
i am installing hdp 2.4 on 3 servers on aws, all thing go well, but in the last step of deployment, there is problem of check hbase with following log:


2016-09-01 09:20:07,334 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2016-09-01 09:20:07,345 - File['/var/lib/ambari-agent/tmp/hbaseSmokeVerify.sh'] {'content': StaticFile('hbaseSmokeVerify.sh'), 'mode': 0755}
2016-09-01 09:20:07,357 - File['/var/lib/ambari-agent/tmp/hbase-smoke.sh'] {'content': Template('hbase-smoke.sh.j2'), 'mode': 0755}
2016-09-01 09:20:07,358 - Writing File['/var/lib/ambari-agent/tmp/hbase-smoke.sh'] because contents don't match
2016-09-01 09:20:07,359 - Execute[' /usr/hdp/current/hbase-client/bin/hbase --config /usr/hdp/current/hbase-client/conf shell /var/lib/ambari-agent/tmp/hbase-smoke.sh && /var/lib/ambari-agent/tmp/hbaseSmokeVerify.sh /usr/hdp/current/hbase-client/conf id1faca91a_date200116 /usr/hdp/current/hbase-client/bin/hbase'] {'logoutput': True, 'tries': 6, 'user': 'ambari-qa', 'try_sleep': 5}
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hdp/2.4.2.0-258/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/2.4.2.0-258/zookeeper/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

ERROR: Can't get the locations

Here is some help for this command:
Start disable of named table:
  hbase> disable 't1'
  hbase> disable 'ns1:t1'
ERROR: Can't get the locations

Here is some help for this command:
Drop the named table. Table must first be disabled:
  hbase> drop 't1'
  hbase> drop 'ns1:t1'
ERROR: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing
	at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2324)
	at org.apache.hadoop.hbase.master.HMaster.checkNamespaceManagerReady(HMaster.java:2329)
	at org.apache.hadoop.hbase.master.HMaster.ensureNamespaceExists(HMaster.java:2522)
	at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1527)
	at org.apache.hadoop.hbase.master.MasterRpcServices.createTable(MasterRpcServices.java:454)
	at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:55401)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
	at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
	at java.lang.Thread.run(Thread.java:745)

Here is some help for this command:
Creates a table. Pass a table name, and a set of column family
specifications (at least one), and, optionally, table configuration.
Column specification can be a simple string (name), or a dictionary
(dictionaries are described below in main help output), necessarily 
including NAME attribute. 
Examples:

Create a table with namespace=ns1 and table qualifier=t1
  hbase> create 'ns1:t1', {NAME => 'f1', VERSIONS => 5}

Create a table with namespace=default and table qualifier=t1
  hbase> create 't1', {NAME => 'f1'}, {NAME => 'f2'}, {NAME => 'f3'}
  hbase> # The above in shorthand would be the following:
  hbase> create 't1', 'f1', 'f2', 'f3'
  hbase> create 't1', {NAME => 'f1', VERSIONS => 1, TTL => 2592000, BLOCKCACHE => true}
  hbase> create 't1', {NAME => 'f1', CONFIGURATION => {'hbase.hstore.blockingStoreFiles' => '10'}}
  
Table configuration options can be put at the end.
Examples:

  hbase> create 'ns1:t1', 'f1', SPLITS => ['10', '20', '30', '40']
  hbase> create 't1', 'f1', SPLITS => ['10', '20', '30', '40']
  hbase> create 't1', 'f1', SPLITS_FILE => 'splits.txt', OWNER => 'johndoe'
  hbase> create 't1', {NAME => 'f1', VERSIONS => 5}, METADATA => { 'mykey' => 'myvalue' }
  hbase> # Optionally pre-split the table into NUMREGIONS, using
  hbase> # SPLITALGO ("HexStringSplit", "UniformSplit" or classname)
  hbase> create 't1', 'f1', {NUMREGIONS => 15, SPLITALGO => 'HexStringSplit'}
  hbase> create 't1', 'f1', {NUMREGIONS => 15, SPLITALGO => 'HexStringSplit', REGION_REPLICATION => 2, CONFIGURATION => {'hbase.hregion.scan.loadColumnFamiliesOnDemand' => 'true'}}

You can also keep around a reference to the created table:

  hbase> t1 = create 't1', 'f1'

Which gives you a reference to the table named 't1', on which you can then
call methods.





with this problem, i can still finish the deployment and go into the web dashboard of ambari and see hbase is well started.

but when i go into hbase shell(successfully go into shell), and then try to create a table(or use status), there is problem of :

ERROR: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing

and i check hbase log in my server, it says :

ERROR [Thread-68] master.HMaster: Master failed to complete initialization after 900000ms. Please consider submitting a bug report including a thread dump of this process.

I've already try some method mentioned on the internet like:

1.

Please stop hbase on your cluster first. And restart them with
certain sequences: first regionserver on all nodes, then hmaster.

2.

stop hbase and zookeeper

wipe out datadir: version-2 of zookeeper

restart zookeeper and then restart hbase

with these two solution, i still have the same problem, can someone help me?

19 REPLIES 19

Re: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing error

can you attach the master logs, take and attach multiple jstack (in internval of 1 min) during initilization of master.

Re: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing error

New Contributor

hello,

as i've already indicate in the question, my hbase master log is following:

ERROR [Thread-68] master.HMaster: Master failed to complete initialization after 900000ms. Please consider submitting a bug report including a thread dump of this process.

and "multiple jstack during initilization of master." i don't know where i can get it?

Re: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing error

New Contributor

when i create table or tape status in hbase shell, i can see:

ERROR: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2324) at org.apache.hadoop.hbase.master.HMaster.checkNamespaceManagerReady(HMaster.java:2329) at org.apache.hadoop.hbase.master.HMaster.ensureNamespaceExists(HMaster.java:2522) at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1527) at org.apache.hadoop.hbase.master.MasterRpcServices.createTable(MasterRpcServices.java:454) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:55401) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Here is some help for this command: Creates a table. Pass a table name, and a set of column family specifications (at least one), and, optionally, table configuration. Column specification can be a simple string (name), or a dictionary (dictionaries are described below in main help output), necessarily including NAME attribute. Examples: Create a table with namespace=ns1 and table qualifier=t1 hbase> create 'ns1:t1', {NAME => 'f1', VERSIONS => 5} Create a table with namespace=default and table qualifier=t1 hbase> create 't1', {NAME => 'f1'}, {NAME => 'f2'}, {NAME => 'f3'} hbase> # The above in shorthand would be the following: hbase> create 't1', 'f1', 'f2', 'f3' hbase> create 't1', {NAME => 'f1', VERSIONS => 1, TTL => 2592000, BLOCKCACHE => true} hbase> create 't1', {NAME => 'f1', CONFIGURATION => {'hbase.hstore.blockingStoreFiles' => '10'}} Table configuration options can be put at the end. Examples: hbase> create 'ns1:t1', 'f1', SPLITS => ['10', '20', '30', '40'] hbase> create 't1', 'f1', SPLITS => ['10', '20', '30', '40'] hbase> create 't1', 'f1', SPLITS_FILE => 'splits.txt', OWNER => 'johndoe' hbase> create 't1', {NAME => 'f1', VERSIONS => 5}, METADATA => { 'mykey' => 'myvalue' } hbase> # Optionally pre-split the table into NUMREGIONS, using hbase> # SPLITALGO ("HexStringSplit", "UniformSplit" or classname) hbase> create 't1', 'f1', {NUMREGIONS => 15, SPLITALGO => 'HexStringSplit'} hbase> create 't1', 'f1', {NUMREGIONS => 15, SPLITALGO => 'HexStringSplit', REGION_REPLICATION => 2, CONFIGURATION => {'hbase.hregion.scan.loadColumnFamiliesOnDemand' => 'true'}} You can also keep around a reference to the created table: hbase> t1 = create 't1', 'f1' Which gives you a reference to the table named 't1', on which you can then call methods.

Re: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing error

you can take a thread dump by running "jstack <master processid> > /tmp/hmaster_1.jstack".. take multiple of those during initilization.

Re: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing error

New Contributor

I use

ps aux | grep HMaster

to show processid of hmaster, i can see:

hbase 13711 37.9 3.3 2844668 274072 ? Sl 09:19 59:23 /usr/jdk64/jdk1.8.0_60/bin/java -Dproc_master -XX:OnOutOfMemoryError=kill -9 %p -Dhdp.version=2.4.2.0-258 -XX:+UseConcMarkSweepGC -XX:ErrorFile=/var/log/hbase/hs_err_pid%p.log -Djava.io.tmpdir=/tmp -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:/var/log/hbase/gc.log-201609010919 -Xmx1024m -Dhbase.log.dir=/var/log/hbase -Dhbase.log.file=hbase-hbase-master-ec2-13-26-256-150.eu-central-1.compute.amazonaws.com.log -Dhbase.home.dir=/usr/hdp/current/hbase-master/bin/.. -Dhbase.id.str=hbase -Dhbase.root.logger=INFO,RFA -Djava.library.path=:/usr/hdp/2.4.2.0-258/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.4.2.0-258/hadoop/lib/native -Dhbase.security.logger=INFO,RFAS org.apache.hadoop.hbase.master.HMaster start ubuntu 20760 0.0 0.0 10436 888 pts/2 S+ 11:56 0:00 grep --color=auto HMaster

then i use both 13711, and 20760 like

sudo jstack 13711 >~/1.txt

result is: 13711: Unable to open socket file: target process not responding or HotSpot VM not loaded The -F option can be used when the target process is not responding

can you help me?

Re: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing error

You need to execute the command with the same user "hbase"

Re: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing error

it seems jstack is from different java version than the version on which hbase is running. check your JAVA_HOME for hbase and use jstack from the path $JAVA_HOME/bin/jstack

Re: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing error

New Contributor

so if i use

sudo jstack -F 13711 > ~/1.txt

result is: Attaching to process ID 13711, please wait... Exception in thread "main" java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at sun.tools.jstack.JStack.runJStackTool(JStack.java:136) at sun.tools.jstack.JStack.main(JStack.java:102) Caused by: sun.jvm.hotspot.runtime.VMVersionMismatchException: Supported versions are 24.95-b01. Target VM is 25.60-b23 at sun.jvm.hotspot.runtime.VM.checkVMVersion(VM.java:234) at sun.jvm.hotspot.runtime.VM.<init>(VM.java:297) at sun.jvm.hotspot.runtime.VM.initialize(VM.java:368) at sun.jvm.hotspot.bugspot.BugSpotAgent.setupVM(BugSpotAgent.java:598) at sun.jvm.hotspot.bugspot.BugSpotAgent.go(BugSpotAgent.java:493) at sun.jvm.hotspot.bugspot.BugSpotAgent.attach(BugSpotAgent.java:331) at sun.jvm.hotspot.tools.Tool.start(Tool.java:163) at sun.jvm.hotspot.tools.JStack.main(JStack.java:86) ... 6 more

Re: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing error

New Contributor

hello, i use the java in /usr/jdk64/jdk1.8.0_60/bin

then use command:

sudo -u hbase jstack 13711 >~/1.txt

and the content in 1.txt is:

1.txt

Don't have an account?
Coming from Hortonworks? Activate your account here