Support Questions

Find answers, ask questions, and share your expertise

org.apache.hadoop.hbase.PleaseHoldException: Master is initializing error

avatar
Explorer
hello,
i am installing hdp 2.4 on 3 servers on aws, all thing go well, but in the last step of deployment, there is problem of check hbase with following log:


2016-09-01 09:20:07,334 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2016-09-01 09:20:07,345 - File['/var/lib/ambari-agent/tmp/hbaseSmokeVerify.sh'] {'content': StaticFile('hbaseSmokeVerify.sh'), 'mode': 0755}
2016-09-01 09:20:07,357 - File['/var/lib/ambari-agent/tmp/hbase-smoke.sh'] {'content': Template('hbase-smoke.sh.j2'), 'mode': 0755}
2016-09-01 09:20:07,358 - Writing File['/var/lib/ambari-agent/tmp/hbase-smoke.sh'] because contents don't match
2016-09-01 09:20:07,359 - Execute[' /usr/hdp/current/hbase-client/bin/hbase --config /usr/hdp/current/hbase-client/conf shell /var/lib/ambari-agent/tmp/hbase-smoke.sh && /var/lib/ambari-agent/tmp/hbaseSmokeVerify.sh /usr/hdp/current/hbase-client/conf id1faca91a_date200116 /usr/hdp/current/hbase-client/bin/hbase'] {'logoutput': True, 'tries': 6, 'user': 'ambari-qa', 'try_sleep': 5}
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hdp/2.4.2.0-258/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/2.4.2.0-258/zookeeper/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

ERROR: Can't get the locations

Here is some help for this command:
Start disable of named table:
  hbase> disable 't1'
  hbase> disable 'ns1:t1'
ERROR: Can't get the locations

Here is some help for this command:
Drop the named table. Table must first be disabled:
  hbase> drop 't1'
  hbase> drop 'ns1:t1'
ERROR: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing
	at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2324)
	at org.apache.hadoop.hbase.master.HMaster.checkNamespaceManagerReady(HMaster.java:2329)
	at org.apache.hadoop.hbase.master.HMaster.ensureNamespaceExists(HMaster.java:2522)
	at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1527)
	at org.apache.hadoop.hbase.master.MasterRpcServices.createTable(MasterRpcServices.java:454)
	at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:55401)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
	at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
	at java.lang.Thread.run(Thread.java:745)

Here is some help for this command:
Creates a table. Pass a table name, and a set of column family
specifications (at least one), and, optionally, table configuration.
Column specification can be a simple string (name), or a dictionary
(dictionaries are described below in main help output), necessarily 
including NAME attribute. 
Examples:

Create a table with namespace=ns1 and table qualifier=t1
  hbase> create 'ns1:t1', {NAME => 'f1', VERSIONS => 5}

Create a table with namespace=default and table qualifier=t1
  hbase> create 't1', {NAME => 'f1'}, {NAME => 'f2'}, {NAME => 'f3'}
  hbase> # The above in shorthand would be the following:
  hbase> create 't1', 'f1', 'f2', 'f3'
  hbase> create 't1', {NAME => 'f1', VERSIONS => 1, TTL => 2592000, BLOCKCACHE => true}
  hbase> create 't1', {NAME => 'f1', CONFIGURATION => {'hbase.hstore.blockingStoreFiles' => '10'}}
  
Table configuration options can be put at the end.
Examples:

  hbase> create 'ns1:t1', 'f1', SPLITS => ['10', '20', '30', '40']
  hbase> create 't1', 'f1', SPLITS => ['10', '20', '30', '40']
  hbase> create 't1', 'f1', SPLITS_FILE => 'splits.txt', OWNER => 'johndoe'
  hbase> create 't1', {NAME => 'f1', VERSIONS => 5}, METADATA => { 'mykey' => 'myvalue' }
  hbase> # Optionally pre-split the table into NUMREGIONS, using
  hbase> # SPLITALGO ("HexStringSplit", "UniformSplit" or classname)
  hbase> create 't1', 'f1', {NUMREGIONS => 15, SPLITALGO => 'HexStringSplit'}
  hbase> create 't1', 'f1', {NUMREGIONS => 15, SPLITALGO => 'HexStringSplit', REGION_REPLICATION => 2, CONFIGURATION => {'hbase.hregion.scan.loadColumnFamiliesOnDemand' => 'true'}}

You can also keep around a reference to the created table:

  hbase> t1 = create 't1', 'f1'

Which gives you a reference to the table named 't1', on which you can then
call methods.





with this problem, i can still finish the deployment and go into the web dashboard of ambari and see hbase is well started.

but when i go into hbase shell(successfully go into shell), and then try to create a table(or use status), there is problem of :

ERROR: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing

and i check hbase log in my server, it says :

ERROR [Thread-68] master.HMaster: Master failed to complete initialization after 900000ms. Please consider submitting a bug report including a thread dump of this process.

I've already try some method mentioned on the internet like:

1.

Please stop hbase on your cluster first. And restart them with
certain sequences: first regionserver on all nodes, then hmaster.

2.

stop hbase and zookeeper

wipe out datadir: version-2 of zookeeper

restart zookeeper and then restart hbase

with these two solution, i still have the same problem, can someone help me?

19 REPLIES 19

avatar

can you attach the master logs, take and attach multiple jstack (in internval of 1 min) during initilization of master.

avatar
Explorer

hello,

as i've already indicate in the question, my hbase master log is following:

ERROR [Thread-68] master.HMaster: Master failed to complete initialization after 900000ms. Please consider submitting a bug report including a thread dump of this process.

and "multiple jstack during initilization of master." i don't know where i can get it?

avatar
Explorer

when i create table or tape status in hbase shell, i can see:

ERROR: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2324) at org.apache.hadoop.hbase.master.HMaster.checkNamespaceManagerReady(HMaster.java:2329) at org.apache.hadoop.hbase.master.HMaster.ensureNamespaceExists(HMaster.java:2522) at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1527) at org.apache.hadoop.hbase.master.MasterRpcServices.createTable(MasterRpcServices.java:454) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:55401) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Here is some help for this command: Creates a table. Pass a table name, and a set of column family specifications (at least one), and, optionally, table configuration. Column specification can be a simple string (name), or a dictionary (dictionaries are described below in main help output), necessarily including NAME attribute. Examples: Create a table with namespace=ns1 and table qualifier=t1 hbase> create 'ns1:t1', {NAME => 'f1', VERSIONS => 5} Create a table with namespace=default and table qualifier=t1 hbase> create 't1', {NAME => 'f1'}, {NAME => 'f2'}, {NAME => 'f3'} hbase> # The above in shorthand would be the following: hbase> create 't1', 'f1', 'f2', 'f3' hbase> create 't1', {NAME => 'f1', VERSIONS => 1, TTL => 2592000, BLOCKCACHE => true} hbase> create 't1', {NAME => 'f1', CONFIGURATION => {'hbase.hstore.blockingStoreFiles' => '10'}} Table configuration options can be put at the end. Examples: hbase> create 'ns1:t1', 'f1', SPLITS => ['10', '20', '30', '40'] hbase> create 't1', 'f1', SPLITS => ['10', '20', '30', '40'] hbase> create 't1', 'f1', SPLITS_FILE => 'splits.txt', OWNER => 'johndoe' hbase> create 't1', {NAME => 'f1', VERSIONS => 5}, METADATA => { 'mykey' => 'myvalue' } hbase> # Optionally pre-split the table into NUMREGIONS, using hbase> # SPLITALGO ("HexStringSplit", "UniformSplit" or classname) hbase> create 't1', 'f1', {NUMREGIONS => 15, SPLITALGO => 'HexStringSplit'} hbase> create 't1', 'f1', {NUMREGIONS => 15, SPLITALGO => 'HexStringSplit', REGION_REPLICATION => 2, CONFIGURATION => {'hbase.hregion.scan.loadColumnFamiliesOnDemand' => 'true'}} You can also keep around a reference to the created table: hbase> t1 = create 't1', 'f1' Which gives you a reference to the table named 't1', on which you can then call methods.

avatar

you can take a thread dump by running "jstack <master processid> > /tmp/hmaster_1.jstack".. take multiple of those during initilization.

avatar
Explorer

I use

ps aux | grep HMaster

to show processid of hmaster, i can see:

hbase 13711 37.9 3.3 2844668 274072 ? Sl 09:19 59:23 /usr/jdk64/jdk1.8.0_60/bin/java -Dproc_master -XX:OnOutOfMemoryError=kill -9 %p -Dhdp.version=2.4.2.0-258 -XX:+UseConcMarkSweepGC -XX:ErrorFile=/var/log/hbase/hs_err_pid%p.log -Djava.io.tmpdir=/tmp -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:/var/log/hbase/gc.log-201609010919 -Xmx1024m -Dhbase.log.dir=/var/log/hbase -Dhbase.log.file=hbase-hbase-master-ec2-13-26-256-150.eu-central-1.compute.amazonaws.com.log -Dhbase.home.dir=/usr/hdp/current/hbase-master/bin/.. -Dhbase.id.str=hbase -Dhbase.root.logger=INFO,RFA -Djava.library.path=:/usr/hdp/2.4.2.0-258/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.4.2.0-258/hadoop/lib/native -Dhbase.security.logger=INFO,RFAS org.apache.hadoop.hbase.master.HMaster start ubuntu 20760 0.0 0.0 10436 888 pts/2 S+ 11:56 0:00 grep --color=auto HMaster

then i use both 13711, and 20760 like

sudo jstack 13711 >~/1.txt

result is: 13711: Unable to open socket file: target process not responding or HotSpot VM not loaded The -F option can be used when the target process is not responding

can you help me?

avatar

You need to execute the command with the same user "hbase"

avatar

it seems jstack is from different java version than the version on which hbase is running. check your JAVA_HOME for hbase and use jstack from the path $JAVA_HOME/bin/jstack

avatar
Explorer

so if i use

sudo jstack -F 13711 > ~/1.txt

result is: Attaching to process ID 13711, please wait... Exception in thread "main" java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at sun.tools.jstack.JStack.runJStackTool(JStack.java:136) at sun.tools.jstack.JStack.main(JStack.java:102) Caused by: sun.jvm.hotspot.runtime.VMVersionMismatchException: Supported versions are 24.95-b01. Target VM is 25.60-b23 at sun.jvm.hotspot.runtime.VM.checkVMVersion(VM.java:234) at sun.jvm.hotspot.runtime.VM.<init>(VM.java:297) at sun.jvm.hotspot.runtime.VM.initialize(VM.java:368) at sun.jvm.hotspot.bugspot.BugSpotAgent.setupVM(BugSpotAgent.java:598) at sun.jvm.hotspot.bugspot.BugSpotAgent.go(BugSpotAgent.java:493) at sun.jvm.hotspot.bugspot.BugSpotAgent.attach(BugSpotAgent.java:331) at sun.jvm.hotspot.tools.Tool.start(Tool.java:163) at sun.jvm.hotspot.tools.JStack.main(JStack.java:86) ... 6 more

avatar
Explorer

hello, i use the java in /usr/jdk64/jdk1.8.0_60/bin

then use command:

sudo -u hbase jstack 13711 >~/1.txt

and the content in 1.txt is:

1.txt