Created 09-01-2016 09:42 AM
hello, i am installing hdp 2.4 on 3 servers on aws, all thing go well, but in the last step of deployment, there is problem of check hbase with following log: 2016-09-01 09:20:07,334 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf 2016-09-01 09:20:07,345 - File['/var/lib/ambari-agent/tmp/hbaseSmokeVerify.sh'] {'content': StaticFile('hbaseSmokeVerify.sh'), 'mode': 0755} 2016-09-01 09:20:07,357 - File['/var/lib/ambari-agent/tmp/hbase-smoke.sh'] {'content': Template('hbase-smoke.sh.j2'), 'mode': 0755} 2016-09-01 09:20:07,358 - Writing File['/var/lib/ambari-agent/tmp/hbase-smoke.sh'] because contents don't match 2016-09-01 09:20:07,359 - Execute[' /usr/hdp/current/hbase-client/bin/hbase --config /usr/hdp/current/hbase-client/conf shell /var/lib/ambari-agent/tmp/hbase-smoke.sh && /var/lib/ambari-agent/tmp/hbaseSmokeVerify.sh /usr/hdp/current/hbase-client/conf id1faca91a_date200116 /usr/hdp/current/hbase-client/bin/hbase'] {'logoutput': True, 'tries': 6, 'user': 'ambari-qa', 'try_sleep': 5} SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/hdp/2.4.2.0-258/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/hdp/2.4.2.0-258/zookeeper/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] ERROR: Can't get the locations Here is some help for this command: Start disable of named table: hbase> disable 't1' hbase> disable 'ns1:t1' ERROR: Can't get the locations Here is some help for this command: Drop the named table. Table must first be disabled: hbase> drop 't1' hbase> drop 'ns1:t1' ERROR: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2324) at org.apache.hadoop.hbase.master.HMaster.checkNamespaceManagerReady(HMaster.java:2329) at org.apache.hadoop.hbase.master.HMaster.ensureNamespaceExists(HMaster.java:2522) at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1527) at org.apache.hadoop.hbase.master.MasterRpcServices.createTable(MasterRpcServices.java:454) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:55401) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Here is some help for this command: Creates a table. Pass a table name, and a set of column family specifications (at least one), and, optionally, table configuration. Column specification can be a simple string (name), or a dictionary (dictionaries are described below in main help output), necessarily including NAME attribute. Examples: Create a table with namespace=ns1 and table qualifier=t1 hbase> create 'ns1:t1', {NAME => 'f1', VERSIONS => 5} Create a table with namespace=default and table qualifier=t1 hbase> create 't1', {NAME => 'f1'}, {NAME => 'f2'}, {NAME => 'f3'} hbase> # The above in shorthand would be the following: hbase> create 't1', 'f1', 'f2', 'f3' hbase> create 't1', {NAME => 'f1', VERSIONS => 1, TTL => 2592000, BLOCKCACHE => true} hbase> create 't1', {NAME => 'f1', CONFIGURATION => {'hbase.hstore.blockingStoreFiles' => '10'}} Table configuration options can be put at the end. Examples: hbase> create 'ns1:t1', 'f1', SPLITS => ['10', '20', '30', '40'] hbase> create 't1', 'f1', SPLITS => ['10', '20', '30', '40'] hbase> create 't1', 'f1', SPLITS_FILE => 'splits.txt', OWNER => 'johndoe' hbase> create 't1', {NAME => 'f1', VERSIONS => 5}, METADATA => { 'mykey' => 'myvalue' } hbase> # Optionally pre-split the table into NUMREGIONS, using hbase> # SPLITALGO ("HexStringSplit", "UniformSplit" or classname) hbase> create 't1', 'f1', {NUMREGIONS => 15, SPLITALGO => 'HexStringSplit'} hbase> create 't1', 'f1', {NUMREGIONS => 15, SPLITALGO => 'HexStringSplit', REGION_REPLICATION => 2, CONFIGURATION => {'hbase.hregion.scan.loadColumnFamiliesOnDemand' => 'true'}} You can also keep around a reference to the created table: hbase> t1 = create 't1', 'f1' Which gives you a reference to the table named 't1', on which you can then call methods.
with this problem, i can still finish the deployment and go into the web dashboard of ambari and see hbase is well started.
but when i go into hbase shell(successfully go into shell), and then try to create a table(or use status), there is problem of :
ERROR: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing
and i check hbase log in my server, it says :
ERROR [Thread-68] master.HMaster: Master failed to complete initialization after 900000ms. Please consider submitting a bug report including a thread dump of this process.
I've already try some method mentioned on the internet like:
1.
Please stop hbase on your cluster first. And restart them with certain sequences: first regionserver on all nodes, then hmaster.
2.
stop hbase and zookeeper
wipe out datadir: version-2 of zookeeper
restart zookeeper and then restart hbase
with these two solution, i still have the same problem, can someone help me?
Created 09-01-2016 09:53 AM
can you attach the master logs, take and attach multiple jstack (in internval of 1 min) during initilization of master.
Created 09-01-2016 10:02 AM
hello,
as i've already indicate in the question, my hbase master log is following:
ERROR [Thread-68] master.HMaster: Master failed to complete initialization after 900000ms. Please consider submitting a bug report including a thread dump of this process.
and "multiple jstack during initilization of master." i don't know where i can get it?
Created 09-01-2016 10:05 AM
when i create table or tape status in hbase shell, i can see:
ERROR: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2324) at org.apache.hadoop.hbase.master.HMaster.checkNamespaceManagerReady(HMaster.java:2329) at org.apache.hadoop.hbase.master.HMaster.ensureNamespaceExists(HMaster.java:2522) at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1527) at org.apache.hadoop.hbase.master.MasterRpcServices.createTable(MasterRpcServices.java:454) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:55401) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) Here is some help for this command: Creates a table. Pass a table name, and a set of column family specifications (at least one), and, optionally, table configuration. Column specification can be a simple string (name), or a dictionary (dictionaries are described below in main help output), necessarily including NAME attribute. Examples: Create a table with namespace=ns1 and table qualifier=t1 hbase> create 'ns1:t1', {NAME => 'f1', VERSIONS => 5} Create a table with namespace=default and table qualifier=t1 hbase> create 't1', {NAME => 'f1'}, {NAME => 'f2'}, {NAME => 'f3'} hbase> # The above in shorthand would be the following: hbase> create 't1', 'f1', 'f2', 'f3' hbase> create 't1', {NAME => 'f1', VERSIONS => 1, TTL => 2592000, BLOCKCACHE => true} hbase> create 't1', {NAME => 'f1', CONFIGURATION => {'hbase.hstore.blockingStoreFiles' => '10'}} Table configuration options can be put at the end. Examples: hbase> create 'ns1:t1', 'f1', SPLITS => ['10', '20', '30', '40'] hbase> create 't1', 'f1', SPLITS => ['10', '20', '30', '40'] hbase> create 't1', 'f1', SPLITS_FILE => 'splits.txt', OWNER => 'johndoe' hbase> create 't1', {NAME => 'f1', VERSIONS => 5}, METADATA => { 'mykey' => 'myvalue' } hbase> # Optionally pre-split the table into NUMREGIONS, using hbase> # SPLITALGO ("HexStringSplit", "UniformSplit" or classname) hbase> create 't1', 'f1', {NUMREGIONS => 15, SPLITALGO => 'HexStringSplit'} hbase> create 't1', 'f1', {NUMREGIONS => 15, SPLITALGO => 'HexStringSplit', REGION_REPLICATION => 2, CONFIGURATION => {'hbase.hregion.scan.loadColumnFamiliesOnDemand' => 'true'}} You can also keep around a reference to the created table: hbase> t1 = create 't1', 'f1' Which gives you a reference to the table named 't1', on which you can then call methods.
Created 09-01-2016 11:01 AM
you can take a thread dump by running "jstack <master processid> > /tmp/hmaster_1.jstack".. take multiple of those during initilization.
Created 09-01-2016 12:02 PM
I use
ps aux | grep HMaster
to show processid of hmaster, i can see:
hbase 13711 37.9 3.3 2844668 274072 ? Sl 09:19 59:23 /usr/jdk64/jdk1.8.0_60/bin/java -Dproc_master -XX:OnOutOfMemoryError=kill -9 %p -Dhdp.version=2.4.2.0-258 -XX:+UseConcMarkSweepGC -XX:ErrorFile=/var/log/hbase/hs_err_pid%p.log -Djava.io.tmpdir=/tmp -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:/var/log/hbase/gc.log-201609010919 -Xmx1024m -Dhbase.log.dir=/var/log/hbase -Dhbase.log.file=hbase-hbase-master-ec2-13-26-256-150.eu-central-1.compute.amazonaws.com.log -Dhbase.home.dir=/usr/hdp/current/hbase-master/bin/.. -Dhbase.id.str=hbase -Dhbase.root.logger=INFO,RFA -Djava.library.path=:/usr/hdp/2.4.2.0-258/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.4.2.0-258/hadoop/lib/native -Dhbase.security.logger=INFO,RFAS org.apache.hadoop.hbase.master.HMaster start ubuntu 20760 0.0 0.0 10436 888 pts/2 S+ 11:56 0:00 grep --color=auto HMaster
then i use both 13711, and 20760 like
sudo jstack 13711 >~/1.txt
result is: 13711: Unable to open socket file: target process not responding or HotSpot VM not loaded The -F option can be used when the target process is not responding
can you help me?
Created 09-01-2016 12:04 PM
You need to execute the command with the same user "hbase"
Created 09-01-2016 12:12 PM
it seems jstack is from different java version than the version on which hbase is running. check your JAVA_HOME for hbase and use jstack from the path $JAVA_HOME/bin/jstack
Created 09-01-2016 12:07 PM
so if i use
sudo jstack -F 13711 > ~/1.txt
result is: Attaching to process ID 13711, please wait... Exception in thread "main" java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at sun.tools.jstack.JStack.runJStackTool(JStack.java:136) at sun.tools.jstack.JStack.main(JStack.java:102) Caused by: sun.jvm.hotspot.runtime.VMVersionMismatchException: Supported versions are 24.95-b01. Target VM is 25.60-b23 at sun.jvm.hotspot.runtime.VM.checkVMVersion(VM.java:234) at sun.jvm.hotspot.runtime.VM.<init>(VM.java:297) at sun.jvm.hotspot.runtime.VM.initialize(VM.java:368) at sun.jvm.hotspot.bugspot.BugSpotAgent.setupVM(BugSpotAgent.java:598) at sun.jvm.hotspot.bugspot.BugSpotAgent.go(BugSpotAgent.java:493) at sun.jvm.hotspot.bugspot.BugSpotAgent.attach(BugSpotAgent.java:331) at sun.jvm.hotspot.tools.Tool.start(Tool.java:163) at sun.jvm.hotspot.tools.JStack.main(JStack.java:86) ... 6 more
Created 09-01-2016 12:46 PM
hello, i use the java in /usr/jdk64/jdk1.8.0_60/bin
then use command:
sudo -u hbase jstack 13711 >~/1.txt
and the content in 1.txt is: