Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here. Want to know more about what has changed? Check out the Community News blog.

hbase master not starting

hbase master not starting

Contributor

I recently had to destroy and rebuild my hbase cluster

and now I have the following error

I am on CDH 5.4.8 hbase version 1.0.0

 

 

 

2015-11-02 13:12:50,550 FATAL org.apache.hadoop.hbase.master.HMaster: Failed to become active master
org.apache.hadoop.hbase.util.FileSystemVersionException: HBase file layout needs to be upgraded. You have version null and I want version 8. Consult http://hbase.apache.org/book.html for further information about upgrading HBase. Is your hbase.rootdir valid? If so, you may need to run 'hbase hbck -fixVersionFile'.
        at org.apache.hadoop.hbase.util.FSUtils.checkVersion(FSUtils.java:603)
        at org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:455)
        at org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:146)
        at org.apache.hadoop.hbase.master.MasterFileSystem.<init>(MasterFileSystem.java:126)
        at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:611)
        at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:169)
        at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1484)
        at java.lang.Thread.run(Thread.java:745)
2015-11-02 13:12:50,594 FATAL org.apache.hadoop.hbase.master.HMaster: Unhandled exception. Starting shutdown.
org.apache.hadoop.hbase.util.FileSystemVersionException: HBase file layout needs to be upgraded. You have version null and I want version 8. Consult http://hbase.apache.org/book.html for further information about upgrading HBase. Is your hbase.rootdir valid? If so, you may need to run 'hbase hbck -fixVersionFile'.
        at org.apache.hadoop.hbase.util.FSUtils.checkVersion(FSUtils.java:603)
        at org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:455)
        at org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:146)
        at org.apache.hadoop.hbase.master.MasterFileSystem.<init>(MasterFileSystem.java:126)
        at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:611)
        at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:169)
        at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1484)
        at java.lang.Thread.run(Thread.java:745)

 

 

following the suggested instructions of running

hbase hbck -fixVersionFile

 

I get into the following loop

 

15/11/02 13:23:42 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
15/11/02 13:23:42 INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
15/11/02 13:23:42 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux
15/11/02 13:23:42 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64
15/11/02 13:23:42 INFO zookeeper.ZooKeeper: Client environment:os.version=3.13.0-63-generic
15/11/02 13:23:42 INFO zookeeper.ZooKeeper: Client environment:user.name=hbase
15/11/02 13:23:42 INFO zookeeper.ZooKeeper: Client environment:user.home=/var/lib/hbase
15/11/02 13:23:42 INFO zookeeper.ZooKeeper: Client environment:user.dir=/home/atom
15/11/02 13:23:42 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop2-private.wdc01.infra.ripple.com:2181,hadoop3-private.wdc01.infra.ripple.com:2181,hadoop4-private.wdc01.infra.ripple.com:2181,hadoop5-private.wdc01.infra.ripple.com:2181,hadoop1-private.wdc01.infra.ripple.com:2181 sessionTimeout=120000 watcher=hconnection-0x25373d5c0x0, quorum=hadoop2-private.wdc01.infra.ripple.com:2181,hadoop3-private.wdc01.infra.ripple.com:2181,hadoop4-private.wdc01.infra.ripple.com:2181,hadoop5-private.wdc01.infra.ripple.com:2181,hadoop1-private.wdc01.infra.ripple.com:2181, baseZNode=/hbase
15/11/02 13:23:42 DEBUG zookeeper.ClientCnxn: zookeeper.disableAutoWatchReset is false
15/11/02 13:23:42 INFO zookeeper.ClientCnxn: Opening socket connection to server hadoop1-private.wdc01.infra.ripple.com/10.125.122.206:2181. Will not attempt to authenticate using SASL (unknown error)
15/11/02 13:23:42 INFO zookeeper.ClientCnxn: Socket connection established, initiating session, client: /10.125.122.206:40252, server: hadoop1-private.wdc01.infra.ripple.com/10.125.122.206:2181
15/11/02 13:23:42 DEBUG zookeeper.ClientCnxn: Session establishment request sent on hadoop1-private.wdc01.infra.ripple.com/10.125.122.206:2181
15/11/02 13:23:42 INFO zookeeper.ClientCnxn: Session establishment complete on server hadoop1-private.wdc01.infra.ripple.com/10.125.122.206:2181, sessionid = 0x550c9770a820057, negotiated timeout = 120000
15/11/02 13:23:42 DEBUG zookeeper.ZooKeeperWatcher: hconnection-0x25373d5c0x0, quorum=hadoop2-private.wdc01.infra.ripple.com:2181,hadoop3-private.wdc01.infra.ripple.com:2181,hadoop4-private.wdc01.infra.ripple.com:2181,hadoop5-private.wdc01.infra.ripple.com:2181,hadoop1-private.wdc01.infra.ripple.com:2181, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null
15/11/02 13:23:42 DEBUG zookeeper.ZooKeeperWatcher: hconnection-0x25373d5c-0x550c9770a820057 connected
15/11/02 13:23:42 DEBUG zookeeper.ClientCnxn: Reading reply sessionid:0x550c9770a820057, packet:: clientPath:null serverPath:null finished:false header:: 1,3  replyHeader:: 1,98784249524,-101  request:: '/hbase/hbaseid,F  response::
15/11/02 13:23:42 INFO client.ZooKeeperRegistry: ClusterId read in ZooKeeper is null
15/11/02 13:23:42 DEBUG client.ConnectionManager$HConnectionImplementation: clusterid came back null, using default default-cluster
15/11/02 13:23:42 DEBUG ipc.AbstractRpcClient: Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@23ab1d4a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=false, bind address=null
15/11/02 13:23:42 DEBUG zookeeper.ClientCnxn: Reading reply sessionid:0x550c9770a820057, packet:: clientPath:null serverPath:null finished:false header:: 2,3  replyHeader:: 2,98784249524,0  request:: '/hbase,F  response:: s{98784249191,98784249191,1446491542990,1446491542990,0,20,0,0,0,8,98784249521}
15/11/02 13:23:42 DEBUG zookeeper.ClientCnxn: Reading reply sessionid:0x550c9770a820057, packet:: clientPath:null serverPath:null finished:false header:: 3,4  replyHeader:: 3,98784249524,-101  request:: '/hbase/master,F  response::
15/11/02 13:23:42 DEBUG zookeeper.ZKUtil: hconnection-0x25373d5c-0x550c9770a820057, quorum=hadoop2-private.wdc01.infra.ripple.com:2181,hadoop3-private.wdc01.infra.ripple.com:2181,hadoop4-private.wdc01.infra.ripple.com:2181,hadoop5-private.wdc01.infra.ripple.com:2181,hadoop1-private.wdc01.infra.ripple.com:2181, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error)
15/11/02 13:23:44 DEBUG zookeeper.ClientCnxn: Reading reply sessionid:0x550c9770a820057, packet:: clientPath:null serverPath:null finished:false header:: 4,3  replyHeader:: 4,98784249526,0  request:: '/hbase,F  response:: s{98784249191,98784249191,1446491542990,1446491542990,0,20,0,0,0,8,98784249521}
15/11/02 13:23:44 DEBUG zookeeper.ClientCnxn: Reading reply sessionid:0x550c9770a820057, packet:: clientPath:null serverPath:null finished:false header:: 5,4  replyHeader:: 5,98784249526,-101  request:: '/hbase/master,F  response::


15/11/02 13:25:02 DEBUG zookeeper.ZKUtil: hconnection-0x25373d5c-0x550c9770a820057, quorum=hadoop2-private.wdc01.infra.ripple.com:2181,hadoop3-private.wdc01.infra.ripple.com:2181,hadoop4-private.wdc01.infra.ripple.com:2181,hadoop5-private.wdc01.infra.ripple.com:2181,hadoop1-private.wdc01.infra.ripple.com:2181, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 15/11/02 13:25:11 DEBUG ipc.Client: The ping interval is 60000 ms. 15/11/02 13:25:11 DEBUG ipc.Client: Connecting to hadoop2-private.wdc01.infra.ripple.com/10.125.122.222:8020 15/11/02 13:25:12 DEBUG ipc.Client: IPC Client (1803296480) connection to hadoop2-private.wdc01.infra.ripple.com/10.125.122.222:8020 from hbase: starting, having connections 1 15/11/02 13:25:12 DEBUG ipc.Client: IPC Client (1803296480) connection to hadoop2-private.wdc01.infra.ripple.com/10.125.122.222:8020 from hbase sending #5 15/11/02 13:25:12 DEBUG ipc.Client: IPC Client (1803296480) connection to hadoop2-private.wdc01.infra.ripple.com/10.125.122.222:8020 from hbase got value #5 15/11/02 13:25:12 DEBUG ipc.ProtobufRpcEngine: Call: renewLease took 27ms 15/11/02 13:25:12 DEBUG hdfs.LeaseRenewer: Lease renewed for client DFSClient_NONMAPREDUCE_2118864826_1 15/11/02 13:25:12 DEBUG hdfs.LeaseRenewer: Lease renewer daemon for [DFSClient_NONMAPREDUCE_2118864826_1] with renew id 1 executed 15/11/02 13:25:12 DEBUG azure.NativeAzureFileSystem: finalize() called. 15/11/02 13:25:12 DEBUG azure.NativeAzureFileSystem: finalize() called. 15/11/02 13:25:22 DEBUG ipc.Client: IPC Client (1803296480) connection to hadoop2-private.wdc01.infra.ripple.com/10.125.122.222:8020 from hbase: closed 15/11/02 13:25:22 DEBUG ipc.Client: IPC Client (1803296480) connection to hadoop2-private.wdc01.infra.ripple.com/10.125.122.222:8020 from hbase: stopped, remaining connections 0 15/11/02 13:25:42 DEBUG ipc.Client: The ping interval is 60000 ms. 15/11/02 13:25:42 DEBUG ipc.Client: Connecting to hadoop2-private.wdc01.infra.ripple.com/10.125.122.222:8020 15/11/02 13:25:42 DEBUG ipc.Client: IPC Client (1803296480) connection to hadoop2-private.wdc01.infra.ripple.com/10.125.122.222:8020 from hbase: starting, having connections 1 15/11/02 13:25:42 DEBUG ipc.Client: IPC Client (1803296480) connection to hadoop2-private.wdc01.infra.ripple.com/10.125.122.222:8020 from hbase sending #6 15/11/02 13:25:42 DEBUG ipc.Client: IPC Client (1803296480) connection to hadoop2-private.wdc01.infra.ripple.com/10.125.122.222:8020 from hbase got value #6 15/11/02 13:25:42 DEBUG ipc.ProtobufRpcEngine: Call: renewLease took 3ms 15/11/02 13:25:42 DEBUG hdfs.LeaseRenewer: Lease renewed for client DFSClient_NONMAPREDUCE_2118864826_1 15/11/02 13:25:42 DEBUG hdfs.LeaseRenewer: Lease renewer daemon for [DFSClient_NONMAPREDUCE_2118864826_1] with renew id 1 executed 15/11/02 13:25:42 DEBUG zookeeper.ClientCnxn: Got ping response for sessionid: 0x550c9770a820057 after 0ms 15/11/02 13:25:52 DEBUG ipc.Client: IPC Client (1803296480) connection to hadoop2-private.wdc01.infra.ripple.com/10.125.122.222:8020 from hbase: closed 15/11/02 13:25:52 DEBUG ipc.Client: IPC Client (1803296480) connection to hadoop2-private.wdc01.infra.ripple.com/10.125.122.222:8020 from hbase: stopped, remaining connections 0

 

hdfs fsck /hbase yeilds healthy

 

hbase hbck -fixMeta -fixAssignments

gets into the same loop

15/11/02 13:33:40 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop2-private.wdc01.infra.ripple.com:2181,hadoop3-private.wdc01.infra.ripple.com:2181,hadoop4-private.wdc01.infra.ripple.com:2181,hadoop5-private.wdc01.infra.ripple.com:2181,hadoop1-private.wdc01.infra.ripple.com:2181 sessionTimeout=120000 watcher=hconnection-0x25373d5c0x0, quorum=hadoop2-private.wdc01.infra.ripple.com:2181,hadoop3-private.wdc01.infra.ripple.com:2181,hadoop4-private.wdc01.infra.ripple.com:2181,hadoop5-private.wdc01.infra.ripple.com:2181,hadoop1-private.wdc01.infra.ripple.com:2181, baseZNode=/hbase
15/11/02 13:33:40 DEBUG zookeeper.ClientCnxn: zookeeper.disableAutoWatchReset is false
15/11/02 13:33:40 INFO zookeeper.ClientCnxn: Opening socket connection to server hadoop5-private.wdc01.infra.ripple.com/10.125.122.251:2181. Will not attempt to authenticate using SASL (unknown error)
15/11/02 13:33:40 INFO zookeeper.ClientCnxn: Socket connection established, initiating session, client: /10.125.122.206:37511, server: hadoop5-private.wdc01.infra.ripple.com/10.125.122.251:2181
15/11/02 13:33:40 DEBUG zookeeper.ClientCnxn: Session establishment request sent on hadoop5-private.wdc01.infra.ripple.com/10.125.122.251:2181
15/11/02 13:33:40 INFO zookeeper.ClientCnxn: Session establishment complete on server hadoop5-private.wdc01.infra.ripple.com/10.125.122.251:2181, sessionid = 0x150c9770a7d0062, negotiated timeout = 120000
15/11/02 13:33:40 DEBUG zookeeper.ZooKeeperWatcher: hconnection-0x25373d5c0x0, quorum=hadoop2-private.wdc01.infra.ripple.com:2181,hadoop3-private.wdc01.infra.ripple.com:2181,hadoop4-private.wdc01.infra.ripple.com:2181,hadoop5-private.wdc01.infra.ripple.com:2181,hadoop1-private.wdc01.infra.ripple.com:2181, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null
15/11/02 13:33:40 DEBUG zookeeper.ZooKeeperWatcher: hconnection-0x25373d5c-0x150c9770a7d0062 connected
15/11/02 13:33:40 DEBUG zookeeper.ClientCnxn: Reading reply sessionid:0x150c9770a7d0062, packet:: clientPath:null serverPath:null finished:false header:: 1,3  replyHeader:: 1,98784249776,-101  request:: '/hbase/hbaseid,F  response::
15/11/02 13:33:40 INFO client.ZooKeeperRegistry: ClusterId read in ZooKeeper is null
15/11/02 13:33:40 DEBUG client.ConnectionManager$HConnectionImplementation: clusterid came back null, using default default-cluster
15/11/02 13:33:40 DEBUG ipc.AbstractRpcClient: Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@23ab1d4a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=false, bind address=null
15/11/02 13:33:40 DEBUG zookeeper.ClientCnxn: Reading reply sessionid:0x150c9770a7d0062, packet:: clientPath:null serverPath:null finished:false header:: 2,3  replyHeader:: 2,98784249776,0  request:: '/hbase,F  response:: s{98784249191,98784249191,1446491542990,1446491542990,0,24,0,0,0,8,98784249542}
15/11/02 13:33:40 DEBUG zookeeper.ClientCnxn: Reading reply sessionid:0x150c9770a7d0062, packet:: clientPath:null serverPath:null finished:false header:: 3,4  replyHeader:: 3,98784249776,-101  request:: '/hbase/master,F  response::
15/11/02 13:33:40 DEBUG zookeeper.ZKUtil: hconnection-0x25373d5c-0x150c9770a7d0062, quorum=hadoop2-private.wdc01.infra.ripple.com:2181,hadoop3-private.wdc01.infra.ripple.com:2181,hadoop4-private.wdc01.infra.ripple.com:2181,hadoop5-private.wdc01.infra.ripple.com:2181,hadoop1-private.wdc01.infra.ripple.com:2181, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error)
15/11/02 13:33:42 DEBUG zookeeper.ClientCnxn: Reading reply sessionid:0x150c9770a7d0062, packet:: clientPath:null serverPath:null finished:false header:: 4,3  replyHeader:: 4,98784249776,0  request:: '/hbase,F  response:: s{98784249191,98784249191,1446491542990,1446491542990,0,24,0,0,0,8,98784249542}
15/11/02 13:33:42 DEBUG zookeeper.ClientCnxn: Reading reply sessionid:0x150c9770a7d0062, packet:: clientPath:null serverPath:null finished:false header:: 5,4  replyHeader:: 5,98784249776,-101  request:: '/hbase/master,F  response::
15/11/02 13:33:42 DEBUG zookeeper.ZKUtil: hconnection-0x25373d5c-0x150c9770a7d0062, quorum=hadoop2-private.wdc01.infra.ripple.com:2181,hadoop3-private.wdc01.infra.ripple.com:2181,hadoop4-private.wdc01.infra.ripple.com:2181,hadoop5-private.wdc01.infra.ripple.com:2181,hadoop1-private.wdc01.infra.ripple.com:2181, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error)
15/11/02 13:33:45 DEBUG zookeeper.ClientCnxn: Reading reply sessionid:0x150c9770a7d0062, packet:: clientPath:null serverPath:null finished:false header:: 6,3  replyHeader:: 6,98784249800,0  request:: '/hbase,F  response:: s{98784249191,98784249191,1446491542990,1446491542990,0,24,0,0,0,8,98784249542}
15/11/02 13:33:45 DEBUG zookeeper.ClientCnxn: Reading reply sessionid:0x150c9770a7d0062, packet:: clientPath:null serverPath:null finished:false header:: 7,4  replyHeader:: 7,98784249800,-101  request:: '/hbase/master,F  response::
15/11/02 13:33:45 DEBUG zookeeper.ZKUtil: hconnection-0x25373d5c-0x150c9770a7d0062, quorum=hadoop2-private.wdc01.infra.ripple.com:2181,hadoop3-private.wdc01.infra.ripple.com:2181,hadoop4-private.wdc01.infra.ripple.com:2181,hadoop5-private.wdc01.infra.ripple.com:2181,hadoop1-private.wdc01.infra.ripple.com:2181, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error)
15/11/02 13:33:50 DEBUG ipc.Client: IPC Client (1803296480) connection to hadoop2-private.wdc01.infra.ripple.com/10.125.122.222:8020 from hbase: closed
15/11/02 13:33:50 DEBUG ipc.Client: IPC Client (1803296480) connection to hadoop2-private.wdc01.infra.ripple.com/10.125.122.222:8020 from hbase: stopped, remaining connections 0
15/11/02 13:33:50 DEBUG zookeeper.ClientCnxn: Reading reply sessionid:0x150c9770a7d0062, packet:: clientPath:null serverPath:null finished:false header:: 8,3  replyHeader:: 8,98784249800,0  request:: '/hbase,F  response:: s{98784249191,98784249191,1446491542990,1446491542990,0,24,0,0,0,8,98784249542}
15/11/02 13:33:50 DEBUG zookeeper.ClientCnxn: Reading reply sessionid:0x150c9770a7d0062, packet:: clientPath:null serverPath:null finished:false header:: 9,4  replyHeader:: 9,98784249800,-101  request:: '/hbase/master,F  response::
15/11/02 13:33:50 DEBUG zookeeper.ZKUtil: hconnection-0x25373d5c-0x150c9770a7d0062, quorum=hadoop2-private.wdc01.infra.ripple.com:2181,hadoop3-private.wdc01.infra.ripple.com:2181,hadoop4-private.wdc01.infra.ripple.com:2181,hadoop5-private.wdc01.infra.ripple.com:2181,hadoop1-private.wdc01.infra.ripple.com:2181, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error)

I have no data under hbase and all tables have been destroyed

 

 

 

1 REPLY 1

Re: hbase master not starting

Contributor

since data was lost

ended up removing service

deleting zookeeper node /hbase

restarting cluster

and added back service from scratch