Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

hbase master not starting

hbase master not starting

Contributor

I recently had to destroy and rebuild my hbase cluster

and now I have the following error

I am on CDH 5.4.8 hbase version 1.0.0

 

 

 

2015-11-02 13:12:50,550 FATAL org.apache.hadoop.hbase.master.HMaster: Failed to become active master
org.apache.hadoop.hbase.util.FileSystemVersionException: HBase file layout needs to be upgraded. You have version null and I want version 8. Consult http://hbase.apache.org/book.html for further information about upgrading HBase. Is your hbase.rootdir valid? If so, you may need to run 'hbase hbck -fixVersionFile'.
        at org.apache.hadoop.hbase.util.FSUtils.checkVersion(FSUtils.java:603)
        at org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:455)
        at org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:146)
        at org.apache.hadoop.hbase.master.MasterFileSystem.<init>(MasterFileSystem.java:126)
        at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:611)
        at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:169)
        at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1484)
        at java.lang.Thread.run(Thread.java:745)
2015-11-02 13:12:50,594 FATAL org.apache.hadoop.hbase.master.HMaster: Unhandled exception. Starting shutdown.
org.apache.hadoop.hbase.util.FileSystemVersionException: HBase file layout needs to be upgraded. You have version null and I want version 8. Consult http://hbase.apache.org/book.html for further information about upgrading HBase. Is your hbase.rootdir valid? If so, you may need to run 'hbase hbck -fixVersionFile'.
        at org.apache.hadoop.hbase.util.FSUtils.checkVersion(FSUtils.java:603)
        at org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:455)
        at org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:146)
        at org.apache.hadoop.hbase.master.MasterFileSystem.<init>(MasterFileSystem.java:126)
        at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:611)
        at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:169)
        at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1484)
        at java.lang.Thread.run(Thread.java:745)

 

 

following the suggested instructions of running

hbase hbck -fixVersionFile

 

I get into the following loop

 

15/11/02 13:23:42 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
15/11/02 13:23:42 INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
15/11/02 13:23:42 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux
15/11/02 13:23:42 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64
15/11/02 13:23:42 INFO zookeeper.ZooKeeper: Client environment:os.version=3.13.0-63-generic
15/11/02 13:23:42 INFO zookeeper.ZooKeeper: Client environment:user.name=hbase
15/11/02 13:23:42 INFO zookeeper.ZooKeeper: Client environment:user.home=/var/lib/hbase
15/11/02 13:23:42 INFO zookeeper.ZooKeeper: Client environment:user.dir=/home/atom
15/11/02 13:23:42 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop2-private.wdc01.infra.ripple.com:2181,hadoop3-private.wdc01.infra.ripple.com:2181,hadoop4-private.wdc01.infra.ripple.com:2181,hadoop5-private.wdc01.infra.ripple.com:2181,hadoop1-private.wdc01.infra.ripple.com:2181 sessionTimeout=120000 watcher=hconnection-0x25373d5c0x0, quorum=hadoop2-private.wdc01.infra.ripple.com:2181,hadoop3-private.wdc01.infra.ripple.com:2181,hadoop4-private.wdc01.infra.ripple.com:2181,hadoop5-private.wdc01.infra.ripple.com:2181,hadoop1-private.wdc01.infra.ripple.com:2181, baseZNode=/hbase
15/11/02 13:23:42 DEBUG zookeeper.ClientCnxn: zookeeper.disableAutoWatchReset is false
15/11/02 13:23:42 INFO zookeeper.ClientCnxn: Opening socket connection to server hadoop1-private.wdc01.infra.ripple.com/10.125.122.206:2181. Will not attempt to authenticate using SASL (unknown error)
15/11/02 13:23:42 INFO zookeeper.ClientCnxn: Socket connection established, initiating session, client: /10.125.122.206:40252, server: hadoop1-private.wdc01.infra.ripple.com/10.125.122.206:2181
15/11/02 13:23:42 DEBUG zookeeper.ClientCnxn: Session establishment request sent on hadoop1-private.wdc01.infra.ripple.com/10.125.122.206:2181
15/11/02 13:23:42 INFO zookeeper.ClientCnxn: Session establishment complete on server hadoop1-private.wdc01.infra.ripple.com/10.125.122.206:2181, sessionid = 0x550c9770a820057, negotiated timeout = 120000
15/11/02 13:23:42 DEBUG zookeeper.ZooKeeperWatcher: hconnection-0x25373d5c0x0, quorum=hadoop2-private.wdc01.infra.ripple.com:2181,hadoop3-private.wdc01.infra.ripple.com:2181,hadoop4-private.wdc01.infra.ripple.com:2181,hadoop5-private.wdc01.infra.ripple.com:2181,hadoop1-private.wdc01.infra.ripple.com:2181, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null
15/11/02 13:23:42 DEBUG zookeeper.ZooKeeperWatcher: hconnection-0x25373d5c-0x550c9770a820057 connected
15/11/02 13:23:42 DEBUG zookeeper.ClientCnxn: Reading reply sessionid:0x550c9770a820057, packet:: clientPath:null serverPath:null finished:false header:: 1,3  replyHeader:: 1,98784249524,-101  request:: '/hbase/hbaseid,F  response::
15/11/02 13:23:42 INFO client.ZooKeeperRegistry: ClusterId read in ZooKeeper is null
15/11/02 13:23:42 DEBUG client.ConnectionManager$HConnectionImplementation: clusterid came back null, using default default-cluster
15/11/02 13:23:42 DEBUG ipc.AbstractRpcClient: Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@23ab1d4a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=false, bind address=null
15/11/02 13:23:42 DEBUG zookeeper.ClientCnxn: Reading reply sessionid:0x550c9770a820057, packet:: clientPath:null serverPath:null finished:false header:: 2,3  replyHeader:: 2,98784249524,0  request:: '/hbase,F  response:: s{98784249191,98784249191,1446491542990,1446491542990,0,20,0,0,0,8,98784249521}
15/11/02 13:23:42 DEBUG zookeeper.ClientCnxn: Reading reply sessionid:0x550c9770a820057, packet:: clientPath:null serverPath:null finished:false header:: 3,4  replyHeader:: 3,98784249524,-101  request:: '/hbase/master,F  response::
15/11/02 13:23:42 DEBUG zookeeper.ZKUtil: hconnection-0x25373d5c-0x550c9770a820057, quorum=hadoop2-private.wdc01.infra.ripple.com:2181,hadoop3-private.wdc01.infra.ripple.com:2181,hadoop4-private.wdc01.infra.ripple.com:2181,hadoop5-private.wdc01.infra.ripple.com:2181,hadoop1-private.wdc01.infra.ripple.com:2181, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error)
15/11/02 13:23:44 DEBUG zookeeper.ClientCnxn: Reading reply sessionid:0x550c9770a820057, packet:: clientPath:null serverPath:null finished:false header:: 4,3  replyHeader:: 4,98784249526,0  request:: '/hbase,F  response:: s{98784249191,98784249191,1446491542990,1446491542990,0,20,0,0,0,8,98784249521}
15/11/02 13:23:44 DEBUG zookeeper.ClientCnxn: Reading reply sessionid:0x550c9770a820057, packet:: clientPath:null serverPath:null finished:false header:: 5,4  replyHeader:: 5,98784249526,-101  request:: '/hbase/master,F  response::


15/11/02 13:25:02 DEBUG zookeeper.ZKUtil: hconnection-0x25373d5c-0x550c9770a820057, quorum=hadoop2-private.wdc01.infra.ripple.com:2181,hadoop3-private.wdc01.infra.ripple.com:2181,hadoop4-private.wdc01.infra.ripple.com:2181,hadoop5-private.wdc01.infra.ripple.com:2181,hadoop1-private.wdc01.infra.ripple.com:2181, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 15/11/02 13:25:11 DEBUG ipc.Client: The ping interval is 60000 ms. 15/11/02 13:25:11 DEBUG ipc.Client: Connecting to hadoop2-private.wdc01.infra.ripple.com/10.125.122.222:8020 15/11/02 13:25:12 DEBUG ipc.Client: IPC Client (1803296480) connection to hadoop2-private.wdc01.infra.ripple.com/10.125.122.222:8020 from hbase: starting, having connections 1 15/11/02 13:25:12 DEBUG ipc.Client: IPC Client (1803296480) connection to hadoop2-private.wdc01.infra.ripple.com/10.125.122.222:8020 from hbase sending #5 15/11/02 13:25:12 DEBUG ipc.Client: IPC Client (1803296480) connection to hadoop2-private.wdc01.infra.ripple.com/10.125.122.222:8020 from hbase got value #5 15/11/02 13:25:12 DEBUG ipc.ProtobufRpcEngine: Call: renewLease took 27ms 15/11/02 13:25:12 DEBUG hdfs.LeaseRenewer: Lease renewed for client DFSClient_NONMAPREDUCE_2118864826_1 15/11/02 13:25:12 DEBUG hdfs.LeaseRenewer: Lease renewer daemon for [DFSClient_NONMAPREDUCE_2118864826_1] with renew id 1 executed 15/11/02 13:25:12 DEBUG azure.NativeAzureFileSystem: finalize() called. 15/11/02 13:25:12 DEBUG azure.NativeAzureFileSystem: finalize() called. 15/11/02 13:25:22 DEBUG ipc.Client: IPC Client (1803296480) connection to hadoop2-private.wdc01.infra.ripple.com/10.125.122.222:8020 from hbase: closed 15/11/02 13:25:22 DEBUG ipc.Client: IPC Client (1803296480) connection to hadoop2-private.wdc01.infra.ripple.com/10.125.122.222:8020 from hbase: stopped, remaining connections 0 15/11/02 13:25:42 DEBUG ipc.Client: The ping interval is 60000 ms. 15/11/02 13:25:42 DEBUG ipc.Client: Connecting to hadoop2-private.wdc01.infra.ripple.com/10.125.122.222:8020 15/11/02 13:25:42 DEBUG ipc.Client: IPC Client (1803296480) connection to hadoop2-private.wdc01.infra.ripple.com/10.125.122.222:8020 from hbase: starting, having connections 1 15/11/02 13:25:42 DEBUG ipc.Client: IPC Client (1803296480) connection to hadoop2-private.wdc01.infra.ripple.com/10.125.122.222:8020 from hbase sending #6 15/11/02 13:25:42 DEBUG ipc.Client: IPC Client (1803296480) connection to hadoop2-private.wdc01.infra.ripple.com/10.125.122.222:8020 from hbase got value #6 15/11/02 13:25:42 DEBUG ipc.ProtobufRpcEngine: Call: renewLease took 3ms 15/11/02 13:25:42 DEBUG hdfs.LeaseRenewer: Lease renewed for client DFSClient_NONMAPREDUCE_2118864826_1 15/11/02 13:25:42 DEBUG hdfs.LeaseRenewer: Lease renewer daemon for [DFSClient_NONMAPREDUCE_2118864826_1] with renew id 1 executed 15/11/02 13:25:42 DEBUG zookeeper.ClientCnxn: Got ping response for sessionid: 0x550c9770a820057 after 0ms 15/11/02 13:25:52 DEBUG ipc.Client: IPC Client (1803296480) connection to hadoop2-private.wdc01.infra.ripple.com/10.125.122.222:8020 from hbase: closed 15/11/02 13:25:52 DEBUG ipc.Client: IPC Client (1803296480) connection to hadoop2-private.wdc01.infra.ripple.com/10.125.122.222:8020 from hbase: stopped, remaining connections 0

 

hdfs fsck /hbase yeilds healthy

 

hbase hbck -fixMeta -fixAssignments

gets into the same loop

15/11/02 13:33:40 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop2-private.wdc01.infra.ripple.com:2181,hadoop3-private.wdc01.infra.ripple.com:2181,hadoop4-private.wdc01.infra.ripple.com:2181,hadoop5-private.wdc01.infra.ripple.com:2181,hadoop1-private.wdc01.infra.ripple.com:2181 sessionTimeout=120000 watcher=hconnection-0x25373d5c0x0, quorum=hadoop2-private.wdc01.infra.ripple.com:2181,hadoop3-private.wdc01.infra.ripple.com:2181,hadoop4-private.wdc01.infra.ripple.com:2181,hadoop5-private.wdc01.infra.ripple.com:2181,hadoop1-private.wdc01.infra.ripple.com:2181, baseZNode=/hbase
15/11/02 13:33:40 DEBUG zookeeper.ClientCnxn: zookeeper.disableAutoWatchReset is false
15/11/02 13:33:40 INFO zookeeper.ClientCnxn: Opening socket connection to server hadoop5-private.wdc01.infra.ripple.com/10.125.122.251:2181. Will not attempt to authenticate using SASL (unknown error)
15/11/02 13:33:40 INFO zookeeper.ClientCnxn: Socket connection established, initiating session, client: /10.125.122.206:37511, server: hadoop5-private.wdc01.infra.ripple.com/10.125.122.251:2181
15/11/02 13:33:40 DEBUG zookeeper.ClientCnxn: Session establishment request sent on hadoop5-private.wdc01.infra.ripple.com/10.125.122.251:2181
15/11/02 13:33:40 INFO zookeeper.ClientCnxn: Session establishment complete on server hadoop5-private.wdc01.infra.ripple.com/10.125.122.251:2181, sessionid = 0x150c9770a7d0062, negotiated timeout = 120000
15/11/02 13:33:40 DEBUG zookeeper.ZooKeeperWatcher: hconnection-0x25373d5c0x0, quorum=hadoop2-private.wdc01.infra.ripple.com:2181,hadoop3-private.wdc01.infra.ripple.com:2181,hadoop4-private.wdc01.infra.ripple.com:2181,hadoop5-private.wdc01.infra.ripple.com:2181,hadoop1-private.wdc01.infra.ripple.com:2181, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null
15/11/02 13:33:40 DEBUG zookeeper.ZooKeeperWatcher: hconnection-0x25373d5c-0x150c9770a7d0062 connected
15/11/02 13:33:40 DEBUG zookeeper.ClientCnxn: Reading reply sessionid:0x150c9770a7d0062, packet:: clientPath:null serverPath:null finished:false header:: 1,3  replyHeader:: 1,98784249776,-101  request:: '/hbase/hbaseid,F  response::
15/11/02 13:33:40 INFO client.ZooKeeperRegistry: ClusterId read in ZooKeeper is null
15/11/02 13:33:40 DEBUG client.ConnectionManager$HConnectionImplementation: clusterid came back null, using default default-cluster
15/11/02 13:33:40 DEBUG ipc.AbstractRpcClient: Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@23ab1d4a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=false, bind address=null
15/11/02 13:33:40 DEBUG zookeeper.ClientCnxn: Reading reply sessionid:0x150c9770a7d0062, packet:: clientPath:null serverPath:null finished:false header:: 2,3  replyHeader:: 2,98784249776,0  request:: '/hbase,F  response:: s{98784249191,98784249191,1446491542990,1446491542990,0,24,0,0,0,8,98784249542}
15/11/02 13:33:40 DEBUG zookeeper.ClientCnxn: Reading reply sessionid:0x150c9770a7d0062, packet:: clientPath:null serverPath:null finished:false header:: 3,4  replyHeader:: 3,98784249776,-101  request:: '/hbase/master,F  response::
15/11/02 13:33:40 DEBUG zookeeper.ZKUtil: hconnection-0x25373d5c-0x150c9770a7d0062, quorum=hadoop2-private.wdc01.infra.ripple.com:2181,hadoop3-private.wdc01.infra.ripple.com:2181,hadoop4-private.wdc01.infra.ripple.com:2181,hadoop5-private.wdc01.infra.ripple.com:2181,hadoop1-private.wdc01.infra.ripple.com:2181, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error)
15/11/02 13:33:42 DEBUG zookeeper.ClientCnxn: Reading reply sessionid:0x150c9770a7d0062, packet:: clientPath:null serverPath:null finished:false header:: 4,3  replyHeader:: 4,98784249776,0  request:: '/hbase,F  response:: s{98784249191,98784249191,1446491542990,1446491542990,0,24,0,0,0,8,98784249542}
15/11/02 13:33:42 DEBUG zookeeper.ClientCnxn: Reading reply sessionid:0x150c9770a7d0062, packet:: clientPath:null serverPath:null finished:false header:: 5,4  replyHeader:: 5,98784249776,-101  request:: '/hbase/master,F  response::
15/11/02 13:33:42 DEBUG zookeeper.ZKUtil: hconnection-0x25373d5c-0x150c9770a7d0062, quorum=hadoop2-private.wdc01.infra.ripple.com:2181,hadoop3-private.wdc01.infra.ripple.com:2181,hadoop4-private.wdc01.infra.ripple.com:2181,hadoop5-private.wdc01.infra.ripple.com:2181,hadoop1-private.wdc01.infra.ripple.com:2181, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error)
15/11/02 13:33:45 DEBUG zookeeper.ClientCnxn: Reading reply sessionid:0x150c9770a7d0062, packet:: clientPath:null serverPath:null finished:false header:: 6,3  replyHeader:: 6,98784249800,0  request:: '/hbase,F  response:: s{98784249191,98784249191,1446491542990,1446491542990,0,24,0,0,0,8,98784249542}
15/11/02 13:33:45 DEBUG zookeeper.ClientCnxn: Reading reply sessionid:0x150c9770a7d0062, packet:: clientPath:null serverPath:null finished:false header:: 7,4  replyHeader:: 7,98784249800,-101  request:: '/hbase/master,F  response::
15/11/02 13:33:45 DEBUG zookeeper.ZKUtil: hconnection-0x25373d5c-0x150c9770a7d0062, quorum=hadoop2-private.wdc01.infra.ripple.com:2181,hadoop3-private.wdc01.infra.ripple.com:2181,hadoop4-private.wdc01.infra.ripple.com:2181,hadoop5-private.wdc01.infra.ripple.com:2181,hadoop1-private.wdc01.infra.ripple.com:2181, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error)
15/11/02 13:33:50 DEBUG ipc.Client: IPC Client (1803296480) connection to hadoop2-private.wdc01.infra.ripple.com/10.125.122.222:8020 from hbase: closed
15/11/02 13:33:50 DEBUG ipc.Client: IPC Client (1803296480) connection to hadoop2-private.wdc01.infra.ripple.com/10.125.122.222:8020 from hbase: stopped, remaining connections 0
15/11/02 13:33:50 DEBUG zookeeper.ClientCnxn: Reading reply sessionid:0x150c9770a7d0062, packet:: clientPath:null serverPath:null finished:false header:: 8,3  replyHeader:: 8,98784249800,0  request:: '/hbase,F  response:: s{98784249191,98784249191,1446491542990,1446491542990,0,24,0,0,0,8,98784249542}
15/11/02 13:33:50 DEBUG zookeeper.ClientCnxn: Reading reply sessionid:0x150c9770a7d0062, packet:: clientPath:null serverPath:null finished:false header:: 9,4  replyHeader:: 9,98784249800,-101  request:: '/hbase/master,F  response::
15/11/02 13:33:50 DEBUG zookeeper.ZKUtil: hconnection-0x25373d5c-0x150c9770a7d0062, quorum=hadoop2-private.wdc01.infra.ripple.com:2181,hadoop3-private.wdc01.infra.ripple.com:2181,hadoop4-private.wdc01.infra.ripple.com:2181,hadoop5-private.wdc01.infra.ripple.com:2181,hadoop1-private.wdc01.infra.ripple.com:2181, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error)

I have no data under hbase and all tables have been destroyed

 

 

 

1 REPLY 1
Highlighted

Re: hbase master not starting

Contributor

since data was lost

ended up removing service

deleting zookeeper node /hbase

restarting cluster

and added back service from scratch