Hmaster not starting Hadoop 3.3.5 / hbase 2.4.17

MEHR — Tue, 01 Aug 2023 07:49:58 GMT

Installed hadoop 3.3.5. On Single node server.

hbase-site.xml

<configuration> <property> <name>hbase.master.namespace.init.timeout</name> <value>2400000</value> </property> <property> <name>hbase.regionserver.executor.openregion.threads</name> <value>20</value> </property> <property> <name>hbase.rootdir</name> <value>hdfs://localhost:54310/hbase</value> </property> <property> <name>hbase.cluster.distributed</name> <value>true</value> </property> <property> <name>hbase.zookeeper.quorum</name> <value>localhost</value> </property> <property> <name>hbase.master.ipc.address</name> <value>localhost</value> </property> <property> <name>hbase.master.hostname</name> <value>localhost</value> </property> <property> <name>hbase.regionserver.hostname</name> <value>localhost</value> </property> <property> <name>hbase.regionserver.ipc.address</name> <value>localhost</value> </property> <property> <name>hbase.zookeeper.property.dataDir</name> <value>/usr/local/apache-zookeeper/</value> </property> <property> <name>fs.default.name</name> <value>hdfs://localhost:9000</value> </property> <property> <name>hadoop.home.dir</name> <value>/usr/local/hadoop</value> </property> <property> <name>hbase.master</name> <value>localhost:60000</value> </property> <property> <name>hbase.zookeeper.property.clientPort</name> <value>2181</value> <description>Property from ZooKeeper's config zoo.cfg. The port at which the clients will connect. </description> </property> <property> <name>hbase.master.hostname</name> <value>master.local</value>  </property> </configuration>

hbase-env.sh
export HBASE_MANAGES_ZK=false

hduser@mail:/usr/local/hbase/conf> jps 22418 DataNode 22770 ResourceManager 22930 NodeManager 1925 Jps 327 QuorumPeerMain 856 HRegionServer 22237 NameNode

Getting Failure

2023-08-01 00:22:36,806 INFO [main] zookeeper.ZooKeeper: Client environment:java.library.path=/usr/local/hadoop/lib 2023-08-01 00:22:36,806 INFO [main] zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp 2023-08-01 00:22:36,806 INFO [main] zookeeper.ZooKeeper: Client environment:java.compiler=<NA> 2023-08-01 00:22:36,806 INFO [main] zookeeper.ZooKeeper: Client environment:os.name=Linux 2023-08-01 00:22:36,806 INFO [main] zookeeper.ZooKeeper: Client environment:os.arch=amd64 2023-08-01 00:22:36,806 INFO [main] zookeeper.ZooKeeper: Client environment:os.version=5.3.18-22-default 2023-08-01 00:22:36,806 INFO [main] zookeeper.ZooKeeper: Client environment:user.name=hduser 2023-08-01 00:22:36,806 INFO [main] zookeeper.ZooKeeper: Client environment:user.home=/home/hduser 2023-08-01 00:22:36,806 INFO [main] zookeeper.ZooKeeper: Client environment:user.dir=/usr/local/hbase/bin 2023-08-01 00:22:36,807 INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=90000 watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@3a6f2de3 2023-08-01 00:22:36,814 INFO [main-SendThread(localhost:2181)] zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) 2023-08-01 00:22:36,818 INFO [main-SendThread(localhost:2181)] zookeeper.ClientCnxn: Socket connection established to localhost/127.0.0.1:2181, initiating session 2023-08-01 00:22:39,813 INFO [main-SendThread(localhost:2181)] zookeeper.ClientCnxn: Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x1006c4c36560000, negotiated timeout = 40000 2023-08-01 00:22:39,855 INFO [main] zookeeper.RecoverableZooKeeper: Node /hbase already exists and this is not a retry 2023-08-01 00:22:39,875 INFO [RpcServer.responder] ipc.RpcServer: RpcServer.responder: starting 2023-08-01 00:22:39,875 INFO [RpcServer.listener,port=60000] ipc.RpcServer: RpcServer.listener,port=60000: starting 2023-08-01 00:22:39,907 INFO [master:roadrunner-com:60000] mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-08-01 00:22:39,926 INFO [master:roadrunner-com:60000] http.HttpServer: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter) 2023-08-01 00:22:39,927 INFO [master:roadrunner-com:60000] http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context master 2023-08-01 00:22:39,927 INFO [master:roadrunner-com:60000] http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context static 2023-08-01 00:22:39,931 INFO [master:roadrunner-com:60000] http.HttpServer: Jetty bound to port 60010 2023-08-01 00:22:39,931 INFO [master:roadrunner-com:60000] mortbay.log: jetty-6.1.26 2023-08-01 00:22:40,084 INFO [master:roadrunner-com:60000] mortbay.log: Started SelectChannelConnector@0.0.0.0:60010 2023-08-01 00:22:40,135 DEBUG [main-EventThread] master.ActiveMasterManager: A master is now available 2023-08-01 00:22:40,137 INFO [master:roadrunner-com:60000] master.ActiveMasterManager: Registered Active Master=roadrunner-com.msg.pkvw.co.charter.net,60000,1690874556592 2023-08-01 00:22:40,140 INFO [master:roadrunner-com:60000] Configuration.deprecation: fs.default.name is deprecated. Instead, use fs.defaultFS 2023-08-01 00:22:40,159 FATAL [master:roadrunner-com:60000] master.HMaster: Unhandled exception. Starting shutdown. java.net.ConnectException: Call From mail/47.43.26.44 to localhost:54310 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730) at org.apache.hadoop.ipc.Client.call(Client.java:1351) at org.apache.hadoop.ipc.Client.call(Client.java:1300) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) at com.sun.proxy.$Proxy10.setSafeMode(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy10.setSafeMode(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.setSafeMode(ClientNamenodeProtocolTranslatorPB.java:561) at org.apache.hadoop.hdfs.DFSClient.setSafeMode(DFSClient.java:2146) at org.apache.hadoop.hdfs.DistributedFileSystem.setSafeMode(DistributedFileSystem.java:983) at org.apache.hadoop.hdfs.DistributedFileSystem.setSafeMode(DistributedFileSystem.java:967) at org.apache.hadoop.hbase.util.FSUtils.isInSafeMode(FSUtils.java:474) at org.apache.hadoop.hbase.util.FSUtils.waitOnSafeMode(FSUtils.java:921) at org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:443) at org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:154) at org.apache.hadoop.hbase.master.MasterFileSystem.<init>(MasterFileSystem.java:130) at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:898) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:698) at java.lang.Thread.run(Thread.java:750) Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:715) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:547) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:642) at org.apache.hadoop.ipc.Client$Connection.access$2600(Client.java:314) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1399) at org.apache.hadoop.ipc.Client.call(Client.java:1318) ... 22 more 2023-08-01 00:22:40,160 INFO [master:roadrunner-com:60000] master.HMaster: Aborting 2023-08-01 00:22:40,160 DEBUG [master:roadrunner-com:60000] master.HMaster: Stopping service threads 2023-08-01 00:22:40,161 INFO [master:roadrunner-com:60000] ipc.RpcServer: Stopping server on 60000 2023-08-01 00:22:40,161 INFO [RpcServer.listener,port=60000] ipc.RpcServer: RpcServer.listener,port=60000: stopping 2023-08-01 00:22:40,161 INFO [RpcServer.responder] ipc.RpcServer: RpcServer.responder: stopped 2023-08-01 00:22:40,161 INFO [RpcServer.responder] ipc.RpcServer: RpcServer.responder: stopping 2023-08-01 00:22:40,161 INFO [master:roadrunner-com:60000] master.HMaster: Stopping infoServer 2023-08-01 00:22:40,171 INFO [master:roadrunner-com:60000] mortbay.log: Stopped SelectChannelConnector@0.0.0.0:60010 2023-08-01 00:22:40,303 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down 2023-08-01 00:22:40,303 INFO [master:roadrunner-com:60000] zookeeper.ZooKeeper: Session: 0x1006c4c36560000 closed 2023-08-01 00:22:40,303 INFO [master:roadrunner-com:60000] master.HMaster: HMaster main thread exiting 2023-08-01 00:22:40,303 ERROR [main] master.HMasterCommandLine: Master exiting java.lang.RuntimeException: HMaster Aborted at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:207) at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:135) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:127) at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:3164)

Re: Hmaster not starting Hadoop 3.3.5 / hbase 2.4.17

VidyaSargur — Tue, 01 Aug 2023 08:50:24 GMT

@MEHR, Welcome to our community! To help you get the best possible answer, I have tagged our HBase experts @smdas @Asok who may be able to assist you further.

Please feel free to provide any additional information or details about your query, and we hope that you will find a satisfactory solution to your question.

Re: Hmaster not starting Hadoop 3.3.5 / hbase 2.4.17

Asok — Tue, 01 Aug 2023 16:26:14 GMT

Please ensure that the namenode is up and out of safe mode.

hdfs dfsadmin -safemode get

Re: Hmaster not starting Hadoop 3.3.5 / hbase 2.4.17

MEHR — Tue, 01 Aug 2023 19:16:33 GMT

hduser@mail:~> hdfs dfsadmin -safemode get
2023-08-01 12:13:01,180 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Safe mode is OFF
Namenode is up

hduser@mail:/usr/local/hbase/logs> vi hbase-hduser-master-mail.log
hduser@mail:/usr/local/hbase/logs> jps
7472 Jps
22418 DataNode
22770 ResourceManager
7286 HRegionServer
327 QuorumPeerMain
22237 NameNode

Re: Hmaster not starting Hadoop 3.3.5 / hbase 2.4.17

MEHR — Tue, 01 Aug 2023 23:30:52 GMT

Still HMaster is not up and running

Re: Hmaster not starting Hadoop 3.3.5 / hbase 2.4.17

MEHR — Thu, 03 Aug 2023 01:48:41 GMT

@Asok
Safe mode is OFF
Namenode is up

Still HMaster is not up and running.
Please can you let me know how to fix this

question Re: Hmaster not starting Hadoop 3.3.5 / hbase 2.4.17 in Support Questions

Hmaster not starting Hadoop 3.3.5 / hbase 2.4.17

Re: Hmaster not starting Hadoop 3.3.5 / hbase 2.4.17

Re: Hmaster not starting Hadoop 3.3.5 / hbase 2.4.17

Re: Hmaster not starting Hadoop 3.3.5 / hbase 2.4.17

Re: Hmaster not starting Hadoop 3.3.5 / hbase 2.4.17

Re: Hmaster not starting Hadoop 3.3.5 / hbase 2.4.17