Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

HBASE master not getting started in kerberos environment

HBASE master not getting started in kerberos environment

Explorer

2017-12-03 00:52:44,048 FATAL [hostname:60000.activeMasterManager] mas ter.HMaster: Failed to become active master
java.io.IOException: Timedout 300000ms waiting for namespace table to be ass igned
at org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableN amespaceManager.java:104)
at org.apache.hadoop.hbase.master.HMaster.initNamespace(HMaster.java :1061)
at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitiali zation(HMaster.java:840)
at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:21

ERROR [hostname,60000,1512283354427_ChoreService_1] master.BackupLogCleaner: Failed to get hbase:backup table,therefore will keep all files

5 REPLIES 5
Highlighted

Re: HBASE master not getting started in kerberos environment

Explorer

Can anyone help ? We are getting this error while starting hbase .

Highlighted

Re: HBASE master not getting started in kerberos environment

@Vishakha Agarwal,

1) Are you region servers started ?

2) Can you please run 'hbase hbck' and see if there are any INCONSISTENCIES. Try running below command if you have INCONSISTENCIES and try restarting if the below command fixed the INCONSISTENCIES

hbase hbck -fixAssignments -fixMeta -fixHdfsHoles

3) Can you please attach complete hbase master logs.

Also, attach logs for #2 if there are INCONSISTENCIES detected.

Thanks,

Aditya

Highlighted

Re: HBASE master not getting started in kerberos environment

Explorer

@Aditya Sirna

Responses to your queries :

1. Region server is UP. Master is down.

2. We cannot run 'hbase hbck' since, the master is down.

3. Attaching hbase logs.

Highlighted

Re: HBASE master not getting started in kerberos environment

@Vishakha Agarwal,

Can you please attach the logs. I dont see the logs

Highlighted

Re: HBASE master not getting started in kerberos environment

Expert Contributor
@Vishakha Agarwal

Below is the root cause for the issue from the log file given:

2017-12-02 21:29:59,054 ERROR [MASTER_SERVER_OPERATIONS-am1plccmrhdn01:60000-2] executor.EventHandler: Caught throwable while processing event M_SERVER_SHUTDOWN java.io.IOException: failed log splitting for am1plccmrhdd17.r1-core.r1.hostname.net,16020,1511029796297, will retry at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.resubmit(ServerShutdownHandler.java:357) at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:220) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) Caused by: java.io.IOException: error or interrupted while splitting logs in [hdfs://prod-gfat/apps/hbase/data/WALs/am1plccmrhdd17.r1-core.r1.hostname.net,16020,1511029796297-splitting] Task = installed = 1 done = 0 error = 0 at org.apache.hadoop.hbase.master.SplitLogManager.splitLogDistributed(SplitLogManager.java:290) at org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:393) at org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:366) at org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:288) at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:213) ... 4 more 2017-12-02 21:29:59,055 DEBUG [MASTER_SERVER_OPERATIONS-am1plccmrhdn01:60000-1] master.DeadServer: Finished processing am1plccmrhdd06.r1-core.r1.hostname.net,16020,1511029561545 2017-12-02 21:29:59,055 ERROR [MASTER_SERVER_OPERATIONS-am1plccmrhdn01:60000-4] executor.EventHandler: Caught throwable while processing event M_SERVER_SHUTDOWN java.io.IOException: Server is stopped at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:194) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) 2017-12-02 21:29:59,055 ERROR [MASTER_SERVER_OPERATIONS-am1plccmrhdn01:60000-1] executor.EventHandler: Caught throwable while processing event M_SERVER_SHUTDOWN java.io.IOException: failed log splitting for am1plccmrhdd06.r1-core.r1.hostname.net,16020,1511029561545, will retry at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.resubmit(ServerShutdownHandler.java:357) at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:220) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744)

This means that there are some region which are stuck during the splitting process which are not letting HBASE Master to come up due to WALs inconsistency.

to resolve the issue look for splitting files from (under HDFS): /apps/hbase/data/WALs/

you should be able to see: hdfs://prod-gfat/apps/hbase/data/WALs/am1plccmrhdd17.r1-core.r1.hostname.net,16020,1511029796297-splitting

You can either move this file or remove this file from this location.

Hope this helps.

Thanks

Venkat

Don't have an account?
Coming from Hortonworks? Activate your account here