Hi rki_, It seems like both hbase:meta and hbase:namespace tables are not online. I am attaching the master log for your review, and if you know a way to fix this, can you check it? 2024-09-30 10:11:28,981 WARN [master/dc1-apache-hbase:16000:becomeActiveMaster] master.HMaster (HMaster.java:isRegionOnline(1373)) - hbase:meta,,1.1588230740 is NOT online; state={1588230740 state=OPEN, ts=1727422999057, server=dc1-apache-hbase.mobitel.lk,16020,1727159057270}; ServerCrashProcedures=true. Master startup cannot progress, in holding-pattern until region onlined. 2024-09-30 10:12:28,982 WARN [master/dc1-apache-hbase:16000:becomeActiveMaster] master.HMaster (HMaster.java:isRegionOnline(1373)) - hbase:meta,,1.1588230740 is NOT online; state={1588230740 state=OPEN, ts=1727422999057, server=dc1-apache-hbase.mobitel.lk,16020,1727159057270}; ServerCrashProcedures=true. Master startup cannot progress, in holding-pattern until region onlined. 2024-09-30 10:13:19,391 ERROR [ActiveMasterInitializationMonitor-1727422999267] master.MasterInitializationMonitor (MasterInitializationMonitor.java:run(67)) - Master failed to complete initialization after 900000ms. Please consider submitting a bug report including a thread dump of this process. 2024-09-30 10:13:28,982 WARN [master/dc1-apache-hbase:16000:becomeActiveMaster] master.HMaster (HMaster.java:isRegionOnline(1373)) - hbase:meta,,1.1588230740 is NOT online; state={1588230740 state=OPEN, ts=1727422999057, server=dc1-apache-hbase.mobitel.lk,16020,1727159057270}; ServerCrashProcedures=true. Master startup cannot progress, in holding-pattern until region onlined. 2024-09-30 10:13:36,668 INFO [master:store-WAL-Roller] monitor.StreamSlowMonitor (StreamSlowMonitor.java:<init>(122)) - New stream slow monitor dc1-apache-hbase.mobitel.lk%2C16000%2C1727422992087.1727671416667 2024-09-30 10:13:36,684 INFO [master:store-WAL-Roller] wal.AbstractFSWAL (AbstractFSWAL.java:logRollAndSetupWalProps(834)) - Rolled WAL /hbase/MasterData/WALs/dc1-apache-hbase.mobitel.lk,16000,1727422992087/dc1-apache-hbase.mobitel.lk%2C16000%2C1727422992087.1727670516635 with entries=0, filesize=85 B; new WAL /hbase/MasterData/WALs/dc1-apache-hbase.mobitel.lk,16000,1727422992087/dc1-apache-hbase.mobitel.lk%2C16000%2C1727422992087.1727671416667 2024-09-30 10:13:37,089 INFO [WAL-Archive-0] wal.AbstractFSWAL (AbstractFSWAL.java:archiveLogFile(815)) - Archiving hdfs://192.168.6.205:9000/hbase/MasterData/WALs/dc1-apache-hbase.mobitel.lk,16000,1727422992087/dc1-apache-hbase.mobitel.lk%2C16000%2C1727422992087.1727670516635 to hdfs://192.168.6.205:9000/hbase/MasterData/oldWALs/dc1-apache-hbase.mobitel.lk%2C16000%2C1727422992087.1727670516635 2024-09-30 10:13:37,092 INFO [WAL-Archive-0] region.MasterRegionUtils (MasterRegionUtils.java:moveFilesUnderDir(50)) - Moved hdfs://192.168.6.205:9000/hbase/MasterData/oldWALs/dc1-apache-hbase.mobitel.lk%2C16000%2C1727422992087.1727670516635 to hdfs://192.168.6.205:9000/hbase/oldWALs/dc1-apache-hbase.mobitel.lk%2C16000%2C1727422992087.1727670516635$masterlocalwal$ 2024-09-30 10:14:28,982 WARN [master/dc1-apache-hbase:16000:becomeActiveMaster] master.HMaster (HMaster.java:isRegionOnline(1373)) - hbase:meta,,1.1588230740 is NOT online; state={1588230740 state=OPEN, ts=1727422999057, server=dc1-apache-hbase.mobitel.lk,16020,1727159057270}; ServerCrashProcedures=true. Master startup cannot progress, in holding-pattern until region onlined. 2024-09-30 10:15:28,983 WARN [master/dc1-apache-hbase:16000:becomeActiveMaster] master.HMaster (HMaster.java:isRegionOnline(1373)) - hbase:meta,,1.1588230740 is NOT online; state={1588230740 state=OPEN, ts=1727422999057, server=dc1-apache-hbase.mobitel.lk,16020,1727159057270}; ServerCrashProcedures=true. Master startup cannot progress, in holding-pattern until region onlined. 2024-09-30 10:16:28,983 WARN [master/dc1-apache-hbase:16000:becomeActiveMaster] master.HMaster (HMaster.java:isRegionOnline(1373)) - hbase:meta,,1.1588230740 is NOT online; state={1588230740 state=OPEN, ts=1727422999057, server=dc1-apache-hbase.mobitel.lk,16020,1727159057270}; ServerCrashProcedures=true. Master startup cannot progress, in holding-pattern until region onlined. 2024-09-30 10:16:41,861 INFO [RS-EventLoopGroup-1-1] hbase.Server (ServerRpcConnection.java:processConnectionHeader(550)) - Connection from 192.168.6.205:57364, version=2.5.10, sasl=false, ugi=super (auth:SIMPLE), service=MasterService 2024-09-30 10:17:28,984 WARN [master/dc1-apache-hbase:16000:becomeActiveMaster] master.HMaster (HMaster.java:isRegionOnline(1373)) - hbase:meta,,1.1588230740 is NOT online; state={1588230740 state=OPEN, ts=1727422999057, server=dc1-apache-hbase.mobitel.lk,16020,1727159057270}; ServerCrashProcedures=true. Master startup cannot progress, in holding-pattern until region onlined. 2024-09-30 10:18:28,985 WARN [master/dc1-apache-hbase:16000:becomeActiveMaster] master.HMaster (HMaster.java:isRegionOnline(1373)) - hbase:meta,,1.1588230740 is NOT online; state={1588230740 state=OPEN, ts=1727422999057, server=dc1-apache-hbase.mobitel.lk,16020,1727159057270}; ServerCrashProcedures=true. Master startup cannot progress, in holding-pattern until region onlined. 2024-09-30 10:19:28,985 WARN [master/dc1-apache-hbase:16000:becomeActiveMaster] master.HMaster (HMaster.java:isRegionOnline(1373)) - hbase:meta,,1.1588230740 is NOT online; state={1588230740 state=OPEN, ts=1727422999057, server=dc1-apache-hbase.mobitel.lk,16020,1727159057270}; ServerCrashProcedures=true. Master startup cannot progress, in holding-pattern until region onlined. 2024-09-30 10:20:28,985 WARN [master/dc1-apache-hbase:16000:becomeActiveMaster] master.HMaster (HMaster.java:isRegionOnline(1373)) - hbase:meta,,1.1588230740 is NOT online; state={1588230740 state=OPEN, ts=1727422999057, server=dc1-apache-hbase.mobitel.lk,16020,1727159057270}; ServerCrashProcedures=true. Master startup cannot progress, in holding-pattern until region onlined. Thank you!
... View more
