Tables not online

MrNicen — Mon, 23 Dec 2024 08:28:48 GMT

We're still facing issues with RIT, and now there's an additional frustrating problem: several tables are not coming online.

In the HBase meta table, the state is set to "\x08\x02", which indicates the table is "DISABLING" The region state is CLOSED, but modifying these states hasn't resolved the issue.

I've tried addressing this by using put statements and the hbck2 tool, but the problem persists. The current HBase version is 2.0.2.

Re: Tables not online

Shelton — Tue, 31 Dec 2024 17:49:59 GMT

@MrNicen

This is a very common problem where the table gets stuck in a DISABLING state.

First, please try these series of diagnostic and repair steps:

First, verify the current state:

echo "scan 'hbase:meta'" | hbase shell

Try to force the table state change using HBCK2:

# Set table to ENABLED state hbase hbck -j ./hbase-hbck2-2.0.2.jar setTableState <table_name> ENABLED

# Download HBCK2 if not already present wget https://repository.apache.org/content/repositories/releases/org/apache/hbase/hbase-hbck2/2.0.2/hbase-hbck2-2.0.2.jar

If that doesn't work, try cleaning the znode:

# Connect to ZooKeeper ./zkCli.sh -server localhost:2181 # Check the table znode ls /hbase/table/<table_name>
# Delete the table znode if present rmr /hbase/table/<table_name>

If the issue persists, try manually updating the meta table:

hbase shell # Disable table disable '<table_name>' # Wait a few seconds, then enable enable '<table_name>' # If that fails, try force disable disable_all '<table_name>'

If still stuck, try these repair commands:

# Clear the META table state echo "put 'hbase:meta', '<table_name>', 'table:state', '\x08\x00'" | hbase shell # Recreate the regions hbase hbck -j ./hbase-hbck2-2.0.2.jar assigns <table_name>

As a last resort, try a full cleanup:

# Stop HBase ./bin/stop-hbase.sh # Clear ZooKeeper data ./zkCli.sh -server localhost:2181 rmr /hbase # Remove the META directory rm -rf /hbase/data/hbase/meta # Start HBase ./bin/start-hbase.sh # Recreate the table structure hbase shell create '<table_name>', {NAME => 'cf'} # Adjust column families as needed

If none of these steps work, we can try a more aggressive approach:

Back up your data:

hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot <snapshot_name> -copy-to hdfs://backup-cluster/hbase

Try a clean META rebuild:

# Stop HBase ./bin/stop-hbase.sh # Clear META rm -rf /hbase/data/default/hbase/meta # Start HBase in repair mode env HBASE_OPTS="-XX:+UseParNewGC -XX:+UseConcMarkSweepGC" ./bin/hbase org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair # Start HBase normally ./bin/start-hbase.sh

Additional troubleshooting tips:

Check HBase logs for specific errors:

tail -f /var/log/hbase/hbase-master.log

Verify cluster health:

hbase hbck -details

Monitor region transitions:

echo "scan 'hbase:meta', {COLUMNS => 'info:regioninfo'}" | hbase shell

If you encounter any specific errors during these steps, please share them and I can provide more targeted solutions.

question Tables not online in Support Questions

Tables not online

Re: Tables not online