Tables not online

[#1]

We're still facing issues with RIT, and now there's an additional frustrating problem: several tables are not coming online.

In the HBase meta table, the state is set to "\x08\x02", which indicates the table is "DISABLING" The region state is CLOSED, but modifying these states hasn't resolved the issue.

I've tried addressing this by using put statements and the hbck2 tool, but the problem persists. The current HBase version is 2.0.2.

[#2]

@MrNicen 

This is a very common problem where the table gets stuck in a DISABLING state.

First, please try these  series of diagnostic and repair steps:

1. First, verify the current state:

Spoiler

echo "scan 'hbase:meta'" | hbase shell

1. Try to force the table state change using HBCK2:

Spoiler

# Set table to ENABLED state hbase hbck -j ./hbase-hbck2-2.0.2.jar setTableState <table_name> ENABLED

# Download HBCK2 if not already present wget https://repository.apache.org/content/repositories/releases/org/apache/hbase/hbase-hbck2/2.0.2/hbase...

1. If that doesn't work, try cleaning the znode:

Spoiler

# Connect to ZooKeeper ./zkCli.sh -server localhost:2181 # Check the table znode ls /hbase/table/<table_name>
# Delete the table znode if present rmr /hbase/table/<table_name>

1. If the issue persists, try manually updating the meta table:

Spoiler

hbase shell # Disable table disable '<table_name>' # Wait a few seconds, then enable enable '<table_name>' # If that fails, try force disable disable_all '<table_name>'

1. If still stuck, try these repair commands:

Spoiler

# Clear the META table state echo "put 'hbase:meta', '<table_name>', 'table:state', '\x08\x00'" | hbase shell # Recreate the regions hbase hbck -j ./hbase-hbck2-2.0.2.jar assigns <table_name>

1. As a last resort, try a full cleanup:

Spoiler

# Stop HBase ./bin/stop-hbase.sh # Clear ZooKeeper data ./zkCli.sh -server localhost:2181 rmr /hbase # Remove the META directory rm -rf /hbase/data/hbase/meta # Start HBase ./bin/start-hbase.sh # Recreate the table structure hbase shell create '<table_name>', {NAME => 'cf'} # Adjust column families as needed

If none of these steps work, we can try a more aggressive approach:

1. Back up your data:

Spoiler

hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot <snapshot_name> -copy-to hdfs://backup-cluster/hbase

Try a clean META rebuild:

Spoiler

# Stop HBase ./bin/stop-hbase.sh # Clear META rm -rf /hbase/data/default/hbase/meta # Start HBase in repair mode env HBASE_OPTS="-XX:+UseParNewGC -XX:+UseConcMarkSweepGC" ./bin/hbase org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair # Start HBase normally ./bin/start-hbase.sh

Additional troubleshooting tips:

1. Check HBase logs for specific errors:

Spoiler

tail -f /var/log/hbase/hbase-master.log

Verify cluster health:

Spoiler

hbase hbck -details

1. Monitor region transitions:

Spoiler

echo "scan 'hbase:meta', {COLUMNS => 'info:regioninfo'}" | hbase shell

If you encounter any specific errors during these steps, please share them and I can provide more targeted solutions.