question hbase 2.0 table stuck in 'DISABLING' state and region is stuck in 'OPENING' state in Support Questions

hbase 2.0 table stuck in 'DISABLING' state and region is stuck in 'OPENING' state

jack_madden — Sun, 13 Jan 2019 12:24:53 GMT

I have a table stuck in 'DISABLING' state and one of the four table regions is stuck in 'OPENING' state. Hbase 2.0 hbck doesn't allow fixing tables.

./hbase zkcli, ls /hbase-unsecure/table/<table name> doesn't show anything

restarting the hbase master, regionservers, and zookeeper doesn't resolve

How do I fix this issue.

Re: hbase 2.0 table stuck in 'DISABLING' state and region is stuck in 'OPENING' state

jack_madden — Sun, 13 Jan 2019 12:28:02 GMT

hbase(main):002:0> is_enabled 'EDA_CONTRACTS_TEMP'
false
Took 0.1362 seconds
=> false
hbase(main):003:0> is_disabled 'EDA_CONTRACTS_TEMP'
false
Took 0.0290 seconds
=> 1

./hbase hbck -details
Number of regions in transition: 1
  EDA_CONTRACTS_TEMP,,1545342660748.52fdc258feeeae9285551f2cc231d841. state=OPENING, ts=Thu Jan 10 19:07:32 MST 2019 (177746s ago), server=null

Re: hbase 2.0 table stuck in 'DISABLING' state and region is stuck in 'OPENING' state

jack_madden — Mon, 14 Jan 2019 03:52:03 GMT

When I try to unassign the region (close) I see this in the log. Is there a way to manually set the state of the region or force it to close.

2019-01-13 12:46:06,272 WARN [PEWorker-16] assignment.RegionTransitionProcedure: Failed transition, suspend 2048secs pid=2931, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=EDA_CONTRACTS_TEMP, region=52fdc258feeeae9285551f2cc231d841, server=hadoop-2,16020,1546877712972; rit=OPENING, location=hadoop-2,16020,1546877712972; waiting on rectified condition fixed by other Procedure or operator intervention org.apache.hadoop.hbase.exceptions.UnexpectedStateException: Expected [SPLITTING, SPLIT, MERGING, OPEN, CLOSING] so could move to CLOSING but current state=OPENING

Re: hbase 2.0 table stuck in 'DISABLING' state and region is stuck in 'OPENING' state

jack_madden — Mon, 14 Jan 2019 23:03:24 GMT

This was a temp table. I wasn't able to drop the table b/c it was in a 'DISABLING' state, not 'DISABLED'. I was able to figure out a way to manually remove the table.

Remove the table directory in hdfs, /apps/hbase/data/data/default/<table_name>.

Remove the rows in hbase:meta that referenced the table, deleteall 'hbase:meta', '<row_name>'

Stop hbase, zookeeper. Start zookeeper, hbase.

Re: hbase 2.0 table stuck in 'DISABLING' state and region is stuck in 'OPENING' state

jack_madden — Tue, 15 Jan 2019 00:01:06 GMT

This was a temp table. I wasn't able to drop the table b/c it was in a 'DISABLING' state, not 'DISABLED'. I was able to figure out a way to manually remove the table.

Remove the table directory in hdfs, /apps/hbase/data/data/default/<table_name>.

Remove the rows in hbase:meta that referenced the table, deleteall 'hbase:meta', '<row_name>'

Stop hbase, zookeeper. Start zookeeper, hbase.

Re: hbase 2.0 table stuck in 'DISABLING' state and region is stuck in 'OPENING' state

sheshk11 — Sat, 02 Mar 2019 20:44:01 GMT

Hi,

No need to do any deletion. Just follow the steps here to resolve the issue without any data loss: https://community.hortonworks.com/questions/242343/hbase-table-is-stuck-in-disabling-state-neither-en.html

Thanks,

Shesh