Support Questions

Find answers, ask questions, and share your expertise

HBase - Region in Transition

avatar
Explorer

HBase keep having region in transition: 

 

Regions in Transition

Region State RIT time (ms)

1588230740hbase:meta,,1.1588230740 state=FAILED_OPEN, ts=Thu Apr 23 12:15:49 ICT 2015 (8924s ago), server=02slave.mabu.com,60020,14297655798238924009
Total number of Regions in Transition for more than 60000 milliseconds1 
Total number of Regions in Transition1

 

I've try "sudo -u hbase hbase hbck -repair" and also "unassign 'hbase:meta,,1.1588230740'" but still can't fix the problem.

1 ACCEPTED SOLUTION

avatar
Community Manager

1. Stop HBase
2. Move your original /hbase back into place
3. Use a zookeeper cli such as "hbase zkcli"[1] and run "rmr /hbase" to delete the HBase znodes
4. Restart HBase. It will recreate the znodes

 

If Hbase fails to start after this, you can always try the offline Meta repair:
hbase org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair

 

Also check for inconsistencies after HBase is up.  As the hbase user, run "hbase hbck -details". If there are inconsistencies reported, normally I would use the "ERROR" messages from the hbck output to decide on the best repair method, but since you were willing to start over just run "hbase hbck -repair".

 

If the above fails, you can always try the offline Meta repair:

hbase org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair

 

[1] http://hbase.apache.org/book.html#trouble.tools
[2] http://hbase.apache.org/book.html#hbck.in.depth



David Wilder, Community Manager


Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Learn more about the Cloudera Community:

Terms of Service

Community Guidelines

How to use the forum

View solution in original post

18 REPLIES 18

avatar
Rising Star

Sounds like a bug, check the bugs reported about transitions and compare with the version you are using.

If repair doesn't work, the only solution I see is to do a snapshot, truncate the table and then import the snapshot (maybe try to import into another table before you truncate the main one).

avatar
Explorer

Thanks for reply Gonzalo,

I can't do anything with hbase right now (can't disable, drop table, can't event view sample table on hue), everything just stuct, so i just manually delete all table data on HDFS, just keep the default(sample) table, but it still won't work.

avatar
Rising Star

If restarting the master din't do anything I would say the hbase znode is messed up in zookeeper.

If you have nothing to lose stop hbase, delete the znode in zookeeper, delete the hbase folder in hdfs and start hbase.

avatar
Explorer

Thanks Gonzalo,

 

I've been try to delete the "/hbase" folder ( just move it to /tmp), but when i restart Hbase, the hbase master can't start, as i remember is because of user authentication of "/" belong to HDFS, and i don't want to chown "/" to hbase.

Even after i create a new "/hbase" and chow it to hbase:hbase, the hbase master still won't start unless i move back the old "/hbase"

 

About znode in zookeeper, i really don't know much about it, i just know my ZooKeeper Znode Parent is "/hbase", do i just delete this folder or i have to delete something elsse ? 

avatar
Community Manager

1. Stop HBase
2. Move your original /hbase back into place
3. Use a zookeeper cli such as "hbase zkcli"[1] and run "rmr /hbase" to delete the HBase znodes
4. Restart HBase. It will recreate the znodes

 

If Hbase fails to start after this, you can always try the offline Meta repair:
hbase org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair

 

Also check for inconsistencies after HBase is up.  As the hbase user, run "hbase hbck -details". If there are inconsistencies reported, normally I would use the "ERROR" messages from the hbck output to decide on the best repair method, but since you were willing to start over just run "hbase hbck -repair".

 

If the above fails, you can always try the offline Meta repair:

hbase org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair

 

[1] http://hbase.apache.org/book.html#trouble.tools
[2] http://hbase.apache.org/book.html#hbck.in.depth



David Wilder, Community Manager


Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Learn more about the Cloudera Community:

Terms of Service

Community Guidelines

How to use the forum

avatar
Explorer

Thanks denloe,

 

The command "hbase hbck -repair" solve 6 inconsistencies. And the step that delete znodes make this command working, i think, because i've been trying "hbase hbck -repair" before but it just stuck. HBase working fine now, thank you.

avatar
Explorer

I have tried your steps, but I still have inconsistancies and hbck -repair does not work. My inconsistancies are with data tables and not with META.

I get the following error message:

 

INFO util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned

 

and it eventually times out. I am using CDH 5.4.4 with Hbase 1.0.0. I cannot do anything on Hbase (count, scan etc.). 

avatar
Rising Star

@EugeneM wrote:

I have tried your steps, but I still have inconsistancies and hbck -repair does not work. My inconsistancies are with data tables and not with META.

I get the following error message:

 

INFO util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned

 

and it eventually times out. I am using CDH 5.4.4 with Hbase 1.0.0. I cannot do anything on Hbase (count, scan etc.). 


 

 

In your case, if "hdfs fsck" doesn't fix the files; you are going to have to delete the corrupted hdfs table files.

If you can load the data again, probably the best thing is to delete the /hbase directory in hdfs altogether, restart and load the data again.

avatar
Explorer

Thank you Gonzalo. That was exactly what I was not hoping to hear, but seeing that I could not use hbase, I had to do it. So, after deleting all the files/tables with inconsistancies, I got hbase to work again. Thank you for your help.