How WAL help in recovering region server failure?

[#1]

Hi,

Can someone provide me guidance on "How WAL help in recovering region server failure and what is WAL Spliting?"

[#2 · Accepted solution]

@Rushikesh Deshmukh

See this http://hortonworks.com/blog/apache-hbase-region-splitting-and-merging/

[#3]

really good explanation @Rushikesh Deshmukh http://www.larsgeorge.com/2010/01/hbase-architecture-101-write-ahead-log.html

[#4]

@Neeraj Sabharwal, thanks for sharing this link. Can you also provide information for how WAL help in recovering region server failure?

[#5]

Its in the 2nd link that I shared

The WAL is the lifeline that is needed when disaster strikes. Similar to a BIN log in MySQL it records all changes to the data. This is important in case something happens to the primary storage. So if the server crashes it can effectively replay that log to get everything up to where the server should have been just before the crash. It also means that if writing the record to the WAL fails the whole operation must be considered a failure.

[#6]

@Neeraj Sabharwal, thanks for useful information. I got required information. Closing this thread.

[#7]

@Rushikesh Deshmukh

best reference is HBase reference guide https://hbase.apache.org/book.html

67.6. Write Ahead Log (WAL)

67.6.1. Purpose

The Write Ahead Log (WAL) records all changes to data in HBase, to file-based storage. Under normal operations, the WAL is not needed because data changes move from the MemStore to StoreFiles. However, if a RegionServer crashes or becomes unavailable before the MemStore is flushed, the WAL ensures that the changes to the data can be replayed. If writing to the WAL fails, the entire operation to modify the data fails.

HBase uses an implementation of the WAL interface. Usually, there is only one instance of a WAL per RegionServer. The RegionServer records Puts and Deletes to it, before recording them to the MemStore for the affected [store].

The HLog

Prior to 2.0, the interface for WALs in HBase was named HLog. In 0.94, HLog was the name of the implementation of the WAL. You will likely find references to the HLog in documentation tailored to these older versions.

The WAL resides in HDFS in the /hbase/WALs/ directory (prior to HBase 0.94, they were stored in /hbase/.logs/), with subdirectories per region.

For more general information about the concept of write ahead logs, see the Wikipedia Write-Ahead Log article.

[#8]

@Artem Ervits, thanks for sharing this.

[#9]

@Rushikesh Deshmukh you're welcome, use that first before reading any other material, Lar's blog is really good but the blog post is from 2010, HBase is innovating fast and blogs tend to become dated, information does not get updated if there are changes in WAL. Your first point of reference should always be reference guide, it is one of the most well-documented material for an open source project.

[#10]

@Artem Ervits, thanks for suggession.