question Re: oldWALs folder growing too much.... in Support Questions

oldWALs folder growing too much....

silvio_delval — Mon, 20 Jun 2016 19:01:51 GMT

Hello,

I have been looking for some information about this, but I did not found anything. Our HDFS file system is growing a lot, and I can see about 60% of space is in "oldWALs" folder. Why is it growing so much? How can I get rid of this?

We deployed our cluster with Ambari with almost no customization. We don't have replication to other clusters or so on...everything quite standard.

Best regards,

Silvio

Re: oldWALs folder growing too much....

sshimpi — Mon, 20 Jun 2016 19:05:15 GMT

@Silvio del Valhbase keeps the replication WAL logs until the peer is removed.

remove_peer can fixed the problem.

This can help out - http://stackoverflow.com/questions/28725364/hbase-oldwals-what-it-is-and-how-can-i-clean-it

Re: oldWALs folder growing too much....

silvio_delval — Mon, 20 Jun 2016 19:51:26 GMT

From "list_peers" command we get "0 row(s) in 0.1060 seconds", I tested that before asking and I don't know wht's happening. Any other hints?

Re: oldWALs folder growing too much....

sshimpi — Mon, 20 Jun 2016 20:14:39 GMT

@Silvio del Val

Please post the output of "ls /hbase/replication" (and sub-znodes under it) via your "zookeeper-client" shell command. If there are any znodes under there, you will need to clean them up with rm/rmr in the same shell. Once done, try restarting the HMaster and the cleaner should be able to wipe it away.

If you are a hundred percent sure you do not have any form of replication whatsoever in use, nor have any snapshots, you may choose to also delete the oldWAL directory files manually.

Re: oldWALs folder growing too much....

tyu — Mon, 20 Jun 2016 20:16:45 GMT

There are currently two services which may keep the files in the archive directory. First is a TTL process, which ensures that the WAL files are kept at least for 10 min. This is controlled by hbase.master.logcleaner.ttl configuration property in master.

The other one is replication. If you had replication setup before, the replication processes will hang on to the WAL files until they are replicated. Even if you disabled the replication, the files are still referenced.

You can look at the logs of master from classes (LogCleaner, TimeToLiveLogCleaner, ReplicationLogCleaner) to see whether any exception was thrown.

Re: oldWALs folder growing too much....

tyu — Mon, 20 Jun 2016 20:21:52 GMT

Disabled replication would still hold on to the WAL files because, it guarantees to not lose data between disable and enable. You can execute remove_peer, which frees up the WAL files eligible for deletion. When you re-add replication peer again, the replication would start from the current status, versus if you re-enable a peer, it will continue from where it left off.

Re: oldWALs folder growing too much....

silvio_delval — Mon, 20 Jun 2016 20:56:12 GMT

Hi,

This is the output you request:

[zk: localhost:2181(CONNECTED) 4] ls /hbase/replication

[peers, rs]

[zk: localhost:2181(CONNECTED) 5] ls /hbase/replication/peers

[]

No replication to other peers

Re: oldWALs folder growing too much....

silvio_delval — Mon, 20 Jun 2016 21:16:46 GMT

Well, "archive" folder under /apps/hbase/data" remains "under control" and doesn't grow. My problem is "oldWALs" under same path. I don't have any kind of replication

Re: oldWALs folder growing too much....

silvio_delval — Mon, 20 Jun 2016 21:18:15 GMT

No "peers" at all here

Re: oldWALs folder growing too much....

silvio_delval — Tue, 21 Jun 2016 14:51:06 GMT

Ok, issue is resolved. I had to explicitly add the custom property "hbase.replication=false" to hbase-site.xml (although we have no replication at all and no peers configured) and restart HBase masters. After this, about 50 TB of data in oldWALs folder were deleted automatically in about 10 minutes 🙂

Thank you very much to all of you, you helped me a lot 🙂

Re: oldWALs folder growing too much....

sanketplus — Mon, 17 Apr 2017 17:15:52 GMT

I am in exact same situation as you were in. Adding replaction=false property did not help and it's still growing. and strange part is it's been growing for last 7 days only. It was normal before that. Any clues what's happening here?

Re: oldWALs folder growing too much....

sanketplus — Tue, 18 Apr 2017 21:27:41 GMT

https://community.hortonworks.com/questions/97184/oldwals-not-getting-cleared-even-with-no-replicati.html