question OldWALs are not deleting in Support Questions

OldWALs are not deleting

kvinod — Mon, 07 Dec 2020 12:07:30 GMT

Hello Team,

I am facing issue in my cloudera cluster and HDFS used space is growing and "/hbase/oldWALs" is occupied more than 50%.

I can confirm that, HBase replication is disabled and TTL is set to 1 Min.

hbase master logcleaner ttl = 1 Min

hbase replication = false

HBase logs i can see below warn's,

WARN org apache hadoop hbase master cleaner CleanerChore: A file cleanerhostname 60000.oldLogCleaner is stopped, won't delete any more files in /nameservice1/hbase/oldWALs

And checked list of peers in hbase,

hbase(main):001:0> list_peers
PEER_ID CLUSTER_KEY STATE TABLE_CFS
0 row(s) in 0.2360 seconds

I dont see anything, Please help me with your comments.

Thanks & Regards,

Vinod

Re: OldWALs are not deleting

kvinod — Tue, 08 Dec 2020 06:59:53 GMT

Can someone please help me?

That would be great and thanks in advance...!!

Re: OldWALs are not deleting

smdas — Tue, 08 Dec 2020 11:23:45 GMT

Hello @kvinod

Thanks for using Cloudera Community. Your concern is HBase OldWALs on HDFS Path "/hbase/oldWALs" are occupying a lot of space. HBase Replication isn't being used & TTL is set to 1 Minute.

The HMaster Trace Logs capture the CleanerChore with verbose logging, yet I wish to check if you have tried the following 2 options:

1. Restart the HMaster Service to confirm if any issues with CleanerChore,

2. The Parameter "hbase.replication" is set correctly to False via the Steps shared under Section [1].

3. If the "/hbase/replication" has any entries. If no Replication is utilised (HBase Replication or Lily Indexer), Try removing the "/hbase/replication" ZNode & restart the HMaster Service.

- Smarak

[1] CM=> HBase=> Configuration=> Advanced=> HBase Service Advanced Configuration Snippet (Safety Valve) for hbase-site.xml

Re: OldWALs are not deleting

kvinod — Tue, 08 Dec 2020 13:29:16 GMT

Hello @smdas

Thank you so much for your response.

Followed above steps and find below details,

ls /hbase/replication
[peers, rs]

ls /hbase/replication/peers
[]

Now i have deleted replication in ZNODE and added hbase.replication as false in hbase-site.xml and restarted HBase.

Find below details after restart,

ls /hbase/replication
[peers, rs]

ls /hbase/replication/peers
[]

And now i can see it is cleared and empty under /hbase/oldWALs directory in hdfs.

But if you can observe attached screen shot, That is not enabled right?
Any differences?

Regards,

Vinod

Re: OldWALs are not deleting

smdas — Tue, 08 Dec 2020 13:48:54 GMT

Hello @kvinod

Thanks for the Update. The Replication ZNode being created is expected after restart.

The Checkbox concerning HBase Replication being left unchecked indicates Replication being disabled yet I have observed couple of cases wherein a CM Config wasn't passed to Service Level, causing certain unexpected behaviour. The explicit addition of the Parameter was to ensure the Service (HBase in this case) is aware of the Configuration.

Or, Master Restart (Performed via HBase Restart) may have resolved the issue, by spawning a new CleanerChoreThread. As such, the issue is likely with the HBase Service being unaware of Replication being disabled or HMaster CleanerChore Thread. By explicitly adding the HBase Replication as False & restarting the HBase Service, We covered the 2 possibilities.

- Smarak

Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Re: OldWALs are not deleting

kvinod — Tue, 08 Dec 2020 14:51:34 GMT

Thank you so much and it is resolved.

But why it is not replicating ?

When it is keeping under /oldWALs directory it should replicate right ?

Can you please give me clarification.

Best Regards,

Vinod

Re: OldWALs are not deleting

smdas — Tue, 08 Dec 2020 17:12:58 GMT

Hello @kvinod

As Cluster Replication wasn't being used based on the fact that "list_peers" isn't showing any Peer, It's likely the CleanerChore Thread wasn't performing its duties. Note that WALs are moved to oldWALs once the Last SequenceIDs of the WALs have been persisted to Disk via MemStore Flush. In other words, oldWALs being present doesn't necessarily means that the WALs are being persisted for replication. Now, the Cleanup of oldWALs is CleanerChore Thread responsibility. As we covered above, the HBase Service Restart covered the HMaster Restart, which would ensure the CleanerChore Thread is spawned afresh.

Let me know if the above answers your queries.

- Smarak

Re: OldWALs are not deleting

kvinod — Wed, 09 Dec 2020 07:18:38 GMT

Thanks @smdas for clarification.

Best Regards,

Vinod