OldWALs are not deleting

[#1]

Hello Team,

I am facing issue in my cloudera cluster and HDFS used space is growing and "/hbase/oldWALs" is occupied more than 50%.

I can confirm that, HBase replication is disabled and TTL is set to 1 Min.

hbase master logcleaner ttl = 1 Min

hbase replication = false

HBase logs i can see below warn's,

WARN org apache hadoop hbase master cleaner CleanerChore: A file cleanerhostname 60000.oldLogCleaner is stopped, won't delete any more files in /nameservice1/hbase/oldWALs

And checked list of peers in hbase,

hbase(main):001:0> list_peers
PEER_ID CLUSTER_KEY STATE TABLE_CFS
0 row(s) in 0.2360 seconds

I dont see anything, Please help me with your comments.

Thanks & Regards,

Vinod

[#2]

Can someone please help me?

That would be great and thanks in advance...!!

[#3]

Hello @smdas 

Thank you so much for your response.

Followed above steps and find below details,

ls /hbase/replication
[peers, rs]

ls /hbase/replication/peers
[]

Now i have deleted replication in ZNODE and added hbase.replication as false in hbase-site.xml and restarted HBase.

Find below details after restart,

ls /hbase/replication
[peers, rs]

ls /hbase/replication/peers
[]

And now i can see it is cleared and empty under /hbase/oldWALs directory in hdfs.

But if you can observe attached screen shot, That is not enabled right?
Any differences?

Regards,

Vinod

[#4]

Thank you so much and it is resolved.

But why it is not replicating ?

When it is keeping under /oldWALs directory it should replicate right ?

Can you please give me clarification.

Best Regards,

Vinod

[#5]

Hello @kvinod 

As Cluster Replication wasn't being used based on the fact that "list_peers" isn't showing any Peer, It's likely the CleanerChore Thread wasn't performing its duties. Note that WALs are moved to oldWALs once the Last SequenceIDs of the WALs have been persisted to Disk via MemStore Flush. In other words, oldWALs being present doesn't necessarily means that the WALs are being persisted for replication. Now, the Cleanup of oldWALs is CleanerChore Thread responsibility. As we covered above, the HBase Service Restart covered the HMaster Restart, which would ensure the CleanerChore Thread is spawned afresh. 

Let me know if the above answers your queries.

- Smarak

[#6]

Thanks @smdas for clarification.

Best Regards,

Vinod