Skip to main content
kvinod
kvinod
Gold ⭐️
December 7, 2020
Solved

OldWALs are not deleting

Hello Team,

 

I am facing issue in my cloudera cluster and HDFS used space is growing and "/hbase/oldWALs" is occupied more than 50%.

 

I can confirm that, HBase replication is disabled and TTL is set to 1 Min.

hbase master logcleaner ttl = 1 Min
hbase replication = false
 
HBase logs i can see below warn's,
WARN org apache hadoop hbase master cleaner CleanerChore: A file cleanerhostname 60000.oldLogCleaner is stopped, won't delete any more files in /nameservice1/hbase/oldWALs

 

And checked list of peers in hbase,

hbase(main):001:0> list_peers
PEER_ID CLUSTER_KEY STATE TABLE_CFS
0 row(s) in 0.2360 seconds

 

I dont see anything, Please help me with your comments.

 

 

Thanks & Regards,

Vinod

smdas
smdasAnswer
Gold ⭐️⭐️
December 8, 2020

Hello @kvinod 

 

Thanks for using Cloudera Community. Your concern is HBase OldWALs on HDFS Path "/hbase/oldWALs" are occupying a lot of space. HBase Replication isn't being used & TTL is set to 1 Minute. 

 

The HMaster Trace Logs capture the CleanerChore with verbose logging, yet I wish to check if you have tried the following 2 options:

1. Restart the HMaster Service to confirm if any issues with CleanerChore,

2. The Parameter "hbase.replication" is set correctly to False via the Steps shared under Section [1].

3. If the "/hbase/replication" has any entries. If no Replication is utilised (HBase Replication or Lily Indexer), Try removing the "/hbase/replication" ZNode & restart the HMaster Service.

 

- Smarak

 

[1] CM=> HBase=> Configuration=> Advanced=> HBase Service Advanced Configuration Snippet (Safety Valve) for hbase-site.xml

View in thread

7 replies

kvinod
kvinodAuthor
Gold ⭐️
December 8, 2020

Can someone please help me?

That would be great and thanks in advance...!!

Translate
smdas
Gold ⭐️⭐️
smdasAnswer
Gold ⭐️⭐️
December 8, 2020

Hello @kvinod 

 

Thanks for using Cloudera Community. Your concern is HBase OldWALs on HDFS Path "/hbase/oldWALs" are occupying a lot of space. HBase Replication isn't being used & TTL is set to 1 Minute. 

 

The HMaster Trace Logs capture the CleanerChore with verbose logging, yet I wish to check if you have tried the following 2 options:

1. Restart the HMaster Service to confirm if any issues with CleanerChore,

2. The Parameter "hbase.replication" is set correctly to False via the Steps shared under Section [1].

3. If the "/hbase/replication" has any entries. If no Replication is utilised (HBase Replication or Lily Indexer), Try removing the "/hbase/replication" ZNode & restart the HMaster Service.

 

- Smarak

 

[1] CM=> HBase=> Configuration=> Advanced=> HBase Service Advanced Configuration Snippet (Safety Valve) for hbase-site.xml

Translate
kvinod
kvinodAuthor
Gold ⭐️
December 8, 2020

Hello @smdas 

 

Thank you so much for your response.

Followed above steps and find below details,

 

ls /hbase/replication
[peers, rs]

ls /hbase/replication/peers
[]


Now i have deleted replication in ZNODE and added hbase.replication as false in hbase-site.xml and restarted HBase.

Find below details after restart,

ls /hbase/replication
[peers, rs]

ls /hbase/replication/peers
[]

 

And now i can see it is cleared and empty under /hbase/oldWALs directory in hdfs.

 

But if you can observe attached screen shot, That is not enabled right?
Any differences?

 

HBase_replication.PNG

 

Regards,

Vinod

 

Translate
smdas
Gold ⭐️⭐️
smdas
Gold ⭐️⭐️
December 8, 2020

Hello @kvinod 

 

Thanks for the Update. The Replication ZNode being created is expected after restart.

 

The Checkbox concerning HBase Replication being left unchecked indicates Replication being disabled yet I have observed couple of cases wherein a CM Config wasn't passed to Service Level, causing certain unexpected behaviour. The explicit addition of the Parameter was to ensure the Service (HBase in this case) is aware of the Configuration.

 

Or, Master Restart (Performed via HBase Restart) may have resolved the issue, by spawning a new CleanerChoreThread. As such, the issue is likely with the HBase Service being unaware of Replication being disabled or HMaster CleanerChore Thread. By explicitly adding the HBase Replication as False & restarting the HBase Service, We covered the 2 possibilities. 

 

- Smarak

Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Translate
Terms & ConditionsAccessibility statement