Skip to main content
SDL
SDL
Silver ⭐️⭐️
January 18, 2021
Solved

infra-solr autoDeletePeriodSeconds

Hello,

 

if the autoDeletePeriodSeconds is set to 86400, will that trigger the deletion at midnight? We are not seeing the index getting deleted and our cluster is down nightly, so just curious. Also if I donwconfig, edit this setting and upconfig, I am seeing that configuration is getting reverted next day. Not sure why it is getting reverted.

Any suggestions?

 

sdl

smdas
smdasAnswer
Gold ⭐️⭐️
October 14, 2022

Hello @SDL 

 

This is an Old Thread & I assume your Team have moved on, yet wish to Update this Post for future references. It was observed that such Overnight Restart were resetting the default CleanUp (24 Hours) set via [1] in SolrConfig.XML of the respective Solr Collection (Sample from Ranger_Audits Collection). This caused the CleanUp to be postponed on a daily basis & causes Document PileUp beyond their Expiration. If Customer are restarting the Service nightly, It's advisable to set the CleanUp from 24 Hours to a Lower Value (Like, 20 or 22 Hours). 

 

Regards, Smarak

 

[1] 

<processor class="solr.processor.DocExpirationUpdateProcessorFactory">
<int name="autoDeletePeriodSeconds">86400</int>
<str name="ttlFieldName">_ttl_</str>
<str name="expirationFieldName">_expire_at_</str>
</processor>

View in thread

5 replies

smdas
Gold ⭐️⭐️
smdas
Gold ⭐️⭐️
January 18, 2021

Hello @SDL 

 

Thanks for using Cloudera Community. As you have used InfraSolr & HDP tag, Kindly confirm the HDP Version wherein the concerned issue is being observed. 

 

Additionally, Link [1] is an excellent article on the concerned Topic & may offer any missing details.

 

- Smarak

 

[1] https://community.cloudera.com/t5/Community-Articles/Solr-TTL-Auto-Purging-Solr-Documents-Ranger-Audits/ta-p/247920

Translate
SDL
SDLAuthor
Silver ⭐️⭐️
January 18, 2021

Thanks Smarak,

 

I did follow the the link and especially the section

Old Installation; Solr Audits = enabled

It worked as I could see older indexes were deleted. I could see disk space went down.

However, for whatever reason, the ttl field reverted back next day. Not sure why/how?

Also the autoDeletePeriodSeconds = 86400 never worked, and I am suspecting that we do stop the instances nightly. I just wanted to confirm that.

 

sdl

Translate
smdas
Gold ⭐️⭐️
smdas
Gold ⭐️⭐️
January 18, 2021

Hello @SDL 

 

Thanks for the Update. Based on your comment, I assume the Configuration is being over-written owing to the nightly stop of the instance, which is using the default configuration. 

 

- Smarak

Translate
smdas
Gold ⭐️⭐️
smdasAnswer
Gold ⭐️⭐️
October 14, 2022

Hello @SDL 

 

This is an Old Thread & I assume your Team have moved on, yet wish to Update this Post for future references. It was observed that such Overnight Restart were resetting the default CleanUp (24 Hours) set via [1] in SolrConfig.XML of the respective Solr Collection (Sample from Ranger_Audits Collection). This caused the CleanUp to be postponed on a daily basis & causes Document PileUp beyond their Expiration. If Customer are restarting the Service nightly, It's advisable to set the CleanUp from 24 Hours to a Lower Value (Like, 20 or 22 Hours). 

 

Regards, Smarak

 

[1] 

<processor class="solr.processor.DocExpirationUpdateProcessorFactory">
<int name="autoDeletePeriodSeconds">86400</int>
<str name="ttlFieldName">_ttl_</str>
<str name="expirationFieldName">_expire_at_</str>
</processor>

Translate
Terms & ConditionsAccessibility statement