infra-solr autoDeletePeriodSeconds

[#1]

Hello,

if the autoDeletePeriodSeconds is set to 86400, will that trigger the deletion at midnight? We are not seeing the index getting deleted and our cluster is down nightly, so just curious. Also if I donwconfig, edit this setting and upconfig, I am seeing that configuration is getting reverted next day. Not sure why it is getting reverted.

Any suggestions?

sdl

[#2]

Hello @SDL 

Thanks for using Cloudera Community. As you have used InfraSolr & HDP tag, Kindly confirm the HDP Version wherein the concerned issue is being observed. 

Additionally, Link [1] is an excellent article on the concerned Topic & may offer any missing details.

- Smarak

[1] https://community.cloudera.com/t5/Community-Articles/Solr-TTL-Auto-Purging-Solr-Documents-Ranger-Aud...

[#3]

Thanks Smarak,

I did follow the the link and especially the section

Old Installation; Solr Audits = enabled

It worked as I could see older indexes were deleted. I could see disk space went down.

However, for whatever reason, the ttl field reverted back next day. Not sure why/how?

Also the autoDeletePeriodSeconds = 86400 never worked, and I am suspecting that we do stop the instances nightly. I just wanted to confirm that.

sdl

[#4]

Hello @SDL 

Thanks for the Update. Based on your comment, I assume the Configuration is being over-written owing to the nightly stop of the instance, which is using the default configuration. 

- Smarak

[#5]

Thanks Smarak,

In that case, how can we update the default configuration? I was under impression that once I upload my solrconfig.xml for my collection to zookeeper that was it.

Each of my collections, ranger_audits, hadoop_logs and audit_logs has different values. Does that mean, I need to update default configuration for each of these separately (3 edits)? and potentially many more e.g atlas collections etc

sdl

[#6 · Accepted solution]

Hello @SDL 

This is an Old Thread & I assume your Team have moved on, yet wish to Update this Post for future references. It was observed that such Overnight Restart were resetting the default CleanUp (24 Hours) set via [1] in SolrConfig.XML of the respective Solr Collection (Sample from Ranger_Audits Collection). This caused the CleanUp to be postponed on a daily basis & causes Document PileUp beyond their Expiration. If Customer are restarting the Service nightly, It's advisable to set the CleanUp from 24 Hours to a Lower Value (Like, 20 or 22 Hours). 

Regards, Smarak

[1] 

<processor class="solr.processor.DocExpirationUpdateProcessorFactory">
<int name="autoDeletePeriodSeconds">86400</int>
<str name="ttlFieldName">_ttl_</str>
<str name="expirationFieldName">_expire_at_</str>
</processor>