ERROR [Index Provenance Events-1]Write.lock error in nifi Provenance Events

[#1]

Greetings Cloudera Community!!!

We have recently upgraded from Nifi 1.9.2 version to Nifi version 1.16.2 and we are getting the "Failed to index Provenance Events" Error, not all the time but it occurs occasionally. We have not faced such kind of error in nifi version 1.9.2.

 Can anyone please help us to figure out the possible cause of the occurrence of that error. Though it was not impacting the flow but still its occurrence is random.  Please find the error below in nifi-app.log. 

Logs:

2023-01-04 06:21:23,713 ERROR [Index Provenance Events-1] o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events
org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:877)
at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:891)
at org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1468)
at org.apache.lucene.index.IndexWriter.addDocuments(IndexWriter.java:1444)
at org.apache.nifi.provenance.lucene.LuceneEventIndexWriter.index(LuceneEventIndexWriter.java:70)
at org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:202)
at org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:113)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: org.apache.lucene.store.AlreadyClosedException: Underlying file changed by an external force at 2022-12-14T11:18:50Z, (lock=NativeFSLock(path=/flowfile_repo/nifi/nifi-1.16.2/provenance_repository/lucene-8-index-1671016730507/write.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive valid],creationTime=2022-12-14T11:18:50.521826Z))
at org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.ensureValid(NativeFSLockFactory.java:191)
at org.apache.lucene.store.LockValidatingDirectoryWrapper.createOutput(LockValidatingDirectoryWrapper.java:43)
at org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:43)
at org.apache.lucene.codecs.lucene80.Lucene80DocValuesConsumer.<init>(Lucene80DocValuesConsumer.java:79)
at org.apache.lucene.codecs.lucene80.Lucene80DocValuesFormat.fieldsConsumer(Lucene80DocValuesFormat.java:161)
at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.getInstance(PerFieldDocValuesFormat.java:227)
at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.getInstance(PerFieldDocValuesFormat.java:163)
at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.addNumericField(PerFieldDocValuesFormat.java:109)
at org.apache.lucene.index.NumericDocValuesWriter.flush(NumericDocValuesWriter.java:108)
at org.apache.lucene.index.DefaultIndexingChain.writeDocValues(DefaultIndexingChain.java:345)
at org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:225)
at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:350)
at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:476)
at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:656)
at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:3365)
at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3771)
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3729)
at org.apache.nifi.provenance.lucene.LuceneEventIndexWriter.commit(LuceneEventIndexWriter.java:101)
at org.apache.nifi.provenance.index.lucene.EventIndexTask.commit(EventIndexTask.java:253)
at org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:232)
... 6 common frames omitted

[#2]

Hello @MattWho 

Can you please guide me on this...Do I need to change the configuration of provenance in nifi.properties from 1GB to 10 GB ?

[#3]

@Girish007 
The stack trace above shows this exception of concern:

Caused by: org.apache.lucene.store.AlreadyClosedException: Underlying file changed by an external force at 2022-12-14T11:18:50Z, (lock=NativeFSLock(path=/flowfile_repo/nifi/nifi-1.16.2/provenance_repository/lucene-8-index-1671016730507/write.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive valid],creationTime=2022-12-14T11:18:50.521826Z))

1. You appear to placed your provenance_repository inside of the same directory as your flowfile_repository?  
2. The exception stated that the "Underlying file changed by an external force".  This indicates that Some service or script outside of NiFi is modifying these files.  Typically when we see this it is caused by some virus scanning service.  You should make sure that NiFi's repositories are excluded form scanning by any virus software.  This would explain why it works until it does not.
3. Since some external interaction with the provenance repository has introduced corruption there, you will need to stop NiFi, Exclude the provenance_repository (any other repositories while at it), delete the current contents of provenance_repository since it is corrupt, and then start NiFi again.

If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped.

Thank you,

Matt

[#4]

@MattWho Thank you for your valuable suggestion!!

1: We had separate directory for flowfile and provenance repositories earlier as well:

  Path for flowfile : "/flowfile_rep/nifi/nifi-1.16.2/flowfile_repository"

  Path for Provenance: "/flowfile_rep/nifi/nifi-1.16.2/provenance_repository"

As per your suggestion we have changed the path as below and restarted the nifi:

New Path for Flow file: "/flowfile_repo/nifi/nifi-1.16.2_flowfileRepository/flowfile_clean_repository"

New Path for Provinance: "/flowfile_repo/nifi-1.16.2_provenanceRepository"

but it seems the issue still persist...

 As we are already using a new directory path so we don't have to delete the old flowfile and provenance repo files right ?

Please suggest if we have to implement any other solution to resolve this issue..

Thank you!!

[#5]

Can anyone please guide us on the above issue?

We are still getting that error whenever we run a nifi process that generates too many flowfiles.

[#6]

@Girish007 Did you make sure that the NiFi directories and repository directories are excluded from any virus software scanning?  That is typically the external force that is likely to being making changes to these files.

Do you have any other external software or processes scanning or access these directories?

Thanks,

Matt