Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Nifi Data Provenance error 1.11.4

avatar
New Contributor

Hi 

We have recently upgraded Apache Nifi from 1.9.2 to 1.11.4 After the upgrade for any processor if try to right click and view data provenance we are getting the following error

 

 

 Format version is not supported (resource BufferedChecksumIndexInput(MMapIndexInput(path="/opt/nifi/flow/provenance_repository/index-1562239673000/segments_3po3"))): 3 (needs to be between 7 and 9). This version of Lucene only supports indexes created with release 6.0 and later.

 

 

Also we are unable to empty any queue even after stopping the upstream and downstream processors.

It keeps showing Dropping flowfiles from queue and is stuck at zero percent.

 

Currently we are using nifi's own embedded zookeeper in a test environment since we do not have an external zookeeper with 3.5 version yet

 

Can anyone let me know what these errors mean and how they can be resolved.

 

1 REPLY 1

avatar
New Contributor

The short answer is that I don't believe you will be able to retain any provenance going to the new version. Ran into this myself going from 1.7 to 1.10 this week.

 

In the 1.10 release notes you will find they updated the underlying Lucene indices in the Provenance repository from version 4.10 to 8.0 in one go, https://issues.apache.org/jira/browse/NIFI-6182

 

I used the Lucene 5.x, 6.x and then 7.x org.apache.lucene.index.IndexUpgrader function to migrate the Lucene indices more than once but ran into a different error:

This index was initially created with Lucene 6.x while the current version is 8.0.0 and Lucene only supports reading the current and previous major versions.. This version of Lucene only supports indexes created with release 7.0 and later.

From that error I found this article indicating that multiple migrations of the Lucene indices are not supported:

https://stackoverflow.com/questions/58114965/lucene-index-upgrading-from-version-4-6-to-8-0-0