Nifi Data Provenance error 1.11.4

[#1]

Hi 

We have recently upgraded Apache Nifi from 1.9.2 to 1.11.4 After the upgrade for any processor if try to right click and view data provenance we are getting the following error

 Format version is not supported (resource BufferedChecksumIndexInput(MMapIndexInput(path="/opt/nifi/flow/provenance_repository/index-1562239673000/segments_3po3"))): 3 (needs to be between 7 and 9). This version of Lucene only supports indexes created with release 6.0 and later.

Also we are unable to empty any queue even after stopping the upstream and downstream processors.

It keeps showing Dropping flowfiles from queue and is stuck at zero percent.

Currently we are using nifi's own embedded zookeeper in a test environment since we do not have an external zookeeper with 3.5 version yet

Can anyone let me know what these errors mean and how they can be resolved.

[#2]

The short answer is that I don't believe you will be able to retain any provenance going to the new version. Ran into this myself going from 1.7 to 1.10 this week.

In the 1.10 release notes you will find they updated the underlying Lucene indices in the Provenance repository from version 4.10 to 8.0 in one go, https://issues.apache.org/jira/browse/NIFI-6182

I used the Lucene 5.x, 6.x and then 7.x org.apache.lucene.index.IndexUpgrader function to migrate the Lucene indices more than once but ran into a different error:

This index was initially created with Lucene 6.x while the current version is 8.0.0 and Lucene only supports reading the current and previous major versions.. This version of Lucene only supports indexes created with release 7.0 and later.

From that error I found this article indicating that multiple migrations of the Lucene indices are not supported:

https://stackoverflow.com/questions/58114965/lucene-index-upgrading-from-version-4-6-to-8-0-0