Losing processor's state on nifi upgrade

[#1]

Hi team!

We are currently running a standalone nifi 1.13.2.

Our upgrade strategy is 1.13.2 (outside docker) -> 1.25.0 (dockerized) -> 2.0.0-M2 (dockerized)

Reference: https://cwiki.apache.org/confluence/display/NIFI/Migration+Guidance#MigrationGuidance-Migratingto2.0...

When we upgrade to 1.25.0 (dockerized) all states are still there, but after updating to 2.0.0-M2 (from 1.25.0) we no longer have the processor's states.

Our docker volumes are these

- /opt/nifi/nifi-current/database_repository:/opt/nifi/nifi-current/database_repository
- /opt/nifi/nifi-current/flowfile_repository:/opt/nifi/nifi-current/flowfile_repository
- /opt/nifi/nifi-current/content_repository:/opt/nifi/nifi-current/content_repository
- /opt/nifi/nifi-current/provenance_repository:/opt/nifi/nifi-current/provenance_repository
- /opt/nifi/nifi-current/state:/opt/nifi/nifi-current/state
- /opt/nifi/nifi-current/logs:/opt/nifi/nifi-current/logs
- /opt/nifi/nifi-current/conf:/opt/nifi/nifi-current/conf
- /opt/nifi/nifi-current/extras:/opt/nifi/nifi-current/extras

Any help would be appreciated!

Thank you!

[#2]

@double_w, Welcome to our community! To help you get the best possible answer, I have tagged in our NiFi experts @SAMSAL @MattWho @ckumar  who may be able to assist you further.

Please feel free to provide any additional information or details about your query, and we hope that you will find a satisfactory solution to your question.

[#3]

@double_w 

NiFi state configuration is setup in the state-management.xml file located by default in NiFi's conf directory.   When you say that state is being lost, are you specifically having issues with missing "cluster" state, "local" state, or both? 

Is your NiFi clustered or a standalone (non cluster) deployment?

Default cluster state is stored in zookeeper (are you using an external zookeeper or the embedded zookeeper?).   Any chance that your NiFi-2.0 is configured to use the new "kubernetes-provider" for cluster state management?  If so, did you configure the "nifi.state.management.provider.cluster.previous" property in nifi.properties with the previous "zk-provider", so that on first startup, it migrates your cluster state from zk-provider to the empty kubernetes-provider?

Default local-provider stores state in ./state/local.  

Have you compared the state-management.xml file configurations between both environments?

Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

[#4]

Hi @MattWho,

Thank you for answering my question.

It's a standalone installation

The provider confs are

- nifi.state.management.provider.local=local-provider

- nifi.state.management.configuration.file=./conf/state-management.xml

And isn't a cluster node

- nifi.cluster.is.node=false

I also didn't detected any changes on state-management.xml after updating to 2.0.0-M2.

I detected changes after start the new version in /state/local files, I thought it could be some serialization changes, but I couldn't see major changes in state file format.

I'm out of new ideas, my last one would be migrate the local state contents to zookeeper, then update, then migrate the states back to local, or keep on zookeeper, I don't know if this is possible, if is please provide me some directions.

Thank you again

[#5]

@double_w 

Can you share some details on which specific components you are using that appear to lose state after upgrade?

The upgrade from 1.13.2 to 1.25.0 is a large leap.  Did you test out your dataflow after upgrading to 1.25.0?  Was state still working correctly before migrating then to 2.0.0-M3?

I am unaware if any way to migrate local state to Zookeeper.

While I do not have an answer for you here, the more details you share the more i can look into it possibly as I have time.

Thanks,
Matt

[#6]

Hi @MattWho

About your questions

• Can you share some details on which specific components you are using that appear to lose state after upgrade?
  • All processors even the valid ones
• Did you test out your dataflow after upgrading to 1.25.0?
  • Yes it works as expected
• Was state still working correctly before migrating then to 2.0.0-M2?
  • Yes, the problem only happens when upgrading from 1.25.0 to 2.0.0-M2 (I tried M1 too, same problem 😕)

To help with this I created a flow to reproduce the problem it will require docker.

# Setup nifi 1.25.0 installation

mkdir nifi_upgrade_test

docker run -d --rm -v ./nifi_upgrade_test:/nifi_upgrade_test -d -p 8443:8443 --name nifi apache/nifi:1.25.0

## wait until nifi starts
## It can be checked with docker logs nifi -f

docker exec nifi /opt/nifi/nifi-current/bin/nifi.sh set-single-user-credentials admin supersecret1

docker restart nifi

# go https://localhost:8443/nifi/
# user: admin
# pass: supersecret1

# Create a "dummy" state
# 1) Add two processors GenerateFlowFile and UpdateAttribute
# 2) Add relationship GenerateFlowFile -> success -> UpdateAttribute
# 3) UpdateAttribute: add a property called "something" with value "${getStateValue("something"):plus(1)}"
# 4) UpdateAttribute: update property "Store State" to "Store state locally"
# 5) UpdateAttribute: update property "Stateful Variables Initial Value" to 0
# 6) UpdateAttribute: update "success" and "set state fail" relationship to terminate
# 7) Run once both processors first GenerateFlowFile and then UpdateAttribute
# Now UpdateAttribute have state something=1

# Prepare migration: copy important files to /nifi_upgrade_test
docker exec nifi cp conf/ content_repository/ database_repository/ flowfile_repository/ provenance_repository/ state/ /nifi_upgrade_test -rv

docker kill nifi

# replace flow.xml.gz to flow.json.gz
sed -i "s/flow.xml.gz/flow.json.gz/g" nifi_upgrade_test/conf/nifi.properties

# volumes
# ./nifi_upgrade_test/conf:/opt/nifi/nifi-current/conf
# ./nifi_upgrade_test/content_repository:/opt/nifi/nifi-current/content_repository
# ./nifi_upgrade_test/database_repository:/opt/nifi/nifi-current/database_repository
# ./nifi_upgrade_test/flowfile_repository:/opt/nifi/nifi-current/flowfile_repository
# ./nifi_upgrade_test/provenance_repository:/opt/nifi/nifi-current/provenance_repository
# ./nifi_upgrade_test/state:/opt/nifi/nifi-current/state
# ./nifi_upgrade_test:/nifi_upgrade_test

# Start nifi 2.0.0-M2 installation
docker run -d --rm -v ./nifi_upgrade_test/conf:/opt/nifi/nifi-current/conf -v ./nifi_upgrade_test/content_repository:/opt/nifi/nifi-current/content_repository -v ./nifi_upgrade_test/database_repository:/opt/nifi/nifi-current/database_repository -v ./nifi_upgrade_test/flowfile_repository:/opt/nifi/nifi-current/flowfile_repository -v ./nifi_upgrade_test/provenance_repository:/opt/nifi/nifi-current/provenance_repository -v ./nifi_upgrade_test/state:/opt/nifi/nifi-current/state -v ./nifi_upgrade_test:/nifi_upgrade_test -d -p 8443:8443 --name nifi apache/nifi:2.0.0-M2

# go https://localhost:8443/nifi/ again
# user: admin
# pass: supersecret1

After migration the processors are still valid, but the states have been lost

Thank you again

[#7]

@MattWho 

You will see this behavior even if you don't use docker only nifi standard binaries.

I validated this copying those dirs below from 1.25.0 installation to 2.0.0-M2 installation both without using docker.

• conf/
• content_repository/
• database_repository/
• flowfile_repository/
• provenance_repository/
• state/

Thank you again