Failed to connect node to cluster because local flow controller partially updated.

[#1]

I have that problem and I have followed the steps of copying the flow.xml.gz to the nodes that do not connect, but after copying them, I have problems to decrypt since a proper key is generated for the gz of that moment

this is the log

2019-10-08 16:59:57,735 ERROR org.apache.nifi.web.server.JettyServer: Unable to load flow due to: org.apache.nifi.controller.serialization.FlowSynchronizationException: org.apache.nifi.encrypt.EncryptionException: There was a problem decrypting a sensitive flow configuration value. Check that the nifi.sensitive.props.key value in nifi.properties matches the value used to encrypt the flow.xml.gz file
org.apache.nifi.controller.serialization.FlowSynchronizationException: org.apache.nifi.encrypt.EncryptionException: There was a problem decrypting a sensitive flow configuration value. Check that the nifi.sensitive.props.key value in nifi.properties matches the value used to encrypt the flow.xml.gz file
at org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:478)
at org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1296)
at org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:88)
at org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:812)
at org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:476)
at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:1009)
at org.apache.nifi.NiFi.<init>(NiFi.java:158)
at org.apache.nifi.NiFi.<init>(NiFi.java:72)
at org.apache.nifi.NiFi.main(NiFi.java:297)
Caused by: org.apache.nifi.encrypt.EncryptionException: There was a problem decrypting a sensitive flow configuration value. Check that the nifi.sensitive.props.key value in nifi.properties matches the value used to encrypt the flow.xml.gz file
at org.apache.nifi.controller.serialization.FlowFromDOMFactory.decrypt(FlowFromDOMFactory.java:552)

I don`t what to do whit this. thank 

[#2]

@Gcima009 

When you generate templates in NiFi, they are stripped of all encrypted values. When importing those templates into another NiFi cluster, Check your node that is not starting? has any values in the below parameters? You will have to populate all the processor and controller tasks passwords manually.
Backing up flow.xml.gz or flow.tar file will capture the entire flow exactly as it is, encrypted sensitive passwords and all. NiFi will not start if it cannot decrypt these encrypted sensitive properties contained in the flow.xml. When sensitive properties e.g passwords are added they are encrypted using these settings from your nifi.properties file:

# security properties #
nifi.sensitive.props.key=
nifi.sensitive.props.key.protected=
nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
nifi.sensitive.props.provider=BC
nifi.sensitive.props.additional.keys=

In order to drop your entire flow.xml.gz or flow.tar onto another clean NiFi, these values must all match exactly.

Ref: http://www.contemplatingdata.com/2017/08/28/apache-nifi-sensitive-properties-need-know/

[#3]

Hello @Gcima009 

The flow.xml.gz includes all passwords set in the components you added within the NiFi UI.  Every passwords would have be encrypted using the sensitive properties password set in the nifi.properties file:

nifi.sensitive.props.key=

The stack trace you shared is telling us that this NiFi is unable to decrypt the sensitive properties encrypted in the flow.xml.gz file provided to this NiFi using the sensitive properties password configured in the nifi.properties file.  In other words the NiFi were you copied this flow.xml.gz file from is using a different sensitive properties key.

Edited the nifi.properties file on this NiFi so that it is using the same password used by the NiFi were you copied the flow.xml.gz file from.

Hope this helps,
Matt

[#4]

Hi @MattWho, I'll tell you what I'm doing

1 demolition of node XX that is disconnected from the cluster by different flow.xml

2 copy flow-xml.gz of the node that is fine to node XX

3 in the nifipropoerties file setting nifi.sensitive.props.key of the node disconnected by the nifi.sensitive.props.key of the node that is connected

4 subtract from nodeXX

but every time I restart a node it generates a new folder

this is an example:

drwxr-x--x 6 nifi      nifi 520 oct 15 10:45 1216-nifi-NIFI_NODE

drwxr-x--x 6 nifi      nifi 520 oct 15 10:45 1211-nifi-NIFI_NODE

drwxr-x--x 6 nifi      nifi 520 oct 15 10:33 1201-nifi-NIFI_NODE

drwxr-x--x 3 zookeeper zookeeper 360 oct 15 10:30 1207-zookeeper-server

drwxr-x--x 3 zookeeper zookeeper 380 oct 15 10:30 1022-zookeeper-server

drwxr-x--x 6 nifi      nifi 520 oct 15 10:29 1196-nifi-NIFI_NODE

drwxr-x--x 6 nifi      nifi 520 oct 15 10:08 1175-nifi-NIFI_NODE

The nifi.properties is in every folder like XXXX-nifi-NIFI_NODE

I don't know how to stop this folder creation, because it doesn't have sense change de key in the last folder when node is down

Thanks

[#5 · Accepted solution]

@Gcima009 

It appears you are running CFM 1.0.0 which has a known issue with the sensitive props key.
If a sensitive props key is not set in CFM NiFi configs, Cm randomly creates one for you.  The bug here is that a different random sensitive props key is generated on each node.  Since NiFi requires that all nodes in the flow have the exact same flow.xml.gz, the sensitive props key used to decrypt  passwords within that flow.xml.gz must also be the same.  This bug was addressed in CFM 1.0.1

Anytime you manually edit the files in /var/run/cloudera-scm-agent/process/<num>-nifi-NIFI_NODE and then restart that service via CM, it will detect a change and build new configs in a new folder.

To get around this issue...
1. On one of the nodes cp the "config.zip" file out of the most recent <num>-nifi-NIFI_NODE to a tmpt directory (/tmp/config)

2. Extract the nifi-globals.xml file ( unzip config.zip staging/nifi-globals.xml )
3. Extract the random NiFi sensitive props value ( grep -A 1 random.nifi.sensitive staging/nifi-globals.xml )

4. Shutdown your NiFi cluster via CM.
5. Use the password obtained via step 3 from one node to manually set the password in CM NiFi configs

6. copy flow.xml.gz frorn same node where you obtained the random password to all other nodes in your cluster.
7. start your NiFi cluster. 

Now all nodes are running with same flow.xml.gz using same sensitive props key password.

Thanks,

Matt

[#6]

It worked fine ... thank you very much

[#7]

Hi,

I'm trying to follow your solution, however step 3 only shows a masked key, as follows:

<name>random.nifi.sensitive.props.key</name>
<value>********</value>

is there anyway that i can work around that?

[#8]

@Abdullah 

If the sensitive props key value is obscured in the globals.xml file, you are running a newer version fo CFM then 1.0.0 where the bug existed where each node in the NiFi cluster ended up with a different random sensitive props key.  In CFM 1.0.1 and newer, the user is required to set this property (it is not longer set to a random value when left blank). 

So perhaps you are having a different issue here?
Did you change the sensitive props key in your CFM NiFi configs and then had an issue with starting your NiFi? 

I suggest starting a new question in the community since you are having a different issue than what is described in this thread.