Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Failed to connect node to cluster because local flow controller partially updated.

avatar
Explorer

I have that problem and I have followed the steps of copying the flow.xml.gz to the nodes that do not connect, but after copying them, I have problems to decrypt since a proper key is generated for the gz of that moment

 

this is the log

2019-10-08 16:59:57,735 ERROR org.apache.nifi.web.server.JettyServer: Unable to load flow due to: org.apache.nifi.controller.serialization.FlowSynchronizationException: org.apache.nifi.encrypt.EncryptionException: There was a problem decrypting a sensitive flow configuration value. Check that the nifi.sensitive.props.key value in nifi.properties matches the value used to encrypt the flow.xml.gz file
org.apache.nifi.controller.serialization.FlowSynchronizationException: org.apache.nifi.encrypt.EncryptionException: There was a problem decrypting a sensitive flow configuration value. Check that the nifi.sensitive.props.key value in nifi.properties matches the value used to encrypt the flow.xml.gz file
at org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:478)
at org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1296)
at org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:88)
at org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:812)
at org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:476)
at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:1009)
at org.apache.nifi.NiFi.<init>(NiFi.java:158)
at org.apache.nifi.NiFi.<init>(NiFi.java:72)
at org.apache.nifi.NiFi.main(NiFi.java:297)
Caused by: org.apache.nifi.encrypt.EncryptionException: There was a problem decrypting a sensitive flow configuration value. Check that the nifi.sensitive.props.key value in nifi.properties matches the value used to encrypt the flow.xml.gz file
at org.apache.nifi.controller.serialization.FlowFromDOMFactory.decrypt(FlowFromDOMFactory.java:552)

 

I don`t what to do whit this. thank 

1 ACCEPTED SOLUTION

avatar
Super Mentor

@Gcima009 

 

It appears you are running CFM 1.0.0 which has a known issue with the sensitive props key.
If a sensitive props key is not set in CFM NiFi configs, Cm randomly creates one for you.  The bug here is that a different random sensitive props key is generated on each node.  Since NiFi requires that all nodes in the flow have the exact same flow.xml.gz, the sensitive props key used to decrypt  passwords within that flow.xml.gz must also be the same.  This bug was addressed in CFM 1.0.1

 

Anytime you manually edit the files in /var/run/cloudera-scm-agent/process/<num>-nifi-NIFI_NODE and then restart that service via CM, it will detect a change and build new configs in a new folder.

To get around this issue...
1. On one of the nodes cp the "config.zip" file out of the most recent <num>-nifi-NIFI_NODE to a tmpt directory (/tmp/config)

2. Extract the nifi-globals.xml file ( unzip config.zip staging/nifi-globals.xml )
3. Extract the random NiFi sensitive props value ( grep -A 1 random.nifi.sensitive staging/nifi-globals.xml )

4. Shutdown your NiFi cluster via CM.
5. Use the password obtained via step 3 from one node to manually set the password in CM NiFi configs

6. copy flow.xml.gz frorn same node where you obtained the random password to all other nodes in your cluster.
7. start your NiFi cluster. 

Now all nodes are running with same flow.xml.gz using same sensitive props key password.

 

Thanks,

Matt

View solution in original post

7 REPLIES 7

avatar
Master Mentor

@Gcima009 

 

When you generate templates in NiFi, they are stripped of all encrypted values. When importing those templates into another NiFi cluster, Check your node that is not starting? has any values in the below parameters? You will have to populate all the processor and controller tasks passwords manually.
Backing up flow.xml.gz or flow.tar file will capture the entire flow exactly as it is, encrypted sensitive passwords and all. NiFi will not start if it cannot decrypt these encrypted sensitive properties contained in the flow.xml. When sensitive properties e.g passwords are added they are encrypted using these settings from your nifi.properties file:

# security properties #
nifi.sensitive.props.key=
nifi.sensitive.props.key.protected=
nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
nifi.sensitive.props.provider=BC
nifi.sensitive.props.additional.keys=

In order to drop your entire flow.xml.gz or flow.tar onto another clean NiFi, these values must all match exactly.

 

Ref: http://www.contemplatingdata.com/2017/08/28/apache-nifi-sensitive-properties-need-know/

avatar
Super Mentor

Hello @Gcima009 

 

The flow.xml.gz includes all passwords set in the components you added within the NiFi UI.  Every passwords would have be encrypted using the sensitive properties password set in the nifi.properties file:

nifi.sensitive.props.key=

 

The stack trace you shared is telling us that this NiFi is unable to decrypt the sensitive properties encrypted in the flow.xml.gz file provided to this NiFi using the sensitive properties password configured in the nifi.properties file.  In other words the NiFi were you copied this flow.xml.gz file from is using a different sensitive properties key.

 

Edited the nifi.properties file on this NiFi so that it is using the same password used by the NiFi were you copied the flow.xml.gz file from.

 

Hope this helps,
Matt

avatar
Explorer

Hi @MattWho, I'll tell you what I'm doing

1 demolition of node XX that is disconnected from the cluster by different flow.xml

2 copy flow-xml.gz of the node that is fine to node XX

3 in the nifipropoerties file setting nifi.sensitive.props.key of the node disconnected by the nifi.sensitive.props.key of the node that is connected

4 subtract from nodeXX

but every time I restart a node it generates a new folder

this is an example:

drwxr-x--x 6 nifi      nifi 520 oct 15 10:45 1216-nifi-NIFI_NODE

drwxr-x--x 6 nifi      nifi 520 oct 15 10:45 1211-nifi-NIFI_NODE

drwxr-x--x 6 nifi      nifi 520 oct 15 10:33 1201-nifi-NIFI_NODE

drwxr-x--x 3 zookeeper zookeeper 360 oct 15 10:30 1207-zookeeper-server

drwxr-x--x 3 zookeeper zookeeper 380 oct 15 10:30 1022-zookeeper-server

drwxr-x--x 6 nifi      nifi 520 oct 15 10:29 1196-nifi-NIFI_NODE

drwxr-x--x 6 nifi      nifi 520 oct 15 10:08 1175-nifi-NIFI_NODE

 

The nifi.properties is in every folder like XXXX-nifi-NIFI_NODE

I don't know how to stop this folder creation, because it doesn't have sense change de key in the last folder when node is down

 

Thanks

 

 

avatar
Super Mentor

@Gcima009 

 

It appears you are running CFM 1.0.0 which has a known issue with the sensitive props key.
If a sensitive props key is not set in CFM NiFi configs, Cm randomly creates one for you.  The bug here is that a different random sensitive props key is generated on each node.  Since NiFi requires that all nodes in the flow have the exact same flow.xml.gz, the sensitive props key used to decrypt  passwords within that flow.xml.gz must also be the same.  This bug was addressed in CFM 1.0.1

 

Anytime you manually edit the files in /var/run/cloudera-scm-agent/process/<num>-nifi-NIFI_NODE and then restart that service via CM, it will detect a change and build new configs in a new folder.

To get around this issue...
1. On one of the nodes cp the "config.zip" file out of the most recent <num>-nifi-NIFI_NODE to a tmpt directory (/tmp/config)

2. Extract the nifi-globals.xml file ( unzip config.zip staging/nifi-globals.xml )
3. Extract the random NiFi sensitive props value ( grep -A 1 random.nifi.sensitive staging/nifi-globals.xml )

4. Shutdown your NiFi cluster via CM.
5. Use the password obtained via step 3 from one node to manually set the password in CM NiFi configs

6. copy flow.xml.gz frorn same node where you obtained the random password to all other nodes in your cluster.
7. start your NiFi cluster. 

Now all nodes are running with same flow.xml.gz using same sensitive props key password.

 

Thanks,

Matt

avatar
Explorer

It worked fine ... thank you very much

avatar
Explorer

Hi,

 

I'm trying to follow your solution, however step 3 only shows a masked key, as follows:

<name>random.nifi.sensitive.props.key</name>
<value>********</value>

 

is there anyway that i can work around that?

avatar
Super Mentor

@Abdullah 

 

If the sensitive props key value is obscured in the globals.xml file, you are running a newer version fo CFM then 1.0.0 where the bug existed where each node in the NiFi cluster ended up with a different random sensitive props key.  In CFM 1.0.1 and newer, the user is required to set this property (it is not longer set to a random value when left blank). 

So perhaps you are having a different issue here?
Did you change the sensitive props key in your CFM NiFi configs and then had an issue with starting your NiFi? 


I suggest starting a new question in the community since you are having a different issue than what is described in this thread.