Skip to main content
Gcima009
Gcima009
Silver ⭐️⭐️
October 8, 2019
Solved

Failed to connect node to cluster because local flow controller partially updated.

I have that problem and I have followed the steps of copying the flow.xml.gz to the nodes that do not connect, but after copying them, I have problems to decrypt since a proper key is generated for the gz of that moment

 

this is the log

2019-10-08 16:59:57,735 ERROR org.apache.nifi.web.server.JettyServer: Unable to load flow due to: org.apache.nifi.controller.serialization.FlowSynchronizationException: org.apache.nifi.encrypt.EncryptionException: There was a problem decrypting a sensitive flow configuration value. Check that the nifi.sensitive.props.key value in nifi.properties matches the value used to encrypt the flow.xml.gz file
org.apache.nifi.controller.serialization.FlowSynchronizationException: org.apache.nifi.encrypt.EncryptionException: There was a problem decrypting a sensitive flow configuration value. Check that the nifi.sensitive.props.key value in nifi.properties matches the value used to encrypt the flow.xml.gz file
at org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:478)
at org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1296)
at org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:88)
at org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:812)
at org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:476)
at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:1009)
at org.apache.nifi.NiFi.<init>(NiFi.java:158)
at org.apache.nifi.NiFi.<init>(NiFi.java:72)
at org.apache.nifi.NiFi.main(NiFi.java:297)
Caused by: org.apache.nifi.encrypt.EncryptionException: There was a problem decrypting a sensitive flow configuration value. Check that the nifi.sensitive.props.key value in nifi.properties matches the value used to encrypt the flow.xml.gz file
at org.apache.nifi.controller.serialization.FlowFromDOMFactory.decrypt(FlowFromDOMFactory.java:552)

 

I don`t what to do whit this. thank 

MattWho
MattWhoAnswer
Titanium ⭐️⭐️⭐️
October 15, 2019

@Gcima009 

 

It appears you are running CFM 1.0.0 which has a known issue with the sensitive props key.
If a sensitive props key is not set in CFM NiFi configs, Cm randomly creates one for you.  The bug here is that a different random sensitive props key is generated on each node.  Since NiFi requires that all nodes in the flow have the exact same flow.xml.gz, the sensitive props key used to decrypt  passwords within that flow.xml.gz must also be the same.  This bug was addressed in CFM 1.0.1

 

Anytime you manually edit the files in /var/run/cloudera-scm-agent/process/<num>-nifi-NIFI_NODE and then restart that service via CM, it will detect a change and build new configs in a new folder.

To get around this issue...
1. On one of the nodes cp the "config.zip" file out of the most recent <num>-nifi-NIFI_NODE to a tmpt directory (/tmp/config)

2. Extract the nifi-globals.xml file ( unzip config.zip staging/nifi-globals.xml )
3. Extract the random NiFi sensitive props value ( grep -A 1 random.nifi.sensitive staging/nifi-globals.xml )

4. Shutdown your NiFi cluster via CM.
5. Use the password obtained via step 3 from one node to manually set the password in CM NiFi configs

6. copy flow.xml.gz frorn same node where you obtained the random password to all other nodes in your cluster.
7. start your NiFi cluster. 

Now all nodes are running with same flow.xml.gz using same sensitive props key password.

 

Thanks,

Matt

View in thread

7 replies

Shelton
Shelton
Gold ⭐️⭐️⭐️
October 8, 2019

@Gcima009 

 

When you generate templates in NiFi, they are stripped of all encrypted values. When importing those templates into another NiFi cluster, Check your node that is not starting? has any values in the below parameters? You will have to populate all the processor and controller tasks passwords manually.
Backing up flow.xml.gz or flow.tar file will capture the entire flow exactly as it is, encrypted sensitive passwords and all. NiFi will not start if it cannot decrypt these encrypted sensitive properties contained in the flow.xml. When sensitive properties e.g passwords are added they are encrypted using these settings from your nifi.properties file:

# security properties #
nifi.sensitive.props.key=
nifi.sensitive.props.key.protected=
nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
nifi.sensitive.props.provider=BC
nifi.sensitive.props.additional.keys=

In order to drop your entire flow.xml.gz or flow.tar onto another clean NiFi, these values must all match exactly.

 

Ref: http://www.contemplatingdata.com/2017/08/28/apache-nifi-sensitive-properties-need-know/

Translate
    MattWho
    Titanium ⭐️⭐️⭐️
    MattWho
    Titanium ⭐️⭐️⭐️
    October 10, 2019

    Hello @Gcima009 

     

    The flow.xml.gz includes all passwords set in the components you added within the NiFi UI.  Every passwords would have be encrypted using the sensitive properties password set in the nifi.properties file:

    nifi.sensitive.props.key=

     

    The stack trace you shared is telling us that this NiFi is unable to decrypt the sensitive properties encrypted in the flow.xml.gz file provided to this NiFi using the sensitive properties password configured in the nifi.properties file.  In other words the NiFi were you copied this flow.xml.gz file from is using a different sensitive properties key.

     

    Edited the nifi.properties file on this NiFi so that it is using the same password used by the NiFi were you copied the flow.xml.gz file from.

     

    Hope this helps,
    Matt

    Translate
      Gcima009
      Gcima009Author
      Silver ⭐️⭐️
      October 15, 2019

      Hi @MattWho, I'll tell you what I'm doing

      1 demolition of node XX that is disconnected from the cluster by different flow.xml

      2 copy flow-xml.gz of the node that is fine to node XX

      3 in the nifipropoerties file setting nifi.sensitive.props.key of the node disconnected by the nifi.sensitive.props.key of the node that is connected

      4 subtract from nodeXX

      but every time I restart a node it generates a new folder

      this is an example:

      drwxr-x--x 6 nifi      nifi 520 oct 15 10:45 1216-nifi-NIFI_NODE

      drwxr-x--x 6 nifi      nifi 520 oct 15 10:45 1211-nifi-NIFI_NODE

      drwxr-x--x 6 nifi      nifi 520 oct 15 10:33 1201-nifi-NIFI_NODE

      drwxr-x--x 3 zookeeper zookeeper 360 oct 15 10:30 1207-zookeeper-server

      drwxr-x--x 3 zookeeper zookeeper 380 oct 15 10:30 1022-zookeeper-server

      drwxr-x--x 6 nifi      nifi 520 oct 15 10:29 1196-nifi-NIFI_NODE

      drwxr-x--x 6 nifi      nifi 520 oct 15 10:08 1175-nifi-NIFI_NODE

       

      The nifi.properties is in every folder like XXXX-nifi-NIFI_NODE

      I don't know how to stop this folder creation, because it doesn't have sense change de key in the last folder when node is down

       

      Thanks

       

       

      Translate
      MattWho
      Titanium ⭐️⭐️⭐️
      MattWhoAnswer
      Titanium ⭐️⭐️⭐️
      October 15, 2019

      @Gcima009 

       

      It appears you are running CFM 1.0.0 which has a known issue with the sensitive props key.
      If a sensitive props key is not set in CFM NiFi configs, Cm randomly creates one for you.  The bug here is that a different random sensitive props key is generated on each node.  Since NiFi requires that all nodes in the flow have the exact same flow.xml.gz, the sensitive props key used to decrypt  passwords within that flow.xml.gz must also be the same.  This bug was addressed in CFM 1.0.1

       

      Anytime you manually edit the files in /var/run/cloudera-scm-agent/process/<num>-nifi-NIFI_NODE and then restart that service via CM, it will detect a change and build new configs in a new folder.

      To get around this issue...
      1. On one of the nodes cp the "config.zip" file out of the most recent <num>-nifi-NIFI_NODE to a tmpt directory (/tmp/config)

      2. Extract the nifi-globals.xml file ( unzip config.zip staging/nifi-globals.xml )
      3. Extract the random NiFi sensitive props value ( grep -A 1 random.nifi.sensitive staging/nifi-globals.xml )

      4. Shutdown your NiFi cluster via CM.
      5. Use the password obtained via step 3 from one node to manually set the password in CM NiFi configs

      6. copy flow.xml.gz frorn same node where you obtained the random password to all other nodes in your cluster.
      7. start your NiFi cluster. 

      Now all nodes are running with same flow.xml.gz using same sensitive props key password.

       

      Thanks,

      Matt

      Translate
        Terms & ConditionsAccessibility statement