@jfs912  NiFi 2.x is secured in the same way as NiFi 1.x was secured.  The Apache NIFi community made a decision to move away from providing a toolkit for generating TLS certificates since it did not make sense to have its own tool to manage certificate creation when there is nothing special the toolkit does (NIFI-12200) .  You can create your own self signed certificates (simple web search will show many resources on how to do this), you can use a certificate service to create yoru certificates and sign them for you, you can setup your own certificate authority for signing your certificates, etc.    It really made no sense for Apache NiFi to maintain this code.  The important parts to remember when creating your NiFi certificates are: Keystore for NiFi: Must contain 1 and only 1 PrivateKey entry PrivateKey entry DN must NOT contain wildcards.  The DN is used as the client identity and use of wildcards in client identities is not a proper security practice. PrivateKey entry must have ExtendedKeyUsage with both ClientAuth and ServerAuth.  NiFi uses this certificate for both client and server authorizations PrivateKey entry must include at least one SAN entry that matches the hostname of the NiFi server on which the certificate will be used. During a TLS exchange hostname verification is performed to make sure the hostname accessed via the URL used matches a SAN entry in the Server certificate the target server responds with.  It is not uncommon for a certificate to contain more then one SAN entry. Recommend that PrivateKey password and Keystore password are the same. Truststore for NiFi: The NiFi truststore contains one too many trusted cert entries.  Some choose to use the Java default cacerts (truststore) file and just add the NiFi additional trusted certs entries to it.  This same truststore is then used on every node in a NiFi cluster. Must contain the complete trusts chain for the NiFi certificates.  A NiFi certificate may be self signed meaning the issuer and signer are the same DistinquishedName (DN).  I this case the public cert for each of yoru NiFi node's certificates needs to be added to the truststore.  A NiFi certificate might be signed by an Intermediate Certificate Authority (CA).  An intermediate CA would be the signer DN for the NIFi certificate. An Intermediate CA will have a different Issuer and Signer DN.  The truststore must contain the public cert for this Intermediate CA.  There may be multiple levels of intermediate CAs before reaching the signer that is the root CA. The root CA public certificate can be identified  because it will have the same DN for issuer and signer.  The truststore must contain the public certificate for the root CA.  Having all the public certificates for every signer for the NiFi certificate to the Root CA makes up the complete trust chain required for trust of the signed NiFi certificate. Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

Preferred to have new question for unique questions, but.... Keep in mind these things when it comes to user identities and authorizations: 1. user identities are case sensitive in NiFi.  So the user identity from oidc must match exactly the initial admin user identity you passed to the file-user-group-provider and the file-access-policy-provider. 2. The file-user-group-provider will only generate the users.xml file if it does NOT already exist.  So if you edit the configuration of the file-user-group-provide after the users.xml already exists, those change will not be made to that existing users.xml file.  (remove it and restart so it gets created again) 3.  The file-access-policy-provider will only generate the authorizations.xml file if it does not exist.  It also will not modify an already existing authorizations.xml file if you make modifications to the provider. 4. Additional user identities and authorization post accessing your NiFi with your initial admin are done directly with the NiFi UI.  Those additions made from the UI will be persisted in the users.xml and authorizations.xml files If above does not help, please start a new community question with the details of your issue. Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@HeniNechi  I am not following the relationship between what you have added and the original question in this thread.   Authentication and Authorization are two separate processes.  Only after successful authentication is the resulting user/client identity passed to the NiFi authorizer that will authorize that user/client identity based on the defined NiFi authorization policies.   You are correct that the default authorizers.xml setup uses the managed authorizer that maps user/client identities from the users.xml file with authorization stored in the authorizations.xml file.   The file-user-group-provider should be generating the users.xml file if it does not already exist during NiFi startup.  If you are having an issue, I recommend starting a new community question with the details of your issue. Thank you, Matt Matt ... View more

@PradNiFi1236  The keystore the NiFi node uses must contain only 1 clientAuth private key.  There is no configuration option in Apache NiFi to specific which private key is used when multiple exist in the keystore.  Of course the truststore will be same on all the nodes and can contain as many Public keys as you want, but must contain the complete trust chain fro yoru NiFi node private certificates.   ... View more

@PradNiFi1236  Not much here to work with.  I suggest first comparing the NiFi configuration files between all your nodes to make sure they are all are identical with exception of hostnames, keystores, and truststores. Are you using a load balancer?  If so, do you see a change in behavior if you enable session affinity (Sticky sessions) in your LB? Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

Possibly related to https://issues.apache.org/jira/browse/NIFI-14462 I suggest reviewing discussion in this Apache Jira and reviewing your "nifi.web.request.timeout" setting in the nifi.properties file. Making adjustments to this setting may help here. Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@MK77  First lets clarify the Zookeeper (ZK) elected roles in Apache NiFi. Primary: ZK elects one node in the cluster as the "Primary" node.   Processor components on the canvas configured to with Execution=Primary node will only get scheduled on that elected primary node.  No other nodes will schedule these processors to execute. Cluster Coordinator: ZK elects one of the nodes as the cluster coordinator.  Other nodes learn which node is the elected cluster coordinator from ZK.  All nodes will send node heartbeats to the cluster coordinator to form the cluster. Any node in the NiFi cluster can be assigned either or both of these roles.  There is no guarantee that the same node(s) will always be assigned these roles.  Even after NiFi cluster is formed and roles are assigned, which nodes are assigned these roles can change. The flow.json.gz contain the dataflows on the canvas that are loaded on startup.  The flow.xml.gz is only loaded if the flow.json.gz is missing.   If NiFi loads the dataflow from the flow.xml.gz, it will generate a flow.json.gz from that flow.xml.gz.   Now on to your problem.... Neither of the log lines you shared point to any problem: Invalid State Cannot replicate request to Node <node-hostname:port> because the node is not connected This log line simply tells you that this node can't replicate a request to anothetr node yet because it has not has not connected yet to the cluster. o.a.n.w.a.c.IllegalClusterStateExceptionMapper org.apache.nifi.cluster.manager.exception.IllegalClusterStateException: The Flow Controller is initializing the Data Flow.. Returning Conflict response. This simply tells you that the flow.json.gz is still being initialized (loaded).   This process needs to complete before the node finishes startup and can join the cluster.  Depending on which Apache NiFi version you are running and the size of yoru dataflow, this can take some time to complete.   What is the complete version of NiFi you are using? Without your full logs it is not possible from what has been shared to tell you what is going on or even if there really is any corruption with your flow.json.gz. One thing you can do is configure yoru NiFi to startup with all components on yoru canvas stopped instead of their last known state.   This can be helpful if you have added a recent new dataflow that is perhaps causing issues initializing at startup. This achieved by changing the following setting in the nifi,properties file. Save a backup of your flow.json.gz before starting after changing this setting.  The saved flow.json.gz will have the original saves state (Running, Stopped, Disabled) of all the components. nifi.flowcontroller.autoResumeState=false   If your NiFi cluster starts fine after making this change, you can restart your dataflows to see if any are having issues. Beyond the above suggestion, there is not enough information shared to suggest anything else. Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more