@RB764  @ckumar is correct that what is failing here is a mutual TLS handshake. "PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target" Above is telling you that one side or the other of this TLS connection is unable to establish trust for the certificate presented during the handshake.  The "PrivateKeyEntry" from the keystore file is being presented and the "trustedCertEntry"s from the truststore file are being used to verify trust for that certificate. In order to establish trust, the complete trust chain needs to be in your truststore files. Your PrivateKeyEntry will have an owner and an issuer.  The issuer is the signer of the certificate. That issuer will have its own public key (TrustedCertEntry) in the truststore files.  That issuer public key will also have an owner and an issuer.  If the issuer DN is not the same as the owner DN, then it is an intermediate CA.  So again you would need the public key for the issuer of that intermediate Public key.  Once you reach the point in the trust chain where the public key owner and issuer are the same DN, then you have the complete trust chain. You can use openssl command to get the complete trustchain in most scenarios. openssl s_client -connect <nifi-hostname>:<nifi port> -showcerts openssl s_client -connect <nifi-regsitry-hostname>:<nifi-registry port> -showcerts The certs will be returned in the format of: -----BEGIN CERTIFICATE----- MIIFljCCA36gAwIBAgINAgO... -----END CERTIFICATE----- You can copy each and save it is a .pem file and import those that are missing in to your truststore files. If NiFi-registry is secured, then NiFi will need to be secured to talk to use it. If NiFi is secured, it is optional to secure NiFi-Registry.  (You can add a http nifi-registry client in NiFi) If you found the information provided by anyone was helpful with your query, please take a moment to login and click "Accept" on every response that helped you to a solution. Matt ... View more

@TRSS_Cloudera  When you "Enable" the DistributedMapCacheServer Controller service with UUID 5aaa327b-9ebb-1d5a-71ad-a349b8578a25, it is failing to enable because the port configured in that controller service is already being used by some other process on the server. Do you have multiple DistributedMapCache controller service configured on your NiFi (perhaps they are both enabled and first is already using that port? Did you check at OS level what process is listening on that port? My guess here is that you have multiple DistributedMapCacheServer controller services setup configured to use same port.  Thus the error on one of them about being unable to bind to the port because it is already in use. I don't think above has anything to do with issue with your wait/notify flow since you say it works (at least with first FlowFile) as expected. As far as the behavior of your Wait/Notify dataflow, it is hard for me to make suggestions on what you are observing with out my detail on your dataflow configuration and use case. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt ... View more

@bk1937  I 100% agree with the advice given by @ckumar  Also consider for a basic LB setup, you'll need to use sticky sessions. You mention that you are "logging in", so I take this to mean that your user(s) are not using certificate based authentication, but rather using a login provider like ldap-provider or kerberos-provider. When you access "https://<load-balancer>/nifi/ " without using sticky sessions, the request may go to node 1 where you get the login window.  You enter your credentials and get back a bearer token that your browser now stores for the "https://<load-balancer>/nifi/ " endpoint.   The issue here is that bearer token is only valid for the host that issued it (node1).  So immediately after node1 sent your browser this token, it attempts to redirect your browser to UI.  Without Sticky Sessions, your LB may send the redirect to node 2 - 5) and those nodes will not know anything about that client bearer token since they will not have the corresponding server side token.  This the token is rejected and your user is treated as anonymous. You'll need to investigate your LB to see how to enable sticky sessions.  Sticky sessions will make sure all subsequent request continue to get routed to same host as original request. If you any of the responses assisted with your query, please take a moment to login and click on "Accept as Solution" below each of those posts. Thank you, Matt ... View more

@alperenboyaci  Things to keep in mind with NiFi authorizations. When a user authenticates with NiFi, they are authenticating only with the node they are connecting with.  That request then gets replicated to all the nodes in the cluster.  This is where the "proxy user request" authorization policy comes in to play.  So the node the user authenticated in to replicated the request on behalf of that authenticated user to the other nodes (this way the user does not need to authenticate to all nodes).   Some request that get replicated will result in data being returned by other nodes in the cluster for the purpose of displaying that data information on the origninating host where the user initiated the action. So in order for that originating node to display that data, it must be authorized to do so. So while you may have authorized your authenticated user to "view the data" for a specific component, you also need to authorize your hosts for the same.  This is why you see the "Insufficient Permissions" dialog box telling you that "server 2" is not authorized to "view the data" for the component requested. In addition to "proxy user requests", nodes would need to be authorized for: 1. "view the data" in order to list a connection queue of a component. 2. "modify the data" in order to empty a connection queue of a component. (your user would also need these same authorizations) Other policies that apply to the NiFi hosts include: 1. "receive data via site-to-site" set on "remote input ports" (set to your local hosts if you are sending FlowFiles to yourself within NiFi. Set to host of other external NiFi instances if receiving FlowFiles from a Remote Process group not on this cluster's hosts). 2. "send data via site-to-site" set on a "remote output port" (same logic as above) 3. "retrieve site-to-site details" set from global menu --> policies. (same logic as above) NiFi's component level authorization policies are set via the "key icon" found in the "operate panel for components added to canvas: Clicking on the key icon will display: Depending in the component selected, some policies may be greyed out if they do not apply to that component.  While NiFi allows you to set policies on every component, it is more typical to set policies on the process group because components (processors, controller services, child process groups) inherit permission from the parent PG unless authorizations have been set on the component itself. When you first installed NiFi and started it for the first time, NiFi would have generated the root level Process Group (it is the canvas you see when you access the UI).  With nothing selected on the canvas, the operate panel should display the name name and UUID for the root Process group (this assumes you have not click and entered a child process group).  There are bread crumbs in the lower left corner to help user navigate the hierarchy of parent --> child Process groups. (label furthest to the left is the root process group:   I know this was a bit extra detail, but hope it helps you be more successful. If you found any of the supplied responses assisted with your queries, please take a moment to login and click on "Accept as Solution" below each of those post.  As a community we want to make sure we share the path to solutions with other community members through "Accept as Solution" marked responses. Thank you, Matt ... View more

@sam_s0ni  Verify that all 3 NiFi nodes are: Running the exact same version of NiFI. Running the exact same version of Java 8 or 11 (only supported by newest releases) Contents of the NiFi lib directory, extensions directory, and any custom lib directories contain exact same content. Try removing the NiFi work directory from all three nodes.  On restart NiFi will rebuild the contents of the work directory from above lib and extensions folders. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt ... View more

@Shanoj You really do not want to be using the Apache NiFi 0.x line anymore.  That release version is more than 6 years old.  Many many security bug fixes and improvements have been made since that time.  Not to mention that the Cluster Manager was a single point of failure.  If it goes down you lose all access to your NiFi.  The Apache NiFi 1.x line introduced 0 master clustering allowing users to access NiFi from any node in the cluster. While I strongly encourage the use of an external zookeeper with Apache NiFi 1.x line, NiFi does offer and embedded zookeeper option. https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#embedded_zookeeper I encourage you to read through the following walkthrough documentation: https://nifi.apache.org/docs/nifi-docs/html/walkthroughs.html It includes sections on installing NiFi, Securing it, and deploying a cluster which even covers using the embedded zookeeper. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt ... View more

@Neil_1992  I agree that the first step here is to increase the open file limit for the user that owns your NiFi process. check your current ulimit by becoming the user the user that owns your NiFi process and executing the "ulimit -a" command. You can also inspect the /etc/security/limits.comf file. NiFi can open a very large number of open files. The more FlowFile load, the larger the dataflows, the more concurrent tasks, etc all contribute to open file handles. I recommend setting the ulimit to a very large value like 999999, restarting NiFi and seeing if your issue persists. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt ... View more