@davehkd You are definitely having issue with your embedded zookeeper. The shared exception indicates NIFi is not able to communicate with it.  The KeeperLoss exception thrown by the ZK client in NiFi most commonly is seen when the ZK does not have quorum because not enough nodes are part of ZK cluster or not all nodes are able to talk to one another. A New NiFi with no flows to load should start very quickly (never 12+ hours). Yours is unable to connect to the zookeeper that is included with and started as part of the NiFi process startup. 1. I'd start with making sure all of your nodes can resolve the zookeeper hostnames (nificlient1, nificlient2, and nificlient3) to proper reachable IP addresses.  If not make sure you add these to your local hosts file on each server so this is possible. 2. I'd check to see if zookeeper server is running on each host and listening on the configured ports. Make sure no other processes are using those ports thus blocking zookeeper from being able to bind to them. 3. Make sure you don't have any firewalls blocking connections to the zk ports.   If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt ... View more

@RodolfoE  Try using the absolute path to the certificate pem file rather than just the filename. If you are executing your curl command from within the directory where your pem file resides, try using "./<pem filename>". Otherwise curl may try looking up yoru pem filename as if it were an alias/nickname it expects find in some NSS database. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt ... View more

@SachinMehndirat    The exception comes from java and not NiFi's core code base. The exception is possibly being thrown as a result of a failed mutual TLS handshake.  Without the complete stack trace and preceding log output it is difficult to give a detailed response here, so below is based on assumptions from what little has been shared. However, when you see "chain" issues (keeping in mind that output shared is incomplete), that points at missing TrustedCertEntry(s) in the truststore used one side or the other of that TLS handshake.   Both the client and server sides of a TLS handshake present their clientAuth or serverAuth certificate info.  These PrivateKeyEntry certificates will have an owner and issuer (signing Certificate Authority (CA) for the owner).  In order for the receiving sides to trust the certificate presented from the other side, it must trust that CA.  This means that within its truststore, it must have a TrustedCertEntry for the CA that signed the PrivateKey presented in the TLS handshake.  BUT... it does not end there.  The CA may be an intermediate CA meaning that the certificate for that CA was signed by yet another CA.  In that case the truststore would also need the TrustedCertEntry for the next CA in that chain of authority.  A complete trust "chain" would consist of all CAs from signer of PrivateKey to the root CA (root CA public key will have same Distinguished Name (DN) for both owner and signer of the TrustedCertEntry). So on one side or the other you do not have the complete trustchain in the truststore. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt ... View more

@davehkd Those log lines look like expected output in the nifi-app.log during startup process.  Niether is an ERROR. - Are you using the embedded zookeeper or an external ZK (strongly recommended)? The first indicates that ZK has not elected a cluster coordinator yet.  This can happen if ZK does not finished coming up yet or does not yet have quorum.  ZK required an odd number of hosts (3, 5, etc) to achieve quorum and without quorum will not function.  3 is the recommended number of ZK hosts to support NiFi.  Once ZK is up and established quorum, I'd expect that WARn log message to go away. The second info message simply means that this node was unaware of an elected cluster coordinator and has requested to be elected to that role.  ZK responded that it had already elected some other node as the cluster coordinator.  This node should receive the elected cluster coordinator from ZK and you should then start seeing in the logs your nodes sending heartbeat messages to the elected cluster coordinator (even the elected cluster coordinator when send a heartbeat to itself.). Only the cluster coordinator will log receiving and processing x number of received heartbeats. My guess here is that you may not have given it enough time to full launch.  When you start NiFi via "../bin/nifi.sh start", it executes the bootstrap process, the bootstrap process then kicks off the main child process for NiFi.  That process you'll see through the nifi-app.log output as it progresses.  NiFi is fully up once you see the log line that states NiFi Ui is available at the following URLs. Now that the NiFi node is fully up it attempts to communicate with ZK and establish itself as part of a cluster.  Especially with embedded ZK in use, this can be delayed until all nodes are up so that ZK has quorum.  So first node to come up may log more lines like above then last node to finish startup.  NiFi handles election based on configuration of these two properties in the nifi.properties file: nifi.cluster.flow.election.max.wait.time (default is 5 mins) nifi.cluster.flow.election.max.candidates (No default, but should be set to number of NiFi instances in cluster)   So basically, NiFi nodes will wait up to 5 minutes or until the configured number of candidates have connected with ZK before flow election happens and NiFi finishes coming up.  Accessing the UI before this happens would result in flow election still in progress. Make sure that the "../conf/<nifi config files>" are all configured same across all nodes with exception of node specific properties like hostnames, keystores, truststores, etc. Hope that after some additional time, your NiFi cluster did finally come up for you. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt   ... View more

@sarithe  NiFi component processors are part of pluggable nar in NiFi.  They are separate from the core NiFi code.  Processors are designed to log their output base on their component processor class.  It then becomes the responsibility of the logback to route those log messages to the appropriate appender.  There is nothing in the log output produced by a component processor that will inherently identify which parent process group it resides within.  But if you were to use a consistent processor naming structure in each of your Process Groups (PG), you may be able to setup some creative filtering in logback based on that naming structure. Bulletins however do include details about the parent Process Group in which the component generating the bulletin resides.   You could build a dataflow in yoru NiFi to handle bulletin notification through the use of the SiteToSiteBulletinReportingTask which is used to send bulletin to a destination remote import port on a target NiFi.  A dataflow on the target NiFi could be built to parse the received bulletin records by the bulletinGroupName json path property so that all records from same PG are kept together.  These  'like' records could then be written out to local filesystem, remote system, used to send email notifications, etc... Example of what a Bulletin sent using the SiteToSiteBulletinReportingTask looks like:   { "objectId" : "541dbd22-aa4b-4a1a-ad58-5d9a0b730e42", "platform" : "nifi", "bulletinId" : 2200, "bulletinCategory" : "Log Message", "bulletinGroupId" : "7e7ad459-0185-1000-ffff-ffff9e0b1503", "bulletinGroupName" : "PG2-Bulletin", "bulletinGroupPath" : "NiFi Flow / Matt's PG / PG2-Bulletin", "bulletinLevel" : "DEBUG", "bulletinMessage" : "UpdateAttribute[id=8c5b3806-9c3a-155b-ba15-260075ce9a6f] Updated attributes for StandardFlowFileRecord[uuid=1b0cb23a-75d8-4493-ba82-c6ea5c7d1ce3,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1672661850924-5, container=default, section=5], offset=969194, length=1024],offset=0,name=bulletin-${nextInt()).txt,size=1024]; transferring to 'success'", "bulletinNodeId" : "e75bf99f-095c-4672-be53-bb5510b3eb5c", "bulletinSourceId" : "8c5b3806-9c3a-155b-ba15-260075ce9a6f", "bulletinSourceName" : "PG1-UpdateAttribute", "bulletinSourceType" : "PROCESSOR", "bulletinTimestamp" : "2023-01-04T20:38:27.776Z" }     If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt ... View more

@doubtguy  Is there a consistency in the timing of when it stops tailing the file (after 15 minutes of starting processor for example)? Interesting that it starts tailing again when you cd to the mounted directory.  This seems characteristic of a SMB/CIFS mount that has gone idle.   If you set up a process to touch that file or cd to that mounted directory every 2 mins or so, does the issue go away? If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt ... View more

@davehkd  The exception you have shared points at the following property being set to false in the nifi.properties file: nifi.cluster.protocol.is.secure=false NiFi nodes communicate with one another over HTTP when this is set to false.  When set to true NiFi nodes with communicate with one another over HTTPS.  Since you have this set to false, it is complaining that you do not have a your NiFi configured to with an HTTP port in the following property in the nifi.properties file: nifi.web.http.port Out of the box, Apache NiFi is configured to start securely over https as a standalone NiFi instance using the Single-User authentication and single-user-authorizer providers: https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#single_user_identity_provider The intent of this provider is to provide a means for easily and quickly starting up a secure NiFi to become familiar with or evaluate NiFi.  It gives that single user full access to everything and provides no mechanism for setting up and authorizing any additional users. When switching to a NiFi cluster, You'll need to setup proper authentication and authorization providers that support secure NiFi clusters.  In a secured NiFi cluster setup, the NiFi nodes will need to authenticate via their certificates over a mutual TLS handshake (unless set to be unsecure as you have setup which I strongly do not recommend).  This in turn means that the NiFi cluster nodes will need to have authorizations setup for proxy, data access, and controller access which the single-user-authorizer does not support. Additionally the single user identity-provider by default on NiFi startup creates a random user name and password which is going to be unique per node. This will not work in a cluster setup since actions performed on node 1 will be replicated to nodes 2 - x nodes as the authenticated user of node 1.  However, nodes 2 - x will not know anything about that user and thus fail authorization.   The single user authentication provider provides a mechanism for you to set a specific username and password which you could make the same on all instance of NiFi. ./bin/nifi.sh set-single-user-credentials <username> <password> My suggestion to you is to first setup a standalone NiFi securely using yoru own configuration for user authentication and user authorization: For user authentication, follow this section of the admin guide: https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#user_authentication The most commonly used method of user authentication used is the ldap-provider: https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#ldap_login_identity_provider For NiFi authorizations, follow this section of the NiFi admin guide: https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#multi-tenant-authorization The most basic managed setup utilizes all of the following authorization providers in below specific order in the authorizers.xml file: https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#fileusergroupprovider https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#fileaccesspolicyprovider https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#standardmanagedauthorizer   These are actual in template format in the default authorizers.xml included with NiFi.  They are likely commented out. Once you have a a secured standalone NiFi instance working, then I would move on to setting up your NiFi cluster.  You'll need to add your NiFi cluster nodes to the authorizers file-user-group-provider and file-access-policy-provider as part of that process which would require you to remove the users.xml and authorizations.xml files generated by those providers so they get recreated to support your initial cluster needed authorizations. These files are only generated by those providers if the do NOT already exist.  Config changes in the providers will not trigger new or modified files. I know there is a lot to take in here, but this will set you up in the best possible way for success. If you found that this response helped with your query, please take a moment to login and select "Accept as Solution" below each response the helped you. Matt     ... View more

@Mostafa12345678  I'd recommend providing more detail in your query here to include things like the version of NiFi being used, the NiFi processor component being used and its configuration, and the full stack trace error from the nifi.app.log. Thanks, Matt ... View more