@SAMSAL  Looking at output provided, you appear to running your Apache NiFi on Windows.  It appears this issue was raised 2 days ago against M3 in Apache Jira here: https://issues.apache.org/jira/browse/NIFI-13394 It is currently unresolved.  You can certainly create an Apache jira account and add additional comments to this jira with your detailed findings. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@SAMSAL  This is some great detail.  I believe you are hitting this bug that has been fixed for the next 2.0.0 milestone release (M4): https://issues.apache.org/jira/browse/NIFI-13329 There will eventually be a 2.0.0 RC release.  That will be the first official Release Candidate for new 2.x versions that will follow all these development milestone releases.   You can create an Apache Jira account that would give you the ability to raise new issues you find directly in the Apache NIFi project.  This is best way to bring your finding to the developer community. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@helk  You can use a single certificate to secure all your nodes, but i would not recommend doing so for security reasons.  You risk compromising all your host if any one of them is compromised.  Additionally NiFi nodes act as clients and not just servers.  This means that all your hosts will identify themselves as the same client (based off DN).  So tracking client initiated actions back to a specific node would be more challenging.  And if auditing is needed, made very difficult. The SAN is meant to be used to differently.  Let's assume you host an endpoint searchengine.com which is back by 100 servers to handle client requests.  When a client tries to access searchengine.com that request may get routed to anyone of those 100 servers.  The certificate issues to each of those 100 servers is unique to each server; however, every single one of them will have the searchengine.com as an additional SAN entry in addition to their unique hostname.  This allows the host verification to still be successful since all 100 are also known as searchengine.com. Your specific issue based on shared output above is caused by the fact that your single certificate does not have "nifi01" in the list of Subject Alternative Names (SAN).  It appears you only added nifi02 and nifi03 as SAN entries. The current hostname verification specs no longer use DN for hostname verification.  Only the SAN entries are used for that.  So all names(hostnames, common names, IPs) that may be used when connecting to a host must be included in the SAN list. NiFi cluster keystore requirements: 1. keystore can contain only ONE privateKeyEntry. 2. PrivateKey can not use wildcards in the DN. 3. PrivateKey must contain both clientAuth and serverAuth Extended Key Usage (EKU). 4. Privatekey must contain at least one SAN entry matching the hostname of server on which keystore will be used. The NiFi truststore must contain the complete trust chain for your cluster node's PrivateKeys.  On truststore is typically copied to and used on all nodes. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@Dave0x1  Some general related information: 1. Java uses heap as needed, but for efficiency does not run Garbage Collection (GC) to free unused heap until typically over 80% allocated heap usage. So not unexpected to see heap utilization of 70% even once data is processed out of your dataflow(s).  So there is nothing unexpected or alarming specific to that 70% heap utilization in itself.  You probably want to look at the GC events (partial and full GC) to see how many and how often they are happening.  What are your current XMS and XMX heap setting for your NiFi.  Heap is requested during execution of NiFi components.  NiFi does not manage the heap or its clean-up, that is a process handled by Java. 2.  When a component is configured with "primary node" execution, it will only be scheduled on the currently elected primary node.  The FlowFiles generated will then only exist on the primary node unless you design into your dataflow(s) redistribution (typically done via load balance configuration on downstream connection of the primary node execution processor component) of those FlowFiles across all your nodes for further downstream processing.  Even with distribution, there will be some deviations in resource usage since you are still doing some additional work on just the primary node. 3. The primary node and cluster coordinator nodes are elected by Zookeeper (ZK) and can change.  Commonly there is some event that triggers a change (current primary node stops heart-beating to ZK, current primary node disconnects from cluster, cluster or primary node is restarted, current primary node shutdown, etc..  You could look at the individual node events in the cluster UI to see when the primary node change to see if aligns with any of these event types.   But even with a primary node change that would not shift heap usage to another node.  While i see nothing of concern with what was shared in your post, the things you want to watch for is memory related logs of concern.   Java out of memory (OOM) alerts indicate a problem that must be addressed. OOM can happen when your designed dataflow(s) try to consume more memory then is allocated to your JVM.  Or is a sign that GC can keep up with the memory demand.  Or heap usage exceeded 80% utilization and GC run was unable to free enough unused heap to get back below that 80% utilization.  While not out of memory, this indicates your dataflow(s) use high active heap (common offenders are merge or split based processors with excessively high number of FlowFiles being merged in a single transaction or a single split producing an excessively large number of output split FlowFiles in a single transaction.  The embedded documentation (usage docs) for the various components indicate if a component has the potential high heap or high CPU usage in the "System Resource Consideration" section.   Here is example form MergeContent: Hope you find this information useful for your query. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@tcherian  NiFi certificates must meet the following criteria: 1.  No wildcards used in the subject DistinquishedName (DN)  2. Included both clientAuth and serverAuth in the ExtendedKeyUsage (EKU) 3. Contains one or more SubjectAlternativeName (SAN) entries. 4. Keystore can only contain 1 PrivateKey entry There are many resources on the web for generating your own self-signed certificates and adding them to a PKCS12 or JKS keystore.  The "Keystore" and "truststore" are both just keystores.  The NiFi "Keystore" contain the PrivateKey entry which Is used by NiFi to identify itself as the server (serverAuth) when connecting to it and as the client (clientAuth) when connecting outward as a client (such as talking to other NiFi's, NiFi-Registry, etc).  The NiFi "truststore" contains one too many TrustedCert entries.  It is common to use the default Java cacerts file (which is just a jks keystore) and add additional TrustedCert entries to it.  The trustedCerts are the public certs that correspond to the PrivateKey that you should never share.  The Trusted certs are the signers of the private keys.  There are intermediate and root trusted cert keys.  An intermediate trust is one where the owner and signer are not the same DN.  A root trust is one where the owner and signer are the same DN.   So you might create a PrivateKey that is signed by intermediate Certificate Authority (CA) and that intermediate CA would be signed by another intermediate CA or a root (CA). The chain of signers between intermediate and root is known as the trustchain.  The Truststore needs to contain complete trust chains for your PrivateKey.    There are even free services out there like Tinycert, but you can also use openssl and keystool to generate self-signed certificates and import them to a keystore.  Just google how to create a certificate and how to import certificate into a keystore. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more