@Darryl  You will certainly have issues using localhost even outside docker with a multi-node NiFi cluster since only the NiFi node installed in the same server as Keycloak would have any chance of connecting with Keycloak.   Understand that a NiFi cluster can bae accessed form any node in the NiFi cluster. So first thing i would have suggested is using the hostname of the actual server that is hosting Keycloak rather then localhost. I am not familiar with Keycloak myself and the article you reference (while i know the author) is very old. A lot has likely evolved/changes with regards to security since it was written.  Glad to hear you had some success even if it was outside of Docker and thank you with sharing what you learned with the community. Matt ... View more

@Darryl  Please start a new community question around your new keycloak integration issues with NiFi. Thank you, Matt ... View more

@jfs912  You should not be configuring your NiFi with larger then necessary heap.  Doing so just leads to very long stop-the-world garbage collection events.   The simple flow you have described would use very little heap memory. So you have a 5 node cluster and each nodes has files in some local directory that each node is pulling from? Is that local directory a mounted directory that is mounted to all nodes or each nodes has its own set of files in the local directly from which getFile is pulling from? Are you seeing backpressure being applied on any of the connections between your processors?  When backpressure is being applied to the upstream processor, NiFi will not schedule that upstream processor until that backpressure is removed. If you can tolerate some latency in your dataflow, you can get better throughput performance with some processors by increasing the Run Duration as well. Dataflow design best practices and designs can also improve performance and better load distribution across all the nodes in your cluster.  You want to minimize as much as possible one node doing bulk of the work load. Adjusting concurrent tasks has multiple elements to it. What is the current CPU load average on each of yoru 5 servers?  First need to know if there is capacity to run more parallel threads. How large is the configured timer driven thread pool in NiFi?  It is from this configured thread pool that all concurrent tasks used by processor components comes from.  If this pool is small, adding more concurrent tasks to processors will improve nothing.  Ability to increase the size of this thread pool is dependent on node's cpu load average.  Thread pool is also applied per node.  So when set to 10 that is 10 threads per each node in your 5 node cluster.    If cpu load average is not high and you increase the size of the Timer Driven Thread pool, you'll want to make small incremental changes to the concurrent tasks on processor and monitor impact on CPU load average. Please help our community grow and thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@BK84  The exception is caused by: Caused by: java.lang.IllegalStateException: The specified class 'org.apache.nifi.controller.state.providers.raft.RaftStateProvider' is not known to this nifi. Your NiFi knows nothing about this class meanung the nifi lib directory does not contain any nar that includes this class: org.apache.nifi.controller.state.providers.raft.RaftStateProvider I also see nothing in Apache NiFi Jira nor in the NiFi 2 documentation mentioning this class or any type of raft state provider. Looking at the properties configured with this provider class, they match the zookeeper properties.  So I suspect you should be using this class instead in your stet provider: org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider   Please help our community grow and thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@Ghilani  The Invalid SNI is caused by the server certificate presented in the TLS exchange not containing a SubjectAlternativeName (SAN) matching the hostname in the request URL. Valid SAN entries are required by the latest version of Java. The solution is to issue new certificates for your NiFi instance(s) that contain all possible SANs used when connecting to the NiFi. So you should have a SAN entry for you NiFi hostname as well as any alternative names used in connection URLs like "nifi" coming form your proxy to NiFi. Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@Darryl    sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target   above indicates a trust issue in a TLS handshake.  However, there is not enough logging shared to understand what action occurred leading to this exception. The ldap-user-group-provider and ldap-provider configurations you shared has this set: <property name="Authentication Strategy">SIMPLE</property> SIMPLE does not use TLS, so I don't believe this is where your exception is coming from unless you made other configuration changes there. My guess would be a Mutual TLS exchange trust issue between nodes in your NiFi cluster resulting in the exception. No the Toolkit included with NiFi 2 no longer includes the tls-toolkit.  This is because there are som many sites and means to create server and client certificates that this really should not be something the done or maintained by the NiFi service. That being said: you could use the tls-toolkit from a NiFi 1.x to create your certificates. You could use an external service to create signed certificates for your NiFi servers. You could create self signed certificates manually yourself. NiFi server certificate minimum requirements are: The Keystore can contain only 1 private key entry. The Private key entry must support both clientAuth and serverAuth ExtendedKeyUsages The private key must contain a SubjectAlternativeName (SAN) entry for the hostname of the server on which the NiFi service is installed. Typically you would create a unique private key for each NiFi node. The truststore can contain 1 too many trusted cert entries (public certificate).  But in order for trust to be established between client and server the truststore must contain the trusted cert entry for the signer of the private key of each NiFi host.    Please help our community grow and thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt   ... View more

@Darryl  I understand your use case.  I use LDAP on every NiFi install I have, but I am not running on Docker containers.  I was highlighting the exception cause and other various configuration issues that will block success with your NiFi configuration setup. As far as making it so your Docker containers can communicate with one another, this discussion in the Docker forum should be able to help you: https://forums.docker.com/t/how-to-create-a-network-of-containers-that-can-communicate-with-each-other-interchangably/134292 Please help our community grow and thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@Darryl  Welcome to the Cloudera Community! From the exception shared the communications exception is Caused by: java.net.UnknownHostException: ldap You have the ldap-user-group-provider configured in your authorizers.xml with: <property name="Url">ldap://ldap:3890</property> But the NiFi host server is unable to resolve "ldap" to an ip address and thus can not communicate with your ldap.  On NiFi startup, it will execute the ldap-user-group-provider to pull in your ldap users and groups based on your ldap search configuration and filters.  If this provider can't execute successfully, NiFi will not continue to start. Also note that you have the same URL configured in your ldap-provider login-identity-providers.xml file.  I see other issues you will hit later once get past the above in your ldap-provider. Also your user search base does not look correct: <property name="User Search Base">CN=users</property> Unfortunately without a sample user ldap entry output, I really can't tell you what the correct value would be for your setup, but the Search Base would typically be as follows: Assume a ldap user has the DN of "cn=John,ou=users,ou=mycompany,ou=nifi". Your Search Base would be "ou=users,ou=mycompany,ou=nifi" In your ldap-user-group-provider, you trying to return all entries with uid=* (The "uid=*" search filter is not necessary here) and then extracting the value from the "cn" field as your user identity for each user returned.  However, in your ldap-provider you are expecting users to login with their "sAMAccountName".  Do your ldap user entries even have a "sAMAccountName=" field/entry?  Assuming your ldap does, yoru ldap-provider is configured to use the user's full DN (<property name="Identity Strategy">USE_DN</property>) as the user's identity.  That full DN will not match the user strings being returned by the "cn" field from your ldap-user-group-provider.   so you should set the below instead in your ldap-provider: <property name="Identity Strategy">USE_USERNAME</property> This will result in the username entered at the login window being used upon successful authentication as the user identity in NiFi which will be looked up by the authorizer to determine authorizations. And if you are logging in user a user's sAMAccountName, then you should set the below in your ldap-user-group-provider: <property name="User Identity Attribute">sAMAccountName</property>   Please help our community grow and thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@ayu_dev  Welcome to the Cloudera Community. Please start a new Cloudera Community question rather then adding a new thread on an existing question from over a year ago.  You'll get much better traction there.  Your query is unrelated to the original question in this thread around log rotation and retention policies. Fell free to ping me in your new community question and I'll be happy to take a look. Thank you, Matt ... View more