Thanks. I don't fully understand the differences between the policies you can modify by right clicking on the canvas and selecting "Manage Access Policies" and the policies you can modify by selecting the dropdown on the top right and selecting "Policies". In either case, the right click and "Manage Access Policies" approach worked. ... View more

@melek6199  Let me try to address each of your statements as there appears to be some misunderstanding of how authentication and authorization works between NiFi and NiFi-Registry. I have a 3-node NiFi cluster, and I want to manage it using NiFi Registry. I configured both NiFi and NiFi Registry with a single certificate using the TLS Toolkit. I also set up LDAP integration. I can successfully connect to both NiFi and NiFi Registry individually using my LDAP users. NiFi-Registry does not manage your NiFi cluster.  These are two different services.  NiFi-Registry is used by NiFi to version control process groups created and managed in NiFi.   It is not a security best practice to use one certificate for all your servers.  You should have one certificate per server.  If you have two services (A NiFi node and NiFi-Registry on the same server, they would both use the same certificate).  In production I would recommend using certificates signed by actual legitimate signing authorities versus TLS toolkit generated certificates and truststore) Your keystores must meet the following requirements: Contain only one PrivateKey Entry That PrivateKey entry supports both ClientAuth and ServerAuth ExtendedKeyUsage (EKU). (Note: NiFi-Registry does not require ClientAuth, but no harm in having it) Contains at least one SAN entry that matches the server's hostname on which the certificate is being used.  Your NiFi/NiFi-Registry Truststore must meet following requirements: Contain a TrustedCertEntry for every signer/issuer of the certificates passed in a mutualTLS handshake (The compete trustchain for every certificate that will be used to communicate between NiFi node and with NiFi-Registry). You can use the NiFi TLS toolkit to generate 4 keystores and 1 truststore you can use with your NiFi and NiFi-Registry services, but make sure you are running with the "--subjectAlternativeNames" option.  Those SAN should include the hostnames of the servers on which the services will run. (Now technically you could create one certificate with SANs for all the hosts and then use that one cert on all hosts, but as I said, not a security best practice).   However, the LDAP user that I added and authorized in the Registry does not appear in NiFi. With the certificate user, I can view the bucket in NiFi Registry from NiFi and perform flow version control. But I cannot do this with my LDAP user. The user that authenticates into NiFi-Registry does not need to exist in NiFi; however, any authenticated user identity authenticated into NiFi must exist and have proper authorization in NiFi-Registry in order to conduct version control operation within NiFi. When you ldap-user authenticates into NiFi you will see that user's "user identity" displayed in the upper right corner (Keep in mind that your user is only authenticated into the NiFi node you access the cluster from and not all the NiFi nodes).  When that user attempts to start version control on a process group, NiFi connects and authenticates with NiFi-Registry via a MutualTLS exchange/handshake. In that connection it will proxy the request on behalf of that "user identity" (case sensitive).  This means that not only do the NiFi node clientAuth certificates need to be authorized in NiFi-Registry to read on "Can Manage Buckets " and read,write,delete on "Can Proxy Requests", the NiFi "user identity" need to be authorized on any bucket you want that "user identity" to be able to use for version control.  (let me know if you need help with how a mutualTLS handshake works) Since your NiFi authenticated ldap "user-identity" has not been added and authorized in NiFi-Registry on any buckets, nothing will appear in the list of available buckets for that "user-identity" in NiFi.   NOTE: Even if I generate separate certificates for NiFi and NiFi Registry and trust each certificate independently, the certificate user does not have permission to view the bucket. This is because the certificate user from the Registry is also not created in NiFi. For this reason, I generated both from the same certificate. From what i shared in response to two section above, you can see that the certificates used by the NiFi host are only used to proxy requests to NiFi-Registry on behalf of the "user identity" authenticated with NiFi. There are few things that don't make sense to me in your shared NiFi-Registry configuration:  NiFi-Registry identity-providers.xml: I see you set " <property name="Authentication Strategy">SIMPLE</property>", yet your ldap URL is "ldaps".  This should then also be set to "LDAPS". NiFi-Registry authorizers.xml: You are using the file-user-group-provider which allows you to manually define an initial set of "user identities" on first startup (node edits after to this config will happen if the users.xml already exists during startup).  This provider also allows for the adding of additional "user identities" later via the NiFi-Registry UI directly. NOTE: There is also an available ldap-user-group-provider that can be used to sync select users "user identity" and groups "group-identity" from ldap into your NiFi-Registry list of identities.  This is helpful if you don;t want to manage your ldap user and group identities manually within NiFi and NIFi-Registry. You are the file-access-policy provider which only created the authorizations.xml file if it does not already exist on startup. In it I can see "Initial Admin Identity">CN=nifi_amadeus_admin  is set; however, in your ldap-provider you have configured "Identity Strategy">USE_USERNAME. I can only assume you did similar in your NiFi setup? It is unlikely that when you are logging into your NiFi you are typing the username as "CN=nifi_amadeua_admin" since this would not be the expected value in the "sAMAccountName" ldap field/attribute.  That means your initial admin "user identity" does not match the identity of your authenticated user (unless you have this set because you are using the a certificate to auth in to the services with the above). In the end, there are the following key things that need to know: user-identities must match and are case sensitive.  ("Bob" and "bob" would be treated as two unique user identities.  So the user identity as displayed in upper right corner in NiFi UI must be authorized on specific bucket(s) in NiFi-Registry in order to successfully use  version control in NIFi.  This you do not have setup correctly yet.  NiFi nodes also need to be properly authorized in NiFi-Registry for mange buckets and all proxy permissions.  The node's user identity is comes from the NiFi node's clientAuth certificate full DN.  That full DN can be modified through the use of the identity.mapping properties in the NiFi-Registry.properties file. Note: In your shared NiFi-Registry.properties file the identity.mapping properties are commented out are not in use, so full DN of NiFi node would be used as node's user identity and need to be authorized which corresponds to with full DN used in the file-access-policy provider you have configured. I know above is a lot of information, but wanted you to fully understand how the authentication  and authorization between NiFi and NiFi-Registry works. Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

Thank you @MattWho @linssab for the help. The Groovy script with the ExecuteScript processor is working fine to convert JSON file content to flow file attributes. ... View more

@MattWho Thanks a lot for detailed answers! ... View more

@AndreyDE  Glad I could help.  Using just controlRate by itself has its limitation as well because it does not optimize for performance/throughput.  You are setting some rate at which batches of FlowFiles will be passed downstream.  Doing just this means multiple things can happen: 1.  Rate is to short resulting in additional batches being passed downstream before previous batch completed processing.  This could potentially lead to large backlogs in flows affecting downstream processing just as you were experiencing previously. 2. Rate is set to long resulting in downstream processing of a batch completing well before ControlRate releases next batch.  Results in slower overall throughput. 3. If some exception occurs in downstream processing, nothing would prevent additional batches from being released into that downstream processing creating a huge backlog. The above are handled by the slightly more complex option C. Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

Maybe you can try another tool, for example, ODBC driver? For me, it works.  ... View more

@asand3r  With Archive disabled, NIFi is no longer tracking the files left in the archive sub-directories.  You can remove those files while NiFi is running.  Just make sure you don't touch the active content_repository claims. Matt ... View more

This is on an isolated system, so screenshots are, well, not impossible, but very difficult. PostHTTP (if not listed, it has "No value set"): URL = our url/contentListener Max Batch Size = 100 MB SSL Context Service = StandardRestrictedSSLContextService Send a FlowFile = true Compression Level = 0 Connection Timeout = 360 sec Data Timeout = 360 sec User Agent = Apache-HttpClient/4.5.5 (Java/1.8.0_262) Content-Type = ${mime.type} Disable Connection Reuse = true ListenHTTP: Base Path = contentListener Listening Port = 9443 SSL Context Service = StandardRestrictedSSLContextService HTTP Protocols = h2 http/1.1 Client Authentication = WANT Authorized Subject DN Pattern = .* Authorized Issuer DN Pattern = .* Max Unconfirmed Flowfile Time = 300 secs HTTP Headers to receive as Attributes (Regex) = .* Return Code = 200 Multipart Request Max Size = 15GB Multipart Read Buffer Size = 1MB Maximum Threadpool Size = 400 Request Time Out = 30 secs This particular ListenHTTP receives from both PostHTTP processors in the same instance, in other instances we own, and from many external sources. Yes, it gets hammered all day long. Under normal circumstances, there is no back pressure on the ListenHTTP processor. It only occurs if there is some bottleneck downstream because of some condition not handled gracefully that might cause queues to back up, but that has occurred very rarely. Thanks for your input! ... View more