@rajivswe_2k7  What is the use case for wanting to hold downstream processing of FlowFiles until a min 10 are queued?  This is not a typical use pattern for NiFi.  While I am sure it could be done without using a scripting processor,I don't think it would be as efficient in terms of resources. Creative use of the MergeContent processor comes to mind here. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@Armel316  You used the wrong composite provider "composite-user-group-provider". Since one of your providers is the "file-user-group-provider" (a configurable provider- meaning one you can add users and groups to via the UI), you need to use the "composite-configurable-user-group-provider". I see in the Apache NiFi-Registry documentation that the example is wrong.   https://nifi.apache.org/docs/nifi-registry-docs/html/administration-guide.html#composite-implementations   <userGroupProvider> <identifier>composite-configurable-user-group-provider</identifier> <class>org.apache.nifi.registry.security.authorization.CompositeConfigurableUserGroupProvider</class> <property name="Configurable User Group Provider">file-user-group-provider</property> <property name="User Group Provider 1">ldap-user-group-provider</property> </userGroupProvider>   What do you see in the nifi-registry-app.log when you try to start version control? What is the output from: openssl s_client -connect <nifi-registry hostname>:<port> -v openssl s_client -connect <nifi node hostname>:<port> -v Above can be checked to verify proper trust exists between NiFi and NiFi-Registry. What is the exact case sensitive user identity displayed in the upper right corner of your NiFi UI for the user authenticated to NiFi that is attempting to start version control? Does the same user Identity (case sensitive) exist in your NiFi-Registry and been assigned read, write, and delete on the bucket? Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt   ... View more

@ZNFY  You may find this other community thread useful for setting up a Docker NiFi cluster: https://community.cloudera.com/t5/Community-Articles/NiFi-cluster-sandbox-on-Docker/ta-p/346271 The Apache NiFi Docker is built around setting up a standalone (non-clustered) instance of NiFi. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@Armel316  I am no nginx expert, but what you need to do is setup your NiFi-Registry authorizers.xml similar to how you already setup your NiFi-authorizers.xml.  You need to add the missing composite-configurable-user-group-provider (setup with file-user-group-provider and ldap-user-group-provider) and modify your file-access-policy-provider to point at the composite-configurable-user-group-provider instead of pointing directly at the ldap-user-group-provider.  The only difference between NiFi and NiFi-Registry authorizers.xml are the class names. Once the file-user-group-provider is actually being used by your NiFi-Registry, you'll need to access NiFi-Registry UI and login as you admin user so you can setup the required policies for your NiFi nodes: "Can proxy user requests" (R,W,D), and "Can manage buckets" (R). Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@Armel316  I see the single-user-authorizer in your NiFi authorizers.xml (hope you are not using this and your nifi.properties is configured to use the managed-authorizer.) Now of course I can not validate yoru configured providers, but can tell you the structure of your NiFi authorizers.xml is valid. However, the structure of your NiFi-Registry authorizers.xml is not valid. Best to read it from bottom up starting with the authorizer which you have as the managed-authorizer.    It calls the file-access-policy-provider which in turn references the ldap-user-group-provider.  I see that like your NiFi authorizers, your NiFi-Registry authorizers.xml also has the file-user-group-provider configured in it, but the configuration within NiFi-Registry, the managed-authorizer will never use it.    Because the file-user-group-provider is not used, it is not possible to setup the authorization policies I mention in my last response which must be setup for the NiFi node identities. Also make sure that both your nifi.properties and nifi-registry.properties files have the same configured identity.mapping.* properties.  I see you created your node identities using the node full DNs.  The identity mapping patterns if setup might be manipulating/trimming those DNs during a mutualTLS handshake causing them to to not match the full DNs. I suggest you also tail the nifi-registry-app.log  while you try to start version control to capture what identities are being used to check authorizations against. You'll also want to validate the contents of your NiFi and NiFi-Registry truststore.jks to make sure mutual trust can be established between the two services using the clientAuth Private keys found in the keystore.jks files on both services. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@Ytch  NiFi is designed to move data and the ListS3 processor is designed to list s3 objects/files and not directories.   On the downstream side, NiFi does not have a processor that just create directories.  Many out processors will create a missing directory if content is to be written must be written to a directory path that does not exist on the target.  But without content those processor will not create any directories. I am clear on what yoru use case is here, but based on yoru description it is not something any of the stock processors would do.  You would need to write something custom (either your own processor or some custom script you can execute via aNiFi scripting processor). Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@Armel316  Lets discuss first what needs to happen successfully when a secured NiFi is connecting with a secured NiFi-Registry. When NiFi connects to NiFi-Registry client URL, it does so using the either the keystore and truststore configured in the NiFi-Registry Client's StandardRestrictedSSLContextService setup within NiFi  or using the keystore and truststore setup in the nifi.properties when no StandardRestrictedSSLContextService was setup in the NiFi-Registry Client.  A mutualTLS handshake will be attempted between NiFi and NiFi-Registry.  NiFi-Registry will "WANT" the client (NiFi) to provide clientAuth certificate.  If one is not provided, NiFi-Registry will proceed using the anonymous user (Anonymous user only has read on public buckets which align with what you shared from developer tools).  So an unsuccessful  mutualTLS handshake is most likely your issue currently. To answer the possible next question.... If It shows "read" on the bucket in developer tools, why does NiFi UI does not show the bucket? This is because the UI opened was for starting version control on an process group on the NiFi canvas.  This UI will only show buckets for which the user identity currently authenticated into NiFi is authorized read and write on. Next question: My NiFi user is authorized read and write on the bucket in NiFi-Registry, so why is bucket not showing? NiFi authenticates with NiFi-Registry via a mutualTLS handshake.  The client/user identity derived form the clientAuth certificate DN for the NIFi node is used as the identity passed to NiFi-Registry.  Assuming the MutualTLS handshake is successful, the node user identity must be authorized "read" on all buckets and "read, write, and delete" on proxy user requests.  This allows the node to proxy request on behalf of the user authenticated in NiFi.  So only the buckets for which the authenticated user identity in NiFi has been authorized read, write, and delete on within NiFi-Registry will be shown in the list. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more