@rajivswe_2k7  Why are you fetching same files twice? I don't follow the "fail" all 5 if any one of them fails.  You successfully wrote some of them to destinations.  So what action are you taking when a partial failure happens (for example only 1 of 5 fails to write to archive? Why not just build dataflow around failure relationships to notify you of the specific files that failed? Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@Armel316  Since you only have two user group providers (ldap and file), that means that both are returning user "xxx".  If the ldap-user-group-provider is returning user "xxx" you don't want to define that same user through the file-user-group-provider.    What this means is that the users.xml file that the file-user-group-provider is loading users from on startup contains user "xxx".    The file-user-group-provider will ONLY generate a users.xml file if one does not already exist.  If one already exist the file-user-group-provider will NOT make any modifications to an existing users.xml if you modify the provider configuration.  Once a users.xml file exists, the expectation is that all future user/group modification happen via the UI. NOTE: The users.xml does not contain any users or group being loaded by other providers in to NiFi memory. So you have two options here: Rename the current users.xml file so a new one is created on startup with only the 3 defined node-identities.  (this is preferred method) Manually modify the users.xml to remove all users that are being synced by the ldap-user-group-provider. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@rajivswe_2k7  What is the use case for wanting to hold downstream processing of FlowFiles until a min 10 are queued?  This is not a typical use pattern for NiFi.  While I am sure it could be done without using a scripting processor,I don't think it would be as efficient in terms of resources. Creative use of the MergeContent processor comes to mind here. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@Armel316  You used the wrong composite provider "composite-user-group-provider". Since one of your providers is the "file-user-group-provider" (a configurable provider- meaning one you can add users and groups to via the UI), you need to use the "composite-configurable-user-group-provider". I see in the Apache NiFi-Registry documentation that the example is wrong.   https://nifi.apache.org/docs/nifi-registry-docs/html/administration-guide.html#composite-implementations   <userGroupProvider> <identifier>composite-configurable-user-group-provider</identifier> <class>org.apache.nifi.registry.security.authorization.CompositeConfigurableUserGroupProvider</class> <property name="Configurable User Group Provider">file-user-group-provider</property> <property name="User Group Provider 1">ldap-user-group-provider</property> </userGroupProvider>   What do you see in the nifi-registry-app.log when you try to start version control? What is the output from: openssl s_client -connect <nifi-registry hostname>:<port> -v openssl s_client -connect <nifi node hostname>:<port> -v Above can be checked to verify proper trust exists between NiFi and NiFi-Registry. What is the exact case sensitive user identity displayed in the upper right corner of your NiFi UI for the user authenticated to NiFi that is attempting to start version control? Does the same user Identity (case sensitive) exist in your NiFi-Registry and been assigned read, write, and delete on the bucket? Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt   ... View more

@phadkev  Would need to see more of the nifi-app.log to better understand what is going on here. Are you seeing the same org.apache.nifi.controller.StandardProcessorNode Timed out while waiting for OnScheduled exception for other components or just this executeScript processor? The exception itself is generic and could be thrown for any processor class. Are you ever seeing the log line telling you the NiFi UI is available at the following urls? If so NiFi is up. Are you seeing NiFi shut back down with some exception and stack trace in the nifi-app.log. What you shared implies NiFi is having issues scheduling this specific processor to execute.  This could very well be caused by an issue with the custom script that was build and used in this processor. If you NiFi is really not coming up, you could modify the nifi.properties file by changing  "nifi.flowcontroller.autoResumeState=true" to "nifi.flowcontroller.autoResumeState=false".  This will allow you NiFi to start without starting any processors.  You could then search the UI for the ExecuteScript component processor with id "acb441ba-c36b-1fdd-53f2-3a4821d43833".    Disable it and start all your other processors.  Restart your NiFi to see if you still have any issues.   This isolates the issue to this processor and your script. Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt   ... View more

@ZNFY  You may find this other community thread useful for setting up a Docker NiFi cluster: https://community.cloudera.com/t5/Community-Articles/NiFi-cluster-sandbox-on-Docker/ta-p/346271 The Apache NiFi Docker is built around setting up a standalone (non-clustered) instance of NiFi. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@Armel316  I am no nginx expert, but what you need to do is setup your NiFi-Registry authorizers.xml similar to how you already setup your NiFi-authorizers.xml.  You need to add the missing composite-configurable-user-group-provider (setup with file-user-group-provider and ldap-user-group-provider) and modify your file-access-policy-provider to point at the composite-configurable-user-group-provider instead of pointing directly at the ldap-user-group-provider.  The only difference between NiFi and NiFi-Registry authorizers.xml are the class names. Once the file-user-group-provider is actually being used by your NiFi-Registry, you'll need to access NiFi-Registry UI and login as you admin user so you can setup the required policies for your NiFi nodes: "Can proxy user requests" (R,W,D), and "Can manage buckets" (R). Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@Armel316  I see the single-user-authorizer in your NiFi authorizers.xml (hope you are not using this and your nifi.properties is configured to use the managed-authorizer.) Now of course I can not validate yoru configured providers, but can tell you the structure of your NiFi authorizers.xml is valid. However, the structure of your NiFi-Registry authorizers.xml is not valid. Best to read it from bottom up starting with the authorizer which you have as the managed-authorizer.    It calls the file-access-policy-provider which in turn references the ldap-user-group-provider.  I see that like your NiFi authorizers, your NiFi-Registry authorizers.xml also has the file-user-group-provider configured in it, but the configuration within NiFi-Registry, the managed-authorizer will never use it.    Because the file-user-group-provider is not used, it is not possible to setup the authorization policies I mention in my last response which must be setup for the NiFi node identities. Also make sure that both your nifi.properties and nifi-registry.properties files have the same configured identity.mapping.* properties.  I see you created your node identities using the node full DNs.  The identity mapping patterns if setup might be manipulating/trimming those DNs during a mutualTLS handshake causing them to to not match the full DNs. I suggest you also tail the nifi-registry-app.log  while you try to start version control to capture what identities are being used to check authorizations against. You'll also want to validate the contents of your NiFi and NiFi-Registry truststore.jks to make sure mutual trust can be established between the two services using the clientAuth Private keys found in the keystore.jks files on both services. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@Ytch  NiFi is designed to move data and the ListS3 processor is designed to list s3 objects/files and not directories.   On the downstream side, NiFi does not have a processor that just create directories.  Many out processors will create a missing directory if content is to be written must be written to a directory path that does not exist on the target.  But without content those processor will not create any directories. I am clear on what yoru use case is here, but based on yoru description it is not something any of the stock processors would do.  You would need to write something custom (either your own processor or some custom script you can execute via aNiFi scripting processor). Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more