@wbivp  Without full ldapsearch output for a user and group, I'd have to make some guesses and assumption with regards to your specific setup, but it looks like you are using AD so here is a configuration you may want to try: <property name="User Search Base">OU=DEVELOPER,DC=msad,DC=local,DC=co</property> <property name="User Object Class">user</property> <property name="User Search Scope">SUBTREE</property> <property name="User Search Filter"></property> <property name="User Identity Attribute">sAMAccountName</property> <property name="User Group Name Attribute">memberOf</property> <property name="User Group Name Attribute - Referenced Group Attribute">distinquishedName</property> <property name="Group Search Base">OU=Groups,OU=DEVELOPER,DC=msad,DC=local,DC=co</property> <property name="Group Object Class">group</property> <property name="Group Search Scope">SUBTREE</property> <property name="Group Search Filter"></property> <property name="Group Name Attribute">cn</property> <property name="Group Member Attribute">member</property> <property name="Group Member Attribute - Referenced User Attribute">distinquishedName</property>   Another thing you should do is enable DEBUG on the ldap-user-group-provider class.  With DEBUG, this provider will output to the nifi-app.log the list of user and groups strings that are being synced from your AD and also show the associations discovered between them.   Adding this logger involves editing the contents of the NiFi's logback.xml file.  Scroll down in the logback.xml until you start seeing lines that start with "<logger..../>" and insert the following new line: <logger name="org.apache.nifi.ldap.tenants.LdapUserGroupProvider" level="DEBUG"/>   Those user and groups strings are absolute and case sensitive.  So they must identically exist with Ranger when they get associated to policies you setup.   If the strings logged in the nifi-app.log do not match what you have in Ranger exactly, you'll need to adjust your configuration until they do.  If the strings don't match but can be manipulated to match using java regular expressions, take a look at the user and group mapping pattern [1] capabilities that can be configured in the nifi.properties file (there are different properties for manipulting identity/user strings and group strings.  These mappings are evaluated against the strings returned by the providers in the authorizers.xml and returned by the login providers before they are used to check authorization.  What is output to the log would be post any mapping being applied. [1] https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#identity-mapping-properties Hope this helps, Matt   ... View more

@wbivp  The exception form the logs you shared is telling you that the ldap-user-group-provider is failing to connect to the configured ldap server using the manager DN and Manager password that has been configured.  So first thing you need to do is validate, re-enter, and test your manager DN and password from command line to make sure they work from the NiFi host.  Any need to engage with your ldap team. Here is your ldap-user-group-provider (I re-order the properties to make it easier to read: <userGroupProvider> <identifier>ldap-user-group-provider</identifier> <class>org.apache.nifi.ldap.tenants.LdapUserGroupProvider</class> <property name="Authentication Strategy">SIMPLE</property> <property name="Manager DN">CN=CDP Admin,OU=DEVELOPER,DC=msad,DC=local,DC=co</property> <property encryption="aes/gcm/256" name="Manager Password">Yf41</property> <property name="TLS - Keystore"></property> <property name="TLS - Keystore Password"></property> <property name="TLS - Keystore Type"></property> <property name="TLS - Truststore"></property> <property name="TLS - Truststore Password"></property> <property name="TLS - Truststore Type"></property> <property name="TLS - Client Auth"></property> <property name="TLS - Protocol"></property> <property name="TLS - Shutdown Gracefully"></property> <property name="Referral Strategy">FOLLOW</property> <property name="Connect Timeout">10 secs</property> <property name="Read Timeout">10 secs</property> <property name="Url">ldap://msad.local.co:389</property> <property name="Page Size"></property> <property name="Sync Interval">30 mins</property> <property name="Group Membership - Enforce Case Sensitivity">false</property> <property name="User Search Base">OU=DEVELOPER,DC=msad,DC=local,DC=co</property> <property name="User Object Class">user</property> <property name="User Search Scope">ONE_LEVEL</property> <property name="User Search Filter"></property> <property name="User Identity Attribute">cn</property> <property name="User Group Name Attribute"></property> <property name="User Group Name Attribute - Referenced Group Attribute"></property> <property name="Group Search Base"></property> <property name="Group Object Class">group</property> <property name="Group Search Scope">ONE_LEVEL</property> <property name="Group Search Filter"></property> <property name="Group Name Attribute">cn</property> <property name="Group Member Attribute"></property> <property name="Group Member Attribute - Referenced User Attribute"></property> </userGroupProvider> Now unrelated to your Manager credentials issue, I see some other configuration issues: 1. "Page Size" <-- I aways recommend that users set this to 500 (this allows NiFi to request return in multiple pages of 500.  If unset, NiFi expects a single return and for large returns the LDAP server likely has a max for how many returns it will provide in a single page). 2. Within the "User" properties I see: - "User Search Scope"  <-- Are all users being synced actually at same level as configured "Search Base"?  Maybe use "SUBTREE" instead. - "User Group Name Attribute"  <--  With this unset, only users strings from the ldap "cn" attribute will be synced since there is no ldap attribute provided which identifies which groups this user belongs to.  This property tells NiFi which ldap user entry attribute contains the groups that the returned user  is a member of. - "User Group Name Attribute - Referenced Group Attribute". <-- by default this uses ldap "DN" attribute 3. Within the "Group" properties I see: - "Group Search Base" <--  With this set to blank a group based sync is not performed.  So with out "User Group Name Attribute" set in the "User" section and no Group sync being executed, NiFi will not get any group strings returned. - "Group Name Attribute" - Set to CN, which tells NiFi to use the value assigned to the ldap "cn" group entry attribute to as the group identity string in NiFi (but since other properties i mentioned are not set, none will be added to NiFi). - "Group Member Attribute" <-- If you were to determine user/group associations via a group sync this would need to be set so that NiFi knows which ldap group entry attribute defines which user are a member to each group returned by the group sync. So.... Step1 : Address the manager DN and/or Manager password issue blocking the provider from being able to successfully connect to your ldap. Step 2: Fix the user and group sync sections so that NiFi can determine property what user and groups to sync and how to determine the associations between those groups.  This requires knowledge of your LDAP/AD user and group entries in your LDAP/AD.  I can't help you further here without an sample ldapsearch output for a user and a group from your ldap that you are trying to use for authorization in NiFi. https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#ldapusergroupprovider Please be sure to login and click "Accept as Solution" on all responses the assisted you within this query. Thank you, Matt ... View more

@wbivp  When NiFi is configured to use the ldap-user-group-provider, it must be able to successfully execute that provider during startup to generate a list of users and groups within NiFi. The exception points that that provider being unable to execute successfully. The exception in the logs shows: javax.naming.AuthenticationException: [LDAP: error code 49 - 80090308: LdapErr: DSID-0C090439, comment: AcceptSecurityContext error, data 52e, v4563] This points at an authentication issue when trying to communicate with your ldap server. (misconfiguration int the provider, bad Manager or Manager password provided) From the NiFi host can you make run a ldapsearch query against your ldap server using all the same configured values from your provider? Without your authorizers.xml file, it would be difficult for me to point out any other misconfigurations if present. If you found the provided response(s) assisted with your query, please take a moment to login and click on "Accept as Solution" below each solution that helped you. Thank you, Matt ... View more

@patrick_hochste  There is nothing built in that can help with this task. The challenges here: 1. A user who can copy and paste a PG can also modify that PG before ti gets started (like change the PG name). 2. The created PG and all the containing components are assigned new unique UUIDs.  The log will use these new UUIDs when component is started. 3. How do you distinguish between a PG that is copied and one that is being created new? 4. How do you distinguish between a PG that is being started after it was stopped versus starting a brand new copied PG? 5. A copied PG will get a copy of all components and their current configuration. Since you reference setting up an ew DB, does that mean changes to some components configurations? ------ While there is nothing built in to NiFi that can help you trigger based on a copy, paste, and start of a PG, if you could automate the entire process externally via a script, you could build a controlled and structured copy and paste capability.  Keep in mind that anything you can do from directly within the NiFi UI, you can also accomplish via the NiFi Rest API.  This flexibility would allow you to automate the copy of an existing PG, the pasting of that PG, modifying of that copied PG, and any other external scripted requirements. If you found the provided response(s) assisted with your query, please take a moment to login and click on "Accept as Solution" below each solution that helped you. Thank you, Matt ... View more

@wbivp  Within Ranger you can authorizer users and/or groups to the policies you define. The Ranger plugin with the NiFi service runs in the background within NiFi that connects with Ranger to download the latest set of policies. What is provided by Ranger is simply user(s) A, B, C strings and/or group(s) X, Y, Z strings are authorized read and/or write to NiFi Resource Identifier(s).  There is nothin in what is downloaded from Ranger that will tell NiFi as the client what users belong to group(s) X, Y, or Z.   This means that NiFi itself needs to be aware of these associations. This is why in the nifi-user.log you see the following:   o.a.n.w.a.c.AccessDeniedExceptionMapper identity[test.username], groups[] does ...    This log line tells us that NiFi is unaware of any groups the the authenticated user string "test.username" is a member.  If NiFi was aware the "groups[]" in this log line would show a comma separated list of all these group strings. NiFi offers numerous user-group-providers that can be added to the authorizers.xml that allow these associations between user and groups to be set.  Your authorizers.xml file shared contains the "cm-user-group-provider" (only used to assign NiFi node hostnames to a group string "nifi") and the "file-user-group-provider" [1] which gives users a way of manually adding group strings and associating users to that group directly from the NiFi UI. So with your current setup, you would login as your authorized user, go to the NiFi Global Menu, and then select "users".  This will open the NiFi Users UI where you should see your initial admin user which you defined in your file-user-group-provider.  You would need to click on the  icon to add additional users and groups manually.  Adding users and groups here has nothing to do with authentication. You are using this Ui to establish user to group associations.  So I would start by creating a new group.  The Identity string used must match case sensitive the exact group string as seen in Ranger. Then you can start adding your user strings (must match user strings case sensitive as seen in Ranger) As you add users you will be able to select the group(s) you added as those that user should be associated with.   Using above as an example, NiFi would then associate user string "JoeSmith" with group string "admins". To see what other user-group-providers exist within your NiFi version, you should look at the "Admin Guide" found under help within your NiFi's embedded documentation access via the UI. A very commonly used user-group-provider is the "ldap-user-group-provider" [2] which can be used to sync user and groups strings from LDAP/AD and establish the associations between them based on what is in LDAP/AD. [1] https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#fileusergroupprovider [2] https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#ldapusergroupprovider   If you found these responses assisted with your query, please take a moment to login and click on "Accept as Solution" below each post.   Thank you, Matt ... View more

@techNerd    After you fetch the two files, you could route the success relationship twice via two different connections.  One of those connection continues those two files down you r existing dataflow path.  The secondary path could go to a replaceText processor that replaces the entire content of the FlowFile with the filename which is store in a FlowFile attribute named "filename".  Editing the content down this secondary path does not affect the content of the FlowFile in the other dataflow path. Now that you have the filename in the content you can use other processors to store that content with the filename wherever you want for future retrieval.  or you could merge those flowfile in to single FlowFile that would then contain the list of filenames before you store it.  This is just a matter of your specific use case for later consumption of this stored data.   Take a look at the putDistributedMapCache processor as an example. If you found this response addressed your query, please take a moment to login and click on "Accept as Solution" below this post.   Thank you, Matt ... View more

@ToTew    Have you tried using the CompressContent [1] processor? This processor can be configured with "Mode" set to "decompress" and "Compression Format" set to "gzip".   [1] https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.14.0/org.apache.nifi.processors.standard.CompressContent/index.html If you found this response addressed your query, please take a moment to login and click on "Accept as Solution" below this post.   Than you, Matt ... View more