@Umakanth  From your shared log lines we can see two things: 1. "LOG 1" shows "StandardFlowFileRecord[uuid=345d9b6d-e9f7-4dd8-ad9a-a9d66fdfd902" and "LOG 2" shows "Successfully sent [StandardFlowFileRecord[uuid=f74eb941-a233-4f9e-86ff-07723940f012". This tells us these "RandomFile1154.txt" are two different FlowFiles. So does not look like RPG sent the same FlowFile twice, but rather sent two FlowFiles with each referencing the same content.  I am not sure how you have your LogAttribute processor configured, but you should look for the log output produced by these two uuids to learn more about these two FlowFiles.  I suspect from your comments you will only find one of these passed through your LogAttribute processor. 2. We can see from both logs that the above two FlowFiles actually point at the exact same content in the content_repository:   "LOG 1" --> claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1599937413340-1, container=default, section=1], offset=1073154, length=237],offset=0,name=RandomFile1154.txt,size=237] "LOG 2" --> claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1599937413340-1, container=default, section=1], offset=1109014, length=237],offset=0,name=RandomFile1154.txt,size=237] This typically happens when a FlowFile becomes cloned somewhere in your dataflow.  For example: when a relationship from a processor is defined twice. Since you saw that GetFile only ingested file once, that rules out GetFile as the source of this duplication.  But had it been GetFile, you would have not seen identical claim information.  LogAttribute only has a single "success" relationship, so if you had drawn two connections with "Success" relationship defined in both, you would have seen duplicates of every ingested content.  So this seems unlikely as well.  Next you have your PutFile processor.  This processor has both "success" and "failure" relationships.  I suspect the "success" relationship is assigned to the connection going to your Remote Process Group" and the "failure" relationship assigned to a connection that loops back on the PutFile itself(?).  Now if you had accidentally drawn the "failure" connection twice (one may be stack on top of the other), anytime a FlowFile failed in the putFile it would have been routed to one failure connection and cloned to other failure connection.  Then on time they both processed successfully by putFile and you end up with the original and clone sent to your RPG. Hope this helps, Matt ... View more

@DivyaKaki    The exception implies that the complete trust chain does not exist to facilitate a successful mutual TLS handshake between this NiFI and the target NiFi-Registry.   NiFi uses the keystore and truststore configured in its nifi.properties and NiFi-Registry uses the keystore and truststore configured in its nifi-registry.properties files. Openssl can be used to public certificates for the complete trust chain: openssl s_client -connect <nifi-registry-hostname>:<port> -showcerts   openssl s_client -connect <nifi-hostname>:<port> -showcerts for each public cert you will see: -----BEGIN CERTIFICATE----- MIIESjCCAzKgAwIBAgINAeO0mqGNiqmBJWlQuDANBgkqhkiG9w0BAQsFADBMMSAw HgYDVQQLExdHbG9iYWxTaWduIFJvb3QgQ0EgLSBSMjETMBEGA1UEChMKR2xvYmFs U2lnbjETMBEGA1UEAxMKR2xvYmFsU2lnbjAeFw0xNzA2MTUwMDAwNDJaFw0yMTEy MTUwMDAwNDJaMEIxCzAJBgNVBAYTAlVTMR4wHAYDVQQKExVHb29nbGUgVHJ1c3Qg U2VydmljZXMxEzARBgNVBAMTCkdUUyBDQSAxTzEwggEiMA0GCSqGSIb3DQEBAQUA A4IBDwAwggEKAoIBAQDQGM9F1IvN05zkQO9+tN1pIRvJzzyOTHW5DzEZhD2ePCnv UA0Qk28FgICfKqC9EksC4T2fWBYk/jCfC3R3VZMdS/dN4ZKCEPZRrAzDsiKUDzRr mBBJ5wudgzndIMYcLe/RGGFl5yODIKgjEv/SJH/UL+dEaltN11BmsK+eQmMF++Ac xGNhr59qM/9il71I2dN8FGfcddwuaej4bXhp0LcQBbjxMcI7JP0aM3T4I+DsaxmK FsbjzaTNC9uzpFlgOIg7rR25xoynUxv8vNmkq7zdPGHXkxWY7oG9j+JkRyBABk7X rJfoucBZEqFJJSPk7XA0LKW0Y3z5oz2D0c1tJKwHAgMBAAGjggEzMIIBLzAOBgNV HQ8BAf8EBAMCAYYwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMBIGA1Ud EwEB/wQIMAYBAf8CAQAwHQYDVR0OBBYEFJjR+G4Q68+b7GCfGJAboOt9Cf0rMB8G A1UdIwQYMBaAFJviB1dnHB7AagbeWbSaLd/cGYYuMDUGCCsGAQUFBwEBBCkwJzAl BggrBgEFBQcwAYYZaHR0cDovL29jc3AucGtpLmdvb2cvZ3NyMjAyBgNVHR8EKzAp MCegJaAjhiFodHRwOi8vY3JsLnBraS5nb29nL2dzcjIvZ3NyMi5jcmwwPwYDVR0g BDgwNjA0BgZngQwBAgIwKjAoBggrBgEFBQcCARYcaHR0cHM6Ly9wa2kuZ29vZy9y ZXBvc2l0b3J5LzANBgkqhkiG9w0BAQsFAAOCAQEAGoA+Nnn78y6pRjd9XlQWNa7H TgiZ/r3RNGkmUmYHPQq6Scti9PEajvwRT2iWTHQr02fesqOqBY2ETUwgZQ+lltoN FvhsO9tvBCOIazpswWC9aJ9xju4tWDQH8NVU6YZZ/XteDSGU9YzJqPjY8q3MDxrz mqepBCf5o8mw/wJ4a2G6xzUr6Fb6T8McDO22PLRL6u3M4Tzs3A2M1j6bykJYi8wW IRdAvKLWZu/axBVbzYmqmwkm5zLSDW5nIAJbELCQCZwMH56t2Dvqofxs6BBcCFIZ USpxu6x6td0V7SvJCCosirSmIatj/9dSSVDQibet8q/7UK4v4ZUN80atnZz1yg== -----END CERTIFICATE----- Above is just example public cert from openssl command against google.com:443 You will need to make sure that every certificate in the chain when run agains NiFi UI is added to the truststore on NiFi-Registry and vice versa. You'll need to restart NiFi and NiFi-Registry before changes to your keystore or truststore files will be read in. Hope this helps, Matt ... View more

@bhara    While ldap/AD queries are generally case insensitive by default, NiFi is not case insensitive. So user "bbhimava" and user "Bbhimava" would be treated as two different users.   Within the nifi.properties file you can utilize identity and group mapping patterns to manipulate the case of the returned user and groups strings. https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#identity-mapping-properties Note that mapping patterns are evaluate in alphanumeric order. First pattern that matches is applied. so make sure less specific patterns like "^(.*)$" are lats to be evaluated. For example: nifi.security.identity.mapping.pattern.dn=<java regex> Above would be evaluated before: nifi.security.identity.mapping.pattern.username=<java regex>   When/If a match is found, the corresponding value is applied and transform performed: nifi.security.identity.mapping.value.dn=$1 nifi.security.identity.mapping.transform.dn=<NONE, LOWER, or UPPER>     Looking at your ldap-user-group-provider configuration, I see the following needed changes: 1. I recommend user set the "Page Size" property. If the number of results exceeds the ldap max default page size, not all results may be returned to NiFi. BY setting a page size, NiFi will asks for results in multiple pages instead of one response with all results.  Generally, defaults are either 500 or 1000, so setting a "Page Size" of 500 is safe. 2. The "User Search Scope" and "Group Search Scope" properties should be set to "SUBTREE" and not "sub". 3. The "User Group Name Attribute" property does not support a comma separated list of attributes. Suggest just setting it to "memberOf".   Without sample output from your ldap/AD for user bbhimava and one of the groups you are trying to sync based upon, it would be impossible for me to validate any of your other settings. The log line shared below:     o.a.n.w.a.c.AccessDeniedExceptionMapper identity[bbhimava], groups[none] does not have permission to access the requested resource. No applicable policies could be found. Returning Forbidden response.     indicates that when NiFi looked for user "bbhimava" was looked for by the NiFi authorizer, no associated groups were returned.  Which means Ranger policies could only be queried for policies assigned directly to "bbhimava". Adding the following line to your  NiFi logback.xml file will give you debug output in your nifi-app.log when the ldap-user-group-provider executes to show you exactly what users and groups were returned based upon your settings and the resulting user/group associates that were discovered. <logger name="org.apache.nifi.ldap.tenants.LdapUserGroupProvider" level="DEBUG"/> Hope this helps, Matt ... View more

@JohnA    The latest versions of NiFi-Registry introduced public buckets.  When accessing the NiFi-Registry's https URL without presenting either a client certificate or responding to a spnego auth challenge (provide spnego properties are configured), the user will access to the public buckets as the anonymous user.  If there are no anonymous buckets the user will simply see the NiFi-Registry Ui with nothing displayed they can use.  To access the non public buckets and any other capability requiring authorization, the user/client would need to authenticate.  In the case where a login provider is setup, the login option will be available in upper right corner which a user can click on to "login". When it comes to NiFi interacting with NiFi-Registry a successful mutual TLS handshake must be successful; otherwise, the connection will happen as anonymous since no user authentication occurred.  You should not be authorizing the anonymous user, but instead fixing the mutual TLS handshake between your NiFi and NIFi-Registry.  The keystore and truststore configured in both the nifi.properties and nifi-registry.properties files are what are used in facilitating that successful handshake.  Since NiFi is acting as the client in this handshake, its PrivateKeyEntry in its keystore must support clientAuth EKU and its truststore must be contain the complete trust chain for the server certificate being present by NiFi-Registry.  The truststore used in NiFi-Registry must contain the complete trust chain for the certificate being presented by the client (NiFi).  The NiFi certificate authenticated string must then be authorized to read ll buckets and proxy within NiFi-Registry.  Additionally any user initiating version control actions from within NiFi must also be authorized within NiFi-Registry for any buckets they will need access to. Hope this helps, Matt ... View more

@sgk    The error you are seeing has nothing to do with authorization at this point.  It is throwing an error during authentication of your user. So your focus at this point is on your ldap-provider configuration since it is handling the authentication of your user.   "The Supplied Username or Password are not valid" indicates that the LDAP search resulted in no returns or the password used was wrong. Observations: 1. Are you using ldap or Active Directory (AD). I see you have set "User Search Filter" to "sAMAccountName={0}".  sAMAccountName is more commonly seen in AD and not LDAP.  Did you try using the ldapsearch command from a terminal window on your NiFI server to make sure you can return a listing for your user using this search filter? ldapsearch -x -H ldap://<ldap-hostname/IP>:<ldap-port> -D "<Manager DN>" -w "<Manager password>" -b "<user search base>" "sAMAccountName=<username>" 2. Not that this has anything to do with successful authentication, but I see you have set "Identity Strategy" to "USE_DN" which then uses the users full DN from ldap to identify that user during authorization actions following successful authorization. If you set this to "USE_USERNAME", the user string type at login will be used.   3. Also has nothing to do with authentication, but I see you are using "CN=localhost, OU=NiFi" as your "node identity 1" value.  Using localhost in your node certificates is not advisable. This should be set to unique value.  Also keep in mind that the keystore used by NiFi must meet the following minimum requirements: - Contain only 1 "PrivateKeyEntry" - The "PrivateKeyEntry" must support both clientAuth and serverAuth ExtendedKeyUsage (EKU). - The "PrivateKeyEntry" must contain at least 1 SubjectAlternativeName (SAN) that matches the hostname of the server on which the certificate is being used. Hope this information helps you progress with your authentication and then authorization setup in NiFi. Matt ... View more

@Ben37    The GetHDFSFileInfo processor will produce one output FlowFile containing a complete listing of all files/directories found based upon the configured "Full path" and "Recurse Subdirectories" properties settings when the property "Group Results" is set to "All". Since you want a single FlowFile for each object listed from HDFS, you will want to set the "Group Results" property to "None".  You should then see a separate FlowFile produced fro each object found. Then in your FetchHDFS processor you would need to set the property "HDFS Filename" to "${hdfs.path}/${hdfs.objectName}". You may also find that you need to insert a RouteOnAttribute processor between your GetHDFSFileInfo and FetchHDFS processors to route out any FlowFIles produced by the GetHDFSFileInfo processor that are for directory objects only (not a file).  You simply add a dynamic property to route any FlowFile produced from GetHDFSFileInfo processor that has the attribute "hdfs.type" set to "file" on to the fetchHDFSprocessor and send all other FlowFiles to the unmatched relationship which you can just auto-terminate. Other things to consider: 1. Keep in mind that the GetHDFSFileInfo processor does not maintain any state, so every time it executes it will list all files/directories from the target regardless of whether they were listed before or not.  The ListHDFS processor uses state. 2. If you are running your dataflow in a NIFi multi-node cluster, every node in your cluster will be performing the same listing (which may not be what you want).  If you only want the list of target files/directories listed by one node, you should configure the GetHDFSFileInfo processor to "execute" on "Primary node" only (configured from processors "scheduling" tab).  You can use load balancing configuration on the connection out of the GetHDFSFileInfo processor to redistribute the produced FlowFiles across all nodes in your cluster before they are processed by the FetchHDFS processor. Hope this helps, Matt ... View more

@venkii    What version of NiFi are you running? There are a set of bugs identified here that are likely related to the issue you described: https://issues.apache.org/jira/browse/NIFI-5948 https://issues.apache.org/jira/browse/NIFI-6020 https://issues.apache.org/jira/browse/NIFI-6027   The good news is that these have all since been resolved.   I would recommend upgrading your NiFi to a version 1.10 or newer. Or HDF 3.4.1.1 or newer Or CFM 1.0.1 or newer   As a workaround, you could shutdown you NiFi and search the users.xml and authorizations.xml file for uuid "938eb61e-bbc4-383a-8475-aee80541b5a5" and remove all references to it.  You would then need to make the exact same changes to the users.xml and authorizations.xml file on every other node in your NiFi cluster or copy the corrected files from one node to the other nodes.   Hope this helps, Matt ... View more