@LKB  You would get better traction/feedback if you start your own community question.  Your query is not very related to issue in this post.    As far as the one question related to this post about encrypted manager password,  @mks27 simply masked it by using "***" in his post.  NiFi does not replace actual password with * when encrypting sensitive passwords. The NiFi Encrypt-Config Toolkit  can be used to encrypt passwords used in various NiFi configuration files: https://nifi.apache.org/docs/nifi-docs/html/toolkit-guide.html#encrypt_config_tool Thank you, Matt ... View more

@viskumar  A Cloudera License is required to access Cloudera distributions located on archive.cloudera.com You can download open source versions of mahout from Apache: https://mahout.apache.org/ Hope this helps you, Matt ... View more

@manishg  I am not clear, can you share the processor type where the expected data is not present? The View Status History "Record Processed" will only appear on some component "Record" type processors (example: PartitionRecord processor).  It would not exist as on processors that don't process "records".   If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@manishg  Copying a file should have resulted in a new timestamp making it newer than the last file you previously listed.  However, moving a file typically does not change timestamps on the file.  My guess here is you moved a file into the directory instead of copy? Matt ... View more

@mks27  I am reading through this post and see multiple conflicting output shared from you which imply config changes were applied between updates added to this post. First you need to understand that NiFi authentication and NiFi authorization are two totally separate processes.  After successful authentication is successful the user identity string is evaluated against any configured Identity mapping patterns configured in the nifi.properties file.  IF a java regex mapping pattern matches against the user identity string returned during authentication, the configured associated identity mapping value is applied.   At this point the user identity string is passed off to the configured authorizer configured in NiFi to verify that the user is authorized for the request endpoint being accessed.  The authorizer must be aware of all user identity strings and those user must be authorized to the resource before a user will be authorized.  It is IMPORTANT to understand that NiFi is case sensitive (Identity bob and BOB would be treated as two different users). Your initial query you stated that the NiFi UI shows successful authentication, but indicates that authorization was then not successful.  We know this because it returned a user identity (determined during authentication) and then reported that user was not known to your NiFi during authorization verification. Unknown user with identity 'cn=Mohit Kumar,ou=FM-Users,ou=Managed services,dc=CORP,dc=SA,dc=ZAIN,dc=COM'. Contact the system administrator. In your same post you shared the DN from your ldapsearch response as: CN=Mohit Kumar,OU=FM-Users,OU=Managed services,DC=CORP,DC=SA,DC=ZAIN,DC=COM As we can see these do not match.  Regardless of above, what NiFi received in response to your authentication request from your ldap is what is displayed in the NiFiUI. Now, in a later post you shared the nifi-user.log output below: 023-05-23 02:53:37,220 INFO [NiFi Web Server-21] o.a.n.w.a.c.AccessDeniedExceptionMapper identity[mohit.kumar], groups[] does not have permission to access the requested resource. Unknown user with identity 'mohit.kumar'. Returning Forbidden response. This log line implies that a user was successfully authenticated with a user identity of "mohit.kumar".  This is not same user as shared in the initial post.  My guess here us that changed your ldap-provider from using: <property name="Identity Strategy">USE_DN</property> to: <property name="Identity Strategy">USE_USERNAME</property> The "USE_USERNAME" is more commonly used.  Upon successful authentication, the username entered at the NiFi login prompt is used as the user identity rather than the DN returned by ldap.   Or you setup some Identity.mapping.pattern that matched in your full DN, extracted just the CN and set it to all lowercase?   NiFi authorization is handled by the authorizers.xml NiFi configuration file. In your authorizers.xml you have the "Managed authorizer" which has a configured dependency on the "File-Access-Policy-Provider" which itself has a configured dependency on "File-User-Group-Provider". The File-User-Group-Provider is responsible for building the users.xml file and populating it with a few initial entries.  This provider will ONLY generate a users.xml file if it does NOT already exist.  So any edits to this configuration after the users.xml file already exists will not be reflected in this file.  I see you have configured this provider to create the following user identity: <property name="Initial User Identity 1">CN=Mohit Kumar,OU=FM-Users,OU=Managed services,DC=CORP,DC=SA,DC=ZAIN,DC=COM</property> This Identity matches neither Identity mentioned earlier that resulted from successful authentication (remember that NiFi is case sensitive). I would recommend changing this to the following and deleting the users.xml so it gets recreated: <property name="Initial User Identity 1">mohit.kumar</property> Make sure you are also use "USE_USERNAME" in your ldap-provider. The File-Access-Policy-Provider is responsible for building the authorizations.xml file only if it does not already exist.   Within this provider you defined who your initial admin user identity should be.  When building the authorizations.xml file for the first time, this initial admin user identity will be granted the authorization needed to act as an administrator. <property name="Initial Admin Identity">CN=Mohit Kumar,OU=FM-Users,OU=Managed services,DC=CORP,DC=SA,DC=ZAIN,DC=COM</property> This should be changed to below and current authorizations.xml (not authorizers.xml) must be deleted so it can be rebuild based on new initial admin: <property name="Initial Admin Identity">mohit.kumar</property>   Now restart your NiFi and login using "mohit.kumar" in the NiFi login window.  I should note that I am assuming here that "mohit.kumar" is your users sAMAccountName value in your LDAP entry. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@MmSs  NiFi is data agnostic.  To NiFi, the content of a FlowFile just bits. To remain data agnostic, NiFi uses what NiFi calls a "FlowFile".   A FlowFile consists of two parts, FlowFile Attributes/Metadata (persisted in FlowFile repository and held in JVM heap memory) and FlowFile content (stored in content claims within content repository).  This way NiFi core does not need to care or know anything about the format of the data/content.  It becomes the responsibility of am individual processor component that needs to read or manipulate the content to understand the bits of content.  The NiFi FlowFile metadata simply records in which content claim the bits exist and at what offset within the claim the content starts and number if bits that follow.   As a far as directory paths go, these become just additional attributes on a FlowFile and have no bearing on NiFi's persistent storage of the FlowFiles content to the content repository. As far as the unpackContent goes, the processor will process both zip1 and zip2 separately.  Unpacked content from zip one is written to a new FlowFile and same hold true for zip2.  So if you stop the processor immediately after your UnpackContent processor and send your zip1 and zip2 FlowFiles through, you can list the content on the outbound relationship to inspect them before further processing.  You'll be able to view the content and the metadata for each output FlowFile.  NiFi does not care if there are multiple FlowFiles with the same filename as NiFi tracks them with unique UUID within NiFi.  What you describe as zip1 content (already queued in inbound connection to PutS3Object being corrupted if zip2 is then extracted) is not possible.  Run both zip 1 and zip2 through your dataflow with putS3Object stopped and inspect the queued FlowFiles as they exist queued before putS3Object is started.  Are queued files on same node in your NiFi cluster?  Is your putS3Object using "${filename}" as the object key?  What happens if you use "{filename}-${uuid}" instead? My guess is issue is in your putS3Object configuration leading to corruption on write to S3. So your issue seems more likely to be a flow design issue then a processor of NiFi FlowFile handling issue.  Sharing all the processors you are using in your dataflow and their configuration may help in pinpointing your design issue. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more