@AyanF  I would hope that if you are securing a processor such as HandleHTTPRequest processor to control access to this endpoint, that you have also secured your NiFi instance/cluster as well? SSL/TLS is a very deep subject which is a bit much to discuss end-to-end here, but i will give a 10,000 foot view to help get you started. If so, then somewhere along the line a keystore and truststore were generated to secure your NiFi. Keystore files (keystore or truststore) are not something unique to NiFi nor any NiFi processor that uses them. Keystore and truststores are used to facilitate TLS handshakes and establish a secure and encrypted connection between a client and a server.  A keystore used by NiFi would contain a single "PrivateKeyEntry". This privateKeyEntry used by HandleHHTPRequest processor must have an Extended Key Usage (EKU) that support at least serverAuth (for securing NiFi itself it would need both serverAuth and clientAuth EKUs).  It must also contain a SubjectAlternativeName (SAN) that contains the hostname of server on which your NiFi is running.  Commonly the same keystore used to secure your NiFi instance is also used in the SSLContext service to secure components used in your dataflows but not a requirement. A Truststore contains one too many "TrustedCertEntry"s.  The trustedCertEntries are the public keys for self-signed certificates, intermediate, and root Certificate Authorities (CA). TLS works the same accessing a secured NiFi or secured NiFi processor as it does accessing any other https site.  The server side of the connection decides if the TLS handshake should be 1-way or mutual.  In a 1 way TLS connection, the secured server provides its certificate to the client.  The client verify that it trusts that server certificate. The server does not require the client to identify itself via a client certificate.  This is how https://www.google.com works.The Server has the option to require/need or want a client certificate.  The HandleHTTPRequest processor which would be server side of the TLS connection can be configured for one of those options.   In a TLS client and server exchange the certificate DN is always used to identify the server to the client and client to the server.  The Client certificate presented must be trusted by the server side as well.  An truststore is included with every Java distribution.  The file is named "cacerts" and it contains a bunch of trustedCertEntrys for well know CAs. You can view the contents of a keystore like the cacerts file using java's keytool command:   keytool -v -list -keystore <path to keystore file>/<keystore filename>   Naturally if you create your own certificate it would be self signed and the cacerts file would need to have the public cert/key for your self signed private cert/key added to it. Here is a example process for creating a keystore: https://docs.oracle.com/cd/E19509-01/820-3503/ggfen/index.html There are also services (some free) you can use to generate a signed certificate.  Those services will also provide you with the public certs for your truststore. https://www.tinycert.org/ But simply having a keystore and truststore on the NiFi (server) side is not enough to establish a secured connection.   The client that is connecting to you NiFi handleHTTPRequest endpoint must also have a truststore.  That truststore must contain the public cert/key if using a self signed server side cert or the complete trustchain for CAs that signed your server certificate.  What is a trustchain?  When you get a server cert signed, it may be signed by an intermediate CA or a root CA.  An intermediate CA is a CA that itself has been signed by yet another CA. A root CA is signed by itself (Owner and issuer are same DistinquishedName (DN)).  The complete trustschain would involve having every public cert from intermediate CA to the rootCA.  Let's say your server cert is signed by intCA1 and intCA1 is signed by rootCA.  RootCA is signed by RootCA.  So the complete trustschain would require having the public cert/key for both intCA1 and rootCA in the truststore. Now if you choose to have your HandleHttpRequest processor to "need authentication", the client must have a certifcate the truststore configured in your SSLContextService is capable of trusting (meaning it needs to complete trustchain in that truststore for that client cert/key.  Keep in mind that any client certificate that is trusted by that truststore will be able to interface with that endpoint. I understand this is a lot, but it only scratches the surface.  Hopefully it is enough to get you were you need to be to secure your NiFi and NiFi components. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt ... View more

@Sanchari  NiFi FlowFiles reside in connection between NiFi component processors.  When a processor gets a thread to execute, it takes the highest priority FlowFile from an inbound connection queue and executes the processor code utilizing that FlowFiles metadata/attributes and content (if processor needs content).  The FlowFile is not transferred to a processors outbound connection(s) until execution is complete. When NiFi is shutdown gracefully (meaning a user has initiated a shutdown), NiFi stops scheduling future component execution.  NiFi then gives existing executing threads a grace period to complete their thread execution.  At the end of that grace period, any still running threads are killed with the JVM.  Since FlowFiles do not transfer to an outbound connection until code execution has completed, and FlowFile that was owned by a thread at the time the thread was killed still remains on the inbound connection.  When NiFi is started again and the dataflows started, the file processing will start over when the processor executes again and executes against the highest priority FlowFile in the connection. Above being said, NiFi will favor data duplication over data loss every time.  It is possible in a small window of time that processor executes and part of that execution is let's say to write a file to a remote server.  NiFi may for example ack the completion of that transfer to the remote system and NiFi JVM was killed before internally it received ack back from target server. So the FlowFile would end up being processed again resulting potentially data duplication on the target server.  These are rare race conditions, but possible. A restart is nothing more than a standard shutdown followed by a start.  The same behavior exists in the shutdown process as described above when a restart is performed. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt ... View more

@EuGras  You have a FlowFile queued somewhere within your dataflow with UUID=  6ce9e262-b20b-4372-a3b9-43c2c00e8caa The connection is trying to read the content for that FlowFile from a content claim found in the content-repository in order to load balance data across nodes in the cluster here: id=1663256223724-231072817, container=default, section=49 <path to>/content_repository/49/1663256223724-231072817 The FlowFile metadata/attributes has recorded that this content should be 2203 bytes in length; however, tis file is only 1130 bytes in size.  So it appears when you had disk issue it resulted in data corruption.  You could use NiFi data provenance to locate this FlowFile by UUID or filename (04190e1f-fdca-4352-a796-6b6c9ce41baa) to determine which connection contains it.   On that connection you could disable load-balance connection configuration, add a routeOnAttribute processor to filter out this one bad FlowFile and auto-terminate it once it is routed out of other FlowFiles that may have been queued in that same connection. This is not to say that you may have other corruption caused by your disk issues besides this one FlowFile.   If you do not care about the data on the nodes that had the disk issues, as another option, you could shutdown that one node, purge the contents of the flowfile_repository and content_repository.  This will effectively delete all flowfiles queued in connections on that one node.  Then restart the NiFi node.  It will construct new content and flowfile repository on startup. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt ... View more

@double_z  NiFi does not allow users to create locally managed user accounts (meaning creating a username and password directly in NiFi. NOTE: The latest versions of NiFi provide a single user login provider just so that NiFi can by default be launched securely.  This single user would have complete access to do everything and you cannot create additional users.  Ideally while this provides some security, it is not a proper recommended deployment method. The good news here is it sounds like you have successfully setup a ldap-provider in your login-identity.providers.xml.   This provider provides a method by which a user can be authenticated via LDAP. Authentication via ldap does not control authorization which is handled within NiFi via the authorizers.xml. When you login to NiFi, the resulting user identity string (case sensitive) resulting from your login is evaluated against any identity mapping patterns you may have configured in your nifi.properties file. After that the resulting identity string (if pattern match was found) or unmodified identity string from login provider is passed to the NiFi authorization process. While NiFi loads the providers from the authorizers.xml from the top down, it is easer for a user to read it from bottom up. - Your are using the "managed-authorizer" which is calling the "file-access-policy-provider" - The "File-access-policy-provider" is responsible for authorizations.xml file and seeding it with an initial set of authorization policies needed for your NiFi nodes (multi-node nifi cluster) and an initial admin user.  You need an initial admin so that the admin user can set additional authorization from with in the NiFi UI. - In your "file-access-policy-provider" you have told the provider to create the authorization policies for a user identity string "freeipa". So the file-access-policy-provider" must first check to see if that user is known to this NiFi.  For that is is configured to use the "composite-configurable-user-group-provider". - The "composite-configurable-user-group-provider" is then configured to get users and associated groups from the "file-user-group-provider" and "ldap-user-group-provider". You can NOT have to user-group-provider return the exact same user identity string. - Your "ldap-user-group-provider" has ben configured to sync user and group identity strings from your LDAP. One of those user being returned is "freeipa". - Your "file-user-group-provider" has been configured to create a local user identity that has the same user identity string of "freeipa".  So now you have two user-group-provider returning the same user identity string, so NiFi has not idea which is correct to use and throws the exception you see about two providers providing same user identity. - What you have shared above also shows the "file-user-group-provider" twice. You can't have same provider defined twice in this file. - Steps to move forward. 1. The file-user-group-provider and the file-access-policy-provider will only create the users.xml file and authorizations.xml file if they do NOT already exist.  So if these two files exist, delete them (authorizers.xml and authorizations.xml are two different files, make sure you delete the correct one) 2. Make sure you ldap-provider is configured to USE_USERNAME and not USE_DN if not already set this way. 3. Unset the "Initial User Identity 1" in the file-user-group-provider.  We don't want this provider creating the freeipa user in the users.xml since your ldap-user-group-provider will be providing this user identity. 4. Leave the initial admin "freeipa" set in the file-access-policy-provider. 5. start your NiFi, it will create a new users.xml and authorizations.xml during startup. At login UI, provide your ldap "freeipa" username and password. Once in the UI, your freeipa user will have all the authorization policies needed to act as and admin.  This does not mean this user has all authorizations, but does have ability to grant additional authorizations to itself or other users.   NiFi global menu (upper right corner) --> users (will allow you to see all users and groups long with their associations to one another synced from LDAP).  It will also show any local users identities you may define (locally defined identifies will show an edit and delete icon next to them).  local user and group identities are only used to set authorizations, they are not able to be used to authenticate in to NiFi. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt ... View more