@vg27  1. So i understand that you have created client certificates for your user.  What authority was used to sign these user certificates?  Was this authority added to the NiFi configured truststore? When you open a browser to NiFi's url, NiFi will respond with a WANT for a clientAuth certificate along with a list of trusted authorities from its truststore.  If your certificate loaded in your browser is  not signed by one of those authorities it will not be presented to NiFi.   If no clientAuth certificate is presented, NiFi will move on to another configured method of user authentication.  The fact that you are seeing  the NiFi login UI, tells me the TLS exchange did not result in a clientAuth certificate being presented by yoru browser. With certificate based mutual Auth there is no login required.  3. "nifi.security.user.login.identity.provider=singleUser" is not a valid configuration. I assume you meant "nifi.security.user.login.identity.provider=single-user-provider.   With "Single-user-provider" configured, the only username and password accepted would be for the single user credentials Nifi auto-generated and output to the logs the first time NiFi was started with that provider configured. If you have no intention of using the single-user-provider, just leave "nifi.security.user.login.identity.provider=" unset. 4. you don't need to worry about sticky sessions if you are only using certificate based authentication, since your client certificate would be passed in every request and their are no tokens involved like in login based providers. If you did decide to use a login-provider like LDAP or Kerberos later, sticky sessions would need to be setup first or you may never be able to access the UI.  Once you enter the username and password, the next request goes is to access UI using that token and if the load balancer were to redirect that to a different node, the UI would not load but instead throw and exception about the unknown user. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@Tanya19 @MaxEcueda  The PutIceberg and PutIcebergCDC processors only offer Hadoop or Hive Catalog  Service provider options currently. The only mention of Glue Catalog i could find in an Apache NiFi JIra was the following still open Jira:  https://issues.apache.org/jira/browse/NIFI-11449 It might be a good idea to create an Apache NiFi jira  with as much detail as you can provide around this improvement request for an additional AWS Glue Catalog provider. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@Kiranq  This error shared: 2024-10-17 08:35:19,764 ERROR [Timer-Driven Process Thread-5] o.a.n.c.s.StandardControllerServiceNode StandardControllerServiceNode[service=CSVRecordLookupService[id=a8b84b00-b0ee-31c8-dbda-7e7e9795ba4b], name=CSVRecordLookupService, active=true] Encountering difficulty enabling. (Validation State is INVALID: ['CSV File' is invalid because CSV File is required, 'Lookup Key Column' is invalid because Lookup Key Column is required]). Will continue trying to enable. Indicates that NiFi is trying to enable a NiFi Controller services loaded from the flow.json.gz during startup, but cannot because it's configuration is invalid.  It is complaining about the configuration of the "CSV File" and "Lookup Key Column" properties. Have you tried starting your NiFi with the following setting in your nifi.properties file set to "false": nifi.flowcontroller.autoResumeState=false This will start NiFi and all components on the canvas will not be started during startup. Also if you NiFi is at the point it is trying to enable components on the canvas, Your NiFi is up and running. As far as the screenshot error, have you verified ownership and permissions on that directory path.  Permissions can be an issue if you started the NiFI service as different users at some point in time resulting in some files created on startup having different ownership.    Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@Krish98  Most NiFi Heap memory issues are directly related to dataflow design.  The Apache NiFi documentation for the individual components generally does a good job with reporting "System Resource Considerations".  So the first step would be to review the documentation for the components you are using to see which list "MEMORY" as system resource consideration.  Example: SplitContent 1.27.0 Then sharing your configuration of those components  might help with providing suggestions that may help you. - Split and Merge processor depending on how they are configured can utilize a lot of heap. - Distributed Map cache also resides in HEAP and can contribute to to significant heap usage depending on configuration and the size of what is being written to it. Beyond components: - NiFi loads the entire flow.json.gz (uncompressed it to heap memory).  This includes any NiFi Templates (Deprecated in Apache NiFi 1.x and removed in newer Apache NiFi 2.x version).  Templates should no longer be used.  Any templates created which are listed in the NiFi templates UI should be downloaded so they are stored outside of NiFi and then deleted from NiFi to reduce heap usage. - NiFi FlowFiles - NiFi FlowFlowFiles are what transition between components via connections in your dataflow(s).  A FlowFile consists of two parts.  FlowFile content stored in content claims in the content_repository and FlowFile metadata/attributes held in heap memory and persisted to flowfile_repository.  So if you are creating a lot of FlowFile attributes on your FlowFiles or creating very large FlowFile attributes (like extract content to an attribute), that can result in high heap usage.   A connection does have a default threshold at which time a swap file is created to reduce heap usage.  Swap files are created with 10,000 FlowFiles in each swap file.  The first swap file would not be created until a connection on a specific node reached 20,000 at which point 10,000 would be moved to a swap file and the 10,000 highest priority would remain in heap.  The default "back pressure object threshold" on a connection is 10,000 meaning that with defaults no connection would ever create a swap file.  Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@Axmediko Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future.  Thanks. ... View more

@Abibee04  Please start a new Community question with the details of your query. I am not clear what you mean by "recover the registry", so the more detail you can provide the better in that new community question by sharing what step you performed and what you are looking to accomplish. Thank you, Matt ... View more

@varungupta  Tell me a bit more about your ListFile processor configuration.   Is the input directory a shared mount across all your nodes or does each node have a unique set of files in the input directory? Is ListFile configured for execution of "All nodes" or "Primary node"? How many files are being listed when ListFile is executing (Is it less then 10,000)? How often is ListFile scheduled to run? Is List File traversing sub-directories for files? I assume you are extracting the sequenceNo from the filename. As far as Enforce Order goes: How are you handling the various relationships (routing via connection to where)? Are you seeing FlowFiles routed to "overlook" relationship? Matt ... View more

@MattWho : Sorry that I missed replying to your questions early , I have made sure to fix the authorizers.xml and now I am able to access the registry UI .    Thanks for all you detailed resposnes and suggestions. ... View more