@vg27  If you have a support contract with Cloudera, you could open a support case where someone could connect directly with you and assist you through your many issues. ------ 1. As i have shared before, the Single-User providers are not designed with the intent of use in a NiFi clustered environment.  They should only be used for standalone NiFi evaluation purposes.  Once you start to get in to the more involved cluster based deployments, you need to use different providers for authentication and authorization.  When using the single-user-provider for authentication, each node can create different credentials which will not work in a cluster environment.    For login based authentication, you should be using LDAP/AD (ldap-provider) or Kerberos (Kerberos-provider).  For authorization, you should be using the managed authorizer.  ------ 2. Are you still using your own generated keystore and truststore with your own created private and public certificates?  Using the NiFi auto-generated keystore and truststore will also not support clustering well as each node will not have a common certificate authority.  ----- 3. The "org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss" exception is an issue with with Zookeeper (ZK) Quorum.  This error can happen if both you nodes are not fully up at time of exception and may also happen because you do not have proper quorum with your ZK.  Quorum consists of and odd number of ZK hosts with min 3.  Strongly encourage the use of an external ZK since anytime one of your nodes goes down, you'll lose access to both nodes. ----- 4. You are using an external https Load Balancer (LB) which means that sticky sessions (session affinity) must be setup since the user token issues when you login is only valid for use with the node that issued it.  So if your LB directs you to node 1 that presents you with login UI, you enter credentials obtaining a user token from node 1, and your LB then redirects to node 2 to load UI, it will fail authentication on node 2 because the request includes the token only good for node 1.  ----- 5. I see you are using a mix of hostnames and IP addresses in your NiFi configurations, so make sure that the node certificates include both as SAN entries to avoid issues. -----     Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@vg27  1. So i understand that you have created client certificates for your user.  What authority was used to sign these user certificates?  Was this authority added to the NiFi configured truststore? When you open a browser to NiFi's url, NiFi will respond with a WANT for a clientAuth certificate along with a list of trusted authorities from its truststore.  If your certificate loaded in your browser is  not signed by one of those authorities it will not be presented to NiFi.   If no clientAuth certificate is presented, NiFi will move on to another configured method of user authentication.  The fact that you are seeing  the NiFi login UI, tells me the TLS exchange did not result in a clientAuth certificate being presented by yoru browser. With certificate based mutual Auth there is no login required.  3. "nifi.security.user.login.identity.provider=singleUser" is not a valid configuration. I assume you meant "nifi.security.user.login.identity.provider=single-user-provider.   With "Single-user-provider" configured, the only username and password accepted would be for the single user credentials Nifi auto-generated and output to the logs the first time NiFi was started with that provider configured. If you have no intention of using the single-user-provider, just leave "nifi.security.user.login.identity.provider=" unset. 4. you don't need to worry about sticky sessions if you are only using certificate based authentication, since your client certificate would be passed in every request and their are no tokens involved like in login based providers. If you did decide to use a login-provider like LDAP or Kerberos later, sticky sessions would need to be setup first or you may never be able to access the UI.  Once you enter the username and password, the next request goes is to access UI using that token and if the load balancer were to redirect that to a different node, the UI would not load but instead throw and exception about the unknown user. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@Tanya19 @MaxEcueda  The PutIceberg and PutIcebergCDC processors only offer Hadoop or Hive Catalog  Service provider options currently. The only mention of Glue Catalog i could find in an Apache NiFi JIra was the following still open Jira:  https://issues.apache.org/jira/browse/NIFI-11449 It might be a good idea to create an Apache NiFi jira  with as much detail as you can provide around this improvement request for an additional AWS Glue Catalog provider. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@jame1997  1. Everything that is exists on the canvas resides in heap and is persisted to the flow.json.gz file written to the NiFi conf directory (default).  With every change made within the NiFi UI, the current flow.json.gz if moved to an archive directory and new flow.json.gz is generated.  All Nodes in a NiFi cluster use the same flow.json.gz.  You don;t need to stop NiFi to make a copy of the flow.json.gz file.  The NiFi flow.json.gz will contain encrypted values for all sensitive properties entered in any component on the NiFi UI canvas. In order for a NiFi instance to load a flow.json.gz, it must be cnfigured with the same "nifi.senstive.props.key" password used by the NiFi where the flow.json originated.  So always a good idea to backup your NiFi's conf directory (especially the nifi.properties file).  Keep in mind that all nodes in the NiFi Cluster must have same sensitive props key password configured, so unless you lose your entire NiFi cluster, you can get the flow.json.gz an sensitive.props.key form any nodes still accessible. 2. I am not crystal clear on what "backup all the processes" means.  Are you referring to all the dataflows built on the NiFi canvas? Some components (processors, controller services, reporting tasks) may have dependencies on other things like local/cluster state, local files, etc.  So those items need to be taken into consideration with any backup planning.  Each node in a NiFi cluster has a local state directory which hold state for components that use local state (listFile for example).  For other components, cluster state is used (cluster state is written to Zookeeper and Zookeeper quorum maintains protection of that information).  State is always changing as your dataflow is running, so it is not something to easily backup specifically for local state that has not redundancy across NiFi nodes.    Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@Kiranq  This error shared: 2024-10-17 08:35:19,764 ERROR [Timer-Driven Process Thread-5] o.a.n.c.s.StandardControllerServiceNode StandardControllerServiceNode[service=CSVRecordLookupService[id=a8b84b00-b0ee-31c8-dbda-7e7e9795ba4b], name=CSVRecordLookupService, active=true] Encountering difficulty enabling. (Validation State is INVALID: ['CSV File' is invalid because CSV File is required, 'Lookup Key Column' is invalid because Lookup Key Column is required]). Will continue trying to enable. Indicates that NiFi is trying to enable a NiFi Controller services loaded from the flow.json.gz during startup, but cannot because it's configuration is invalid.  It is complaining about the configuration of the "CSV File" and "Lookup Key Column" properties. Have you tried starting your NiFi with the following setting in your nifi.properties file set to "false": nifi.flowcontroller.autoResumeState=false This will start NiFi and all components on the canvas will not be started during startup. Also if you NiFi is at the point it is trying to enable components on the canvas, Your NiFi is up and running. As far as the screenshot error, have you verified ownership and permissions on that directory path.  Permissions can be an issue if you started the NiFI service as different users at some point in time resulting in some files created on startup having different ownership.    Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@HenriqueAX  The NiFi nodes themselves require a few authorizations within NiFi registry in order to start version control of a process group in NiFi. The NiFi nodes must be authorized in NiFi-Registry with: 1. "Can proxy user requests" - Read, Write, and Delete 2. "Can manage buckets" - Read Within NiFi you have setup your "Registry Client" that looks like this: For secure (HTTPS) NiFi-Registry URL, when an SSL Context Service is not defined, the default keystore and truststore configured in the NiFi.properties file is used to authenticate with NiFi-Registry through a mutual TLS exchange.  1. Did you setup an SSL Context Service? 2. If so, what keystore and truststore did you configure in the service? 3. Does the truststore used in the nifi-registry.properties file contain the necessary public certificates to include the complete trustchain for your NiFi keystore ClientAuth certificates? From the log snippet shared, it looks like the mutual TLS exchange is not resulting in a trusted clientAuth certificate being passed in the exchange (commonly a trust chain issue). So NiFi-Registry then checks for token in connection which would not exist in a NiFi to NiFi-Registry connection.  So what ends up happening is the connection is established as anonymous (just like you as a user sees when accessing NiFi before clicking login button). 2024-10-17 17:39:54,289 DEBUG [NiFi Registry Web Server-15] o.a.n.r.w.s.a.IdentityFilter Attempting to extract user credentials using X509IdentityProvider 2024-10-17 17:39:54,290 DEBUG [NiFi Registry Web Server-15] o.a.n.r.w.s.a.x.X509CertificateExtractor No client certificate found in request. 2024-10-17 17:39:54,290 DEBUG [NiFi Registry Web Server-15] o.a.n.r.w.s.a.IdentityFilter Attempting to extract user credentials using JwtIdentityProvider 2024-10-17 17:39:54,290 DEBUG [NiFi Registry Web Server-15] o.a.n.r.s.a.BearerAuthIdentityProvider HTTP Bearer Auth credentials not present. Not attempting to extract credentials for authentication. 2024-10-17 17:39:54,290 DEBUG [NiFi Registry Web Server-15] o.a.n.r.w.s.a.AnonymousIdentityFilter Set SecurityContextHolder to anonymous SecurityContext So you most likely need to resolve your trust between NiFi and NiFi-Registry.   Typically both NiFi and NiFi-Registry would use the same truststore containing all intermediate and root certificate authorities in the trust chain for the keystore certificates used on both services. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@vg27  Sounds like you may have gotten past your original issue after following the guidance I provided already?  You new questions don't seem directly related to the original query. ------- The invalid host header can be resolved by modifying the nifi.web.proxy.host property in the nifi.properties file:   nifi.web.proxy.host=20.61.182.212,20.61.182.212:9443   Or by including additional entry(s) in your certificates SAN list. --------- Questions: 1. How re you authenticating this user identity?   CN=PAVANBL   2. Have you modified your nif.properties file so that the managed-authorizer is being used?   nifi.security.user.authorizer=managed-authorizer   3. What method(s) or user authentication have you decided to use? Did you just create an clientAuth certificate that you loaded into your browser to authenticate your user or did you setup some other auth method like kerberos or ldap?   nifi.security.user.login.identity.provider= 4. When it comes to using an load balancer, if you are using any type of login provider to authenticate to your NiFi, you need to configure the load-balancer to use session affinity (a.k.a - Sticky sessions).  When you login into a NiFi node the user is issued a client side token and a corresponding server side token is stored on the node.  That client side user token is send in every request made after login.  The issued client token is ONLY valid for use with the NiFi node that issued it.  So session affinity setup in the load balancer is required to make sure that all subsequent request within that same session go to the same NiFi node. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt         ... View more