@midee  I would recommend setting up two dataflows here. Dataflow 1: Generate FlowFile processor configured to run once very 28 minutes which is connected to an InvokeHTTP processor that is configured to get you Authentication token and write it to a FlowFile attribute.  The InvokeHTTP processor then connects to PutDistributedMapCache processor to store that authentication token. Dataflow 2: This is the flow that will use that stored token for your next Rest API call.  This flow would use a FetchDistributedMapCache processor to get the stored token and then route success to an invokeHTTP processor to make your Rest API call.  If it fails, the failure and No Retry relationships should be looped back to the FetchDistributedMapCache processor so it can get auth token again (perhaps one used previous request just expired and a new one is now loaded in the map cache), so it tries your rest api call again. If you found this addressed your query, please take a moment to login and click "Accept" on this solution. Thank you, Matt ... View more

@ang_coder  The RouteOnAttribute processor establishes a NEW relationship for each dynamic property you add. If you intent is that a single FlowFile must satisfy all conditions to route on, then you should have just one NiFi Expression Language (NEL) statement that covers all conditions resulting a true or false boolean.  If you share your two statements, I'd be happy to help you construct a single NEL statement. butt would be structured something like something like: All 4 lines would need to result in true before FlowFile would be related to this dynamic property's relationship. If you found this addressed your query, please take a moment to login and click "Accept" on this solution. Thank you, Matt   ... View more

@Chakr  So first thing to check is proper configuration of your Authorizers.xml file. The order in which the sections are added is very important. The file is loaded from top down, so you can NOT have a section refer to another section that is not yet loaded.  And you users need to be loaded/created before the authorizations can be loaded/created. So your authorizers.xml file structure should look like this:   <authorizers> <userGroupProvider> <identifier>file-user-group-provider</identifier> <class>org.apache.nifi.authorization.FileUserGroupProvider</class> <property name="Users File">./conf/users.xml</property> <property name="Legacy Authorized Users File"></property> <property name="Initial User Identity 1">CN=chakri, OU=NIFI</property> <property name="Initial User Identity 2">CN=nifi-1,OU=NIFI</property> <property name="Initial User Identity 3">CN=nifi-2,OU=NIFI</property> <property name="Initial User Identity 4">CN=nifi-3,OU=NIFI</property> </userGroupProvider> <accessPolicyProvider> <identifier>file-access-policy-provider</identifier> <class>org.apache.nifi.authorization.FileAccessPolicyProvider</class> <property name="User Group Provider">file-user-group-provider</property> <property name="Authorizations File">./conf/authorizations.xml</property> <property name="Initial Admin Identity">CN=chakri, OU=NIFI</property> <property name="Legacy Authorized Users File"></property> <property name="Node Identity 1">CN=nifi-1,OU=NIFI</property> <property name="Node Identity 2">CN=nifi-2,OU=NIFI</property> <property name="Node Identity 3">CN=nifi-3,OU=NIFI</property> <property name="Node Group"></property> </accessPolicyProvider> <authorizer> <identifier>managed-authorizer</identifier> <class>org.apache.nifi.authorization.StandardManagedAuthorizer</class> <property name="Access Policy Provider">file-access-policy-provider</property> </authorizer> </authorizers>   Note: Changed your authorizer to the newer "StandardManagedAuthorizer". On NiFi startup the "file-user-group-provider" will create the users.xml file only if it does NOT already exist and seed it with all four configured "Initial User Identity" strings.  These strings (case sensitive) will each be assigned a UUID. On NIFi Startup the "file-access-policy-provider" will create the authorizations.xml file only if it does NOT already exist and seed it with relevant policies needed initially for your admin user and your nodes based off the UUIDs create for each user identity generated by the first provider. The the Authorizer is loaded which reads in the users and authorizations from the users.xml and authorizations.xml files that are now created. You should do the following: 1. Edit your authorizers.xml to match above structure above. 2. In your nifi.properties file, make sure following property is set correctly as: nifi.security.user.authorizer=managed-authorizer 3. Remove/rename current users.xml and authorizations.xml files. 4. Start NiFi    You should not be able to successfully access your NiFi's UI. If you fin this addresses your query, please take a moment to login and click "Accept" on this solution. Thank you, Matt ... View more

@Pe  Communications between a secured NiFi-Registry and a Secured NiFi cluster node or standalone NiFi instance will use a mutual TLS connection.  At a very high level... - That means the client (NiFi node) will initiate a connection to the server (NiFi-Registry).  T - The NiFi-Registry will responded via that handshake and present its serverAuth certificate along with a list of trusted entities (this would be the list of DNs from the trustedCertEntries found in the NiFi-Registry's truststore file).  - The Client NiFi node will verify that it trusts the server certificate presented. This means the connecting NiFi node's truststore must contain the complete trust chain for the NiFi-registry's privateKeyEntry certificate. - The client will then respond to that server Hello with a client hello that includes the clientAuth certificate from that NiFi nodes keystore file (Only way it will send a client certificate is if that client certificate was signed by by one of the DNs presented from the NiFi-Registry. No sense sending a client certificate back to NiFi-Registry, if NiFi-Registry is unable to trust it).  - The NiFi-Registry will then use that DN from the NiFi client certificate to identify the connecting NiFi instance.   - NiFi-Registry will then check that that connecting NiFi node is authorized properly. Now many setting up additional NiFi clusters or instances, the same applies to each of those NiFi nodes when communicating to this one NiFi-Registry. - So truststore on every NiFi node must contain the complete trust chain for the NiFi-Registry's certificate. - The NiiFi-Registry's trustsstore file must contain the complete trust chain for the NiFi instances keystore files certificate. (Each NiFi node should have its own unique keystore file that contains only 1 PrivateKeyEntry) - It is very common to create 1 truststore file with every needed trustedCertEntry (public certificates) and then distributed that one truststore to all instance of NiFi and NiFi-Registry. NOTE:  The NiFi keystore files must meet following criteria. 1. Must contain ONLY 1 PrivateKeyEntry. Having more than 1 PrivateKeyEntry will not work as NiFi will not know which to use.  2. The DN used in the PrivateKeyEntry must not contain wildcards. Since NiFi certificate is used for ClientAuth, the PrivateKeyEntry DN is what is presented to identify the node.  Many Authorizers will not support client names with wildcards, plus it is not advisable security wise. 3. The PrivateKeyEntry must have an Extended Key Usage (EKU) that supports both clientAuth and serverAuth 4. The PrivateKeyEntry must have at least 1 SAN entry that matches the hostname for the server on which the keystore is being used. If you found this assisted with your query, please take a moment to login and click "Accept" on this solution. Thank you, Matt ... View more

@Acbx  It looks like your CSV uses commas as the field delimiter.  So the solution i provided parses the entire file line by line and changes all "." to ",".  So, I am guessing that you have other places within your CSV that also had ".", thus creating the additional 5 field columns. Are trying to create a new column for cents?  Is that why you are changing 109.29 tp 109,29? If you are not looking for a new column, how will downstream system parse this edited CSV now that you added a new comma in there? You could write a complex Java regular expression in the Search Value to match only specifically on column number X (Money Column) and then use Replacement Strategy "Regex Replace" to edit it. Let's assume the "Money" Column was column number 5. And then wrap money once converted from 109.29 to 109,29 in quotes so it is not treated as two columns later on.... Search Value: ^(.*?),(.*?),(.*?),(.*?),(.*?),(.*?)$ Replacement Value:  $1,$2,$3,$4,"${'$5':replace(".",",")}",$6   So above would manipulate column 5 only and change 109.29 in to "109,29". Hope this helps you, Matt ... View more

@myuintelli2021  Noticed in another post from you that commented: I am aware that there are 3 TLS certificates (one for each server) stored in keystore and 1 self-signed CA (stored in truststore) for nifi cluster.   NiFi keystore used in each node MUST meet following minimum criteria: - Must contain ONLY 1 PrivateKeyEntry. Having more than 1 PrivateKeyEntry will not work as NiFi will not know which to use.  - The DN used in the PrivateKeyEntry must not contain wildcards. Since NiFi certificate is used for ClientAuth, the PrivateKeyEntry DN is what is presented to identify the node.  Many Authorizers will not support client names with wildcards, plus it is not advisable security wise. - The PrivateKeyEntry must have an Extended Key Usage (EKU) that supports both clientAuth and serverAuth - The PrivateKeyEntry must have at least 1 SAN entry that matches the hostname for the server on which the keystore is being used. Assuming since you used the NiFi CA toolkit to build your keystores and truststore files, you are good here. Just adding this detail in case you switch a some point to using private or publicly signed certificates. Thanks, Matt ... View more

@myuintelli2021  Assisting on your new post here: https://community.cloudera.com/t5/Support-Questions/Nifi-untrusted-proxy-caused-by-Untrusted-Proxy-Exception/m-p/317796/highlight/false#M227327 Your choice of user authentication does not matter here.  Authentication and Authorization processes are handled independently of one another.  The Authentication of users/clients results in a string which is evaluated against identity mapping properties and then passed to the configured authorizer for authorization. Your exception points and missing /proxy authorization for your node strings. Hope this helps, Matt ... View more

@myuintelli2021    The "Untrusted Proxy Exception" issue has nothing to do with Authentication. It is an authorization issue within NiFi. In a NiFi cluster, a users authenticates access to the specific node for which the opened the UI.  Subsequent request to access specific resource endpoints (like being able to view the UI), need to be  the sent to the cluster coordinator node.  The cluster coordinator then replicates that request to all nodes in the cluster and confirms each node responded to that request. When for example node 1 needs to send a request made my user 1 to node 2, node 1 makes this request on behalf of the authenticated user 1.   So node 1 is proxying users 1's request.  So within NiFi authorizations, node 1 must be authorized to act as a proxy. NiFi offers multiple authorizers, so the first question here is which authorizer are you using? How is your authorizers.xml configured? A very simple setup that utilizes a local managed authorizer might look like this:   <authorizers> <userGroupProvider> <identifier>file-user-group-provider</identifier> <class>org.apache.nifi.authorization.FileUserGroupProvider</class> <property name="Users File">./conf/users.xml</property> <property name="Legacy Authorized Users File"></property> <property name="Initial User Identity 1">user@{valid_domain}.com</property> property name="Initial User Identity 2">CN=nifi2.{valid_domain}.com, OU=NIFI</property> property name="Initial User Identity 3">CN=nifi3.{valid_domain}.com, OU=NIFI</property> property name="Initial User Identity 4">CN=nifi4.{valid_domain}.com, OU=NIFI</property> </userGroupProvider> <accessPolicyProvider> <identifier>file-access-policy-provider</identifier> <class>org.apache.nifi.authorization.FileAccessPolicyProvider</class> <property name="User Group Provider">file-user-group-provider</property> <property name="Authorizations File">./conf/authorizations.xml</property> <property name="Initial Admin Identity">user@{valid_domain}.com</property> <property name="Legacy Authorized Users File"></property> <property name="Node Identity 1">CN=nifi2.{valid_domain}.com, OU=NIFI</property> <property name="Node Identity 1">CN=nifi3.{valid_domain}.com, OU=NIFI</property> <property name="Node Identity 1">CN=nifi4.{valid_domain}.com, OU=NIFI</property> </accessPolicyProvider> <authorizer> <identifier>managed-authorizer</identifier> <class>org.apache.nifi.authorization.StandardManagedAuthorizer</class> <property name="Access Policy Provider">file-access-policy-provider</property> </authorizer> </authorizers>   The "file-user-group-provider" is used here to generate the "users.xml" file initially.  Once created, this provider will not edit it if you make changes (only does anything if the file does NOT exist already).  So I would suggest you check this file to make sure yoru initial admin and all 3 of your nodes are present in this file (case sensitive). The "file-access-policy-provider" is used here to generate the "authorizations.xml", which sets up the minimum necessary authorization policies needed fro your initial admin user and your NiFi cluster nodes.  If you look in this file, you will see numerous policies that your initial admins assigned UUID should be authorized for.  You should also see /proxy policies set in here with the assigned UUIDs from your NiFi nodes. If these are missing, you should correct your authorizers.xml, remove the existing users.xml and authorizations.xml files, and restart your NiFi so these two files are created again. Also noticed from your log snippet that it is identifying your nodes via their entire DN (CN=nifi2.{valid_domain}.com, OU=NIFI).   There exists the ability to setup identity mapping properties in the nifi.properties file which are used to reformat authenticated users/nodes. For example, setting below three properties would change "CN=nifi2.{valid_domain}.com, OU=NIFI" into "nifi2.{valid_domain}.com": nifi.security.identity.mapping.pattern.dn=^CN=(.*?), OU=(.*?)$ nifi.security.identity.mapping.value.dn=$1 nifi.security.identity.mapping.transform.dn=LOWER Then all you would need to do is have the lowercase "nifi2.{valid_domain}.com" hostnames populated in your authorizers.xml file. You can create as many sets of identity mapping properties as you like as long as each set using a unique string in the property names (above uses "dn"), but you could create another set using dn2, dn3, kerb2, kerb3, username, etc.   Just keep in mind that all authenticated user/client strings will be evaluated by these mapping properties in alphanumeric order and first matching pattern will have its value and transform applied. After that any additional pattern properties would not get evaluated.   If you found this assisted with your query, please take a moment to login and click "accept" on this solution. Thank you, Matt ... View more

@khaldoune    Some components that maintain state do so because they were developed with intent of being used in NiFi cluster setup to support non cluster friendly protocols. Example (getting data from SFTP server): In Standalone NiFi you would use the GetSFTP processor (does not record state). In Cluster NiFi you would use the ListSFTP (records state) and FetchSFTP processor to do the same task. The ListSFTP processor would be configured to execute on "primary node" only.  That way you do not have every node in your cluster trying to list the same files on yoru target SFTP processor.  Then the success from listSFTP which simply has FlowFiles with no content and only metadata/attributes is connected to a FetchSFTP processor.  That connection between those two processors would be configured to load balance those FlowFiles to all nodes.  Now the heavy work of ingesting the actual content for each of those listed FlowFiles is spread across all nodes in the cluster. Even if you use above processors in a standalone, they will still record state. Cluster state is generally stored to help when a primary node change occurs.  That way the newly elected primary node that now starts executing the primary node only configured processors, will have those processor fetch that last known state from ZK so that it does not list the same files already listed by previous primary node. Just some more context for you on how state is used primarily by components and why. Matt ... View more