Member since
07-30-2019
3470
Posts
1642
Kudos Received
1018
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 301 | 05-06-2026 09:16 AM | |
| 492 | 05-04-2026 05:20 AM | |
| 355 | 05-01-2026 10:15 AM | |
| 524 | 03-23-2026 05:44 AM | |
| 394 | 02-18-2026 09:59 AM |
06-08-2021
06:07 AM
@techNerd The PutSFTP processor contains the following configuration property: Do you have that set to false on the particular putSFTP processor throwing the exception? Thanks, Matt
... View more
06-07-2021
08:34 AM
@Acbx It looks like your CSV uses commas as the field delimiter. So the solution i provided parses the entire file line by line and changes all "." to ",". So, I am guessing that you have other places within your CSV that also had ".", thus creating the additional 5 field columns. Are trying to create a new column for cents? Is that why you are changing 109.29 tp 109,29? If you are not looking for a new column, how will downstream system parse this edited CSV now that you added a new comma in there? You could write a complex Java regular expression in the Search Value to match only specifically on column number X (Money Column) and then use Replacement Strategy "Regex Replace" to edit it. Let's assume the "Money" Column was column number 5. And then wrap money once converted from 109.29 to 109,29 in quotes so it is not treated as two columns later on.... Search Value: ^(.*?),(.*?),(.*?),(.*?),(.*?),(.*?)$ Replacement Value: $1,$2,$3,$4,"${'$5':replace(".",",")}",$6 So above would manipulate column 5 only and change 109.29 in to "109,29". Hope this helps you, Matt
... View more
06-07-2021
08:05 AM
@myuintelli2021 Noticed in another post from you that commented: I am aware that there are 3 TLS certificates (one for each server) stored in keystore and 1 self-signed CA (stored in truststore) for nifi cluster. NiFi keystore used in each node MUST meet following minimum criteria: - Must contain ONLY 1 PrivateKeyEntry. Having more than 1 PrivateKeyEntry will not work as NiFi will not know which to use. - The DN used in the PrivateKeyEntry must not contain wildcards. Since NiFi certificate is used for ClientAuth, the PrivateKeyEntry DN is what is presented to identify the node. Many Authorizers will not support client names with wildcards, plus it is not advisable security wise. - The PrivateKeyEntry must have an Extended Key Usage (EKU) that supports both clientAuth and serverAuth - The PrivateKeyEntry must have at least 1 SAN entry that matches the hostname for the server on which the keystore is being used. Assuming since you used the NiFi CA toolkit to build your keystores and truststore files, you are good here. Just adding this detail in case you switch a some point to using private or publicly signed certificates. Thanks, Matt
... View more
06-07-2021
07:43 AM
@myuintelli2021 The "Untrusted Proxy Exception" issue has nothing to do with Authentication. It is an authorization issue within NiFi. In a NiFi cluster, a users authenticates access to the specific node for which the opened the UI. Subsequent request to access specific resource endpoints (like being able to view the UI), need to be the sent to the cluster coordinator node. The cluster coordinator then replicates that request to all nodes in the cluster and confirms each node responded to that request. When for example node 1 needs to send a request made my user 1 to node 2, node 1 makes this request on behalf of the authenticated user 1. So node 1 is proxying users 1's request. So within NiFi authorizations, node 1 must be authorized to act as a proxy. NiFi offers multiple authorizers, so the first question here is which authorizer are you using? How is your authorizers.xml configured? A very simple setup that utilizes a local managed authorizer might look like this: <authorizers>
<userGroupProvider>
<identifier>file-user-group-provider</identifier>
<class>org.apache.nifi.authorization.FileUserGroupProvider</class>
<property name="Users File">./conf/users.xml</property>
<property name="Legacy Authorized Users File"></property>
<property name="Initial User Identity 1">user@{valid_domain}.com</property>
property name="Initial User Identity 2">CN=nifi2.{valid_domain}.com, OU=NIFI</property>
property name="Initial User Identity 3">CN=nifi3.{valid_domain}.com, OU=NIFI</property>
property name="Initial User Identity 4">CN=nifi4.{valid_domain}.com, OU=NIFI</property>
</userGroupProvider>
<accessPolicyProvider>
<identifier>file-access-policy-provider</identifier>
<class>org.apache.nifi.authorization.FileAccessPolicyProvider</class>
<property name="User Group Provider">file-user-group-provider</property>
<property name="Authorizations File">./conf/authorizations.xml</property>
<property name="Initial Admin Identity">user@{valid_domain}.com</property>
<property name="Legacy Authorized Users File"></property>
<property name="Node Identity 1">CN=nifi2.{valid_domain}.com, OU=NIFI</property>
<property name="Node Identity 1">CN=nifi3.{valid_domain}.com, OU=NIFI</property>
<property name="Node Identity 1">CN=nifi4.{valid_domain}.com, OU=NIFI</property>
</accessPolicyProvider>
<authorizer>
<identifier>managed-authorizer</identifier>
<class>org.apache.nifi.authorization.StandardManagedAuthorizer</class>
<property name="Access Policy Provider">file-access-policy-provider</property>
</authorizer>
</authorizers> The "file-user-group-provider" is used here to generate the "users.xml" file initially. Once created, this provider will not edit it if you make changes (only does anything if the file does NOT exist already). So I would suggest you check this file to make sure yoru initial admin and all 3 of your nodes are present in this file (case sensitive). The "file-access-policy-provider" is used here to generate the "authorizations.xml", which sets up the minimum necessary authorization policies needed fro your initial admin user and your NiFi cluster nodes. If you look in this file, you will see numerous policies that your initial admins assigned UUID should be authorized for. You should also see /proxy policies set in here with the assigned UUIDs from your NiFi nodes. If these are missing, you should correct your authorizers.xml, remove the existing users.xml and authorizations.xml files, and restart your NiFi so these two files are created again. Also noticed from your log snippet that it is identifying your nodes via their entire DN (CN=nifi2.{valid_domain}.com, OU=NIFI). There exists the ability to setup identity mapping properties in the nifi.properties file which are used to reformat authenticated users/nodes. For example, setting below three properties would change "CN=nifi2.{valid_domain}.com, OU=NIFI" into "nifi2.{valid_domain}.com": nifi.security.identity.mapping.pattern.dn=^CN=(.*?), OU=(.*?)$
nifi.security.identity.mapping.value.dn=$1
nifi.security.identity.mapping.transform.dn=LOWER Then all you would need to do is have the lowercase "nifi2.{valid_domain}.com" hostnames populated in your authorizers.xml file. You can create as many sets of identity mapping properties as you like as long as each set using a unique string in the property names (above uses "dn"), but you could create another set using dn2, dn3, kerb2, kerb3, username, etc. Just keep in mind that all authenticated user/client strings will be evaluated by these mapping properties in alphanumeric order and first matching pattern will have its value and transform applied. After that any additional pattern properties would not get evaluated. If you found this assisted with your query, please take a moment to login and click "accept" on this solution. Thank you, Matt
... View more
06-04-2021
01:44 PM
@khaldoune Some components that maintain state do so because they were developed with intent of being used in NiFi cluster setup to support non cluster friendly protocols. Example (getting data from SFTP server): In Standalone NiFi you would use the GetSFTP processor (does not record state). In Cluster NiFi you would use the ListSFTP (records state) and FetchSFTP processor to do the same task. The ListSFTP processor would be configured to execute on "primary node" only. That way you do not have every node in your cluster trying to list the same files on yoru target SFTP processor. Then the success from listSFTP which simply has FlowFiles with no content and only metadata/attributes is connected to a FetchSFTP processor. That connection between those two processors would be configured to load balance those FlowFiles to all nodes. Now the heavy work of ingesting the actual content for each of those listed FlowFiles is spread across all nodes in the cluster. Even if you use above processors in a standalone, they will still record state. Cluster state is generally stored to help when a primary node change occurs. That way the newly elected primary node that now starts executing the primary node only configured processors, will have those processor fetch that last known state from ZK so that it does not list the same files already listed by previous primary node. Just some more context for you on how state is used primarily by components and why. Matt
... View more
06-04-2021
01:33 PM
1 Kudo
@khaldoune For a standalone NiFi (meaning that "nifi.cluster.is.node" is set to false in nifi.properties file), components (processors, controller services, and reporting tasks) that write state will use local state directory to record state. Problem here is that if you switch to being clustered later, there is no way to move the components state from local to zookeeper. NOTE: It is possible to have a 1 node NiFi cluster (offers no HA control plane that way), but it will still require that you have a zookeeper quorum. Hope this helps, Matt
... View more
06-04-2021
01:26 PM
@Griggsy Here is an example of how you could use ReplaceText: Search value: [^\w\d\r\n! @^$*#()_=<>~`|{}:;,.\-\\\?\/]+ The below site is a great way to test above regex against some sample data you have to make sure you are not missing any exceptions you want to keep https://regex101.com/ It also explains this above regex formatting If you found these responses helpful, please take a moment to login and click "Accept" on all solutions that help you. Thanks, Matt
... View more
06-03-2021
09:45 AM
1 Kudo
@khaldoune NiFi state is used by only NiFi components/frameworks bits that are built to use it. Some select components can be configured to use local state even if you are setup with a NiFi cluster. Others select components will use cluster state if NiFi is clustered, but in a standalone NiFi use local state. You can refer to the embedded documentation for each component processor, controller service, or reporting task to see if it uses the state provider. For example, look at the embedded docs for ListFile and you will see a "state management:" section with: Above you see that this processor can use local or cluster state provider and a description of how state us utilized by this component. Configuration of the "Input Directory Location" in the case of this specific processor controls which provider is used. For components that do not use state (bulk of components don't use state), the same section in their embedded docs will reflect: Load balanced connections to do not record state. Load balanced connections copy FlowFiles from one node to another and on confirmation of success, the local copies are removed. So if NiFi is shutdown or dies while data is being copied by a load balanced connection the source NiFi will simply start over distributing the FlowFiles again when it is back online in the cluster. If you found this addressed your query, please take a moment to login and click "Accept" on this solution. Thank you, Matt
... View more
06-02-2021
09:30 AM
@_mark_ As NiFi is an open source product, I recommend joining the community (if you have not already) and opening an Apache NiFi Jira [1] with you proposed enhancements/new features for Apache NiFi to get feedback from the community at large. If you feel you are not there yet in proposing a new feature/enhancement, try engaging via the users mailing list [2] [1] https://issues.apache.org/jira/browse/NIFI [2] https://nifi.apache.org/mailing_lists.html Thanks, Matt
... View more
06-02-2021
07:52 AM
@midee I am not clearly following your use case. FlowFiles consist of two parts, FlowFile attributes/metadata and FlowFile content. You give example with "customefields_12345" and "customefields_12346". Does this mean one FlowFile may have multiple "customefields_<some string>" attributes assigned to it? How do you want to route FlowFiles where only some of those customfield attributes are null while others are not? There are multiple ways to handle this using NiFi Expression Language (NEL) [1] and the routeOnAttribute [2] processor. ${anyMatchingAttribute("customfield.*"):isEmpty()} Above would return "true" if ANY of the NiFi FlowFile attributes starting with "customefield" is empty. note: The isEmpty function returns true if the Subject is null, does not contain any characters or contains only white-space (new line, carriage return, space, tab), false otherwise. There is another NEL subjectless function that would return "true" only if ALL FlowFileAttributes matching the Java regular expression were empty: ${allMatchingAttributes("customfield.*"):isEmpty()} With the RouteOnAttribute processor you create/add dynamic properties and each of those becomes a new routable relationship on the processor. if the NEL statement configured for that dynamic property returns true that FlowFile routes to that relationship. Any FlowFile that does not return true for dynamic properties will get routed to the pre-existing relationship named "unmatched". [1] https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html [2] https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.13.2/org.apache.nifi.processors.standard.RouteOnAttribute/index.html If you found this addressed your query, please take a moment to login and click "Accept" on this solution. Thank you, Matt
... View more