Member since
07-30-2019
3436
Posts
1632
Kudos Received
1012
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 138 | 01-27-2026 12:46 PM | |
| 559 | 01-13-2026 11:14 AM | |
| 1209 | 01-09-2026 06:58 AM | |
| 1004 | 12-17-2025 05:55 AM | |
| 494 | 12-17-2025 05:34 AM |
06-07-2021
04:21 AM
Thanks for both replies, I managed to get it working last week the same way as you have shown Matt. Cheers
... View more
06-04-2021
01:44 PM
@khaldoune Some components that maintain state do so because they were developed with intent of being used in NiFi cluster setup to support non cluster friendly protocols. Example (getting data from SFTP server): In Standalone NiFi you would use the GetSFTP processor (does not record state). In Cluster NiFi you would use the ListSFTP (records state) and FetchSFTP processor to do the same task. The ListSFTP processor would be configured to execute on "primary node" only. That way you do not have every node in your cluster trying to list the same files on yoru target SFTP processor. Then the success from listSFTP which simply has FlowFiles with no content and only metadata/attributes is connected to a FetchSFTP processor. That connection between those two processors would be configured to load balance those FlowFiles to all nodes. Now the heavy work of ingesting the actual content for each of those listed FlowFiles is spread across all nodes in the cluster. Even if you use above processors in a standalone, they will still record state. Cluster state is generally stored to help when a primary node change occurs. That way the newly elected primary node that now starts executing the primary node only configured processors, will have those processor fetch that last known state from ZK so that it does not list the same files already listed by previous primary node. Just some more context for you on how state is used primarily by components and why. Matt
... View more
06-03-2021
02:05 PM
1 Kudo
@MrWilkinson That could be a java-config issue Install JDK 8.0 64 bit.
Install Java to C:/java instead of C:/Program Files.
Recent Windows versions mark everything in C:\Program Files as read only.
Set the JAVA_HOME environment variable using the 8.3 style name conventions.
For example: C:\Program\jdk1.8.0.
Ensure JAVA_HOME is pointing to a 64-bit JRE/JDK.
Ensure your system meets the minimum memory requirement for Windows which is 4GB Please share your install steps?
... View more
06-03-2021
06:32 AM
Hi @MattWho , Thanks for the response. I'll happily create a feature request, though I wasn't sure if I was missing something obvious that would meet my objective/requirements. Thank you for clarifying, I'll go take a look at the links you provided (admittedly I had missed the mailing lists, oops). Kind regards Mark
... View more
06-03-2021
05:33 AM
@aie Has your issue been resolved? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future.
... View more
06-02-2021
07:52 AM
@midee I am not clearly following your use case. FlowFiles consist of two parts, FlowFile attributes/metadata and FlowFile content. You give example with "customefields_12345" and "customefields_12346". Does this mean one FlowFile may have multiple "customefields_<some string>" attributes assigned to it? How do you want to route FlowFiles where only some of those customfield attributes are null while others are not? There are multiple ways to handle this using NiFi Expression Language (NEL) [1] and the routeOnAttribute [2] processor. ${anyMatchingAttribute("customfield.*"):isEmpty()} Above would return "true" if ANY of the NiFi FlowFile attributes starting with "customefield" is empty. note: The isEmpty function returns true if the Subject is null, does not contain any characters or contains only white-space (new line, carriage return, space, tab), false otherwise. There is another NEL subjectless function that would return "true" only if ALL FlowFileAttributes matching the Java regular expression were empty: ${allMatchingAttributes("customfield.*"):isEmpty()} With the RouteOnAttribute processor you create/add dynamic properties and each of those becomes a new routable relationship on the processor. if the NEL statement configured for that dynamic property returns true that FlowFile routes to that relationship. Any FlowFile that does not return true for dynamic properties will get routed to the pre-existing relationship named "unmatched". [1] https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html [2] https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.13.2/org.apache.nifi.processors.standard.RouteOnAttribute/index.html If you found this addressed your query, please take a moment to login and click "Accept" on this solution. Thank you, Matt
... View more
06-02-2021
06:22 AM
1 Kudo
@hegdemahendra The NiFi CLI toolkit [1] can help here to an extent. This toolkit provides the following NiFi-Registry capabilities: registry current-user
registry list-buckets
registry create-bucket
registry delete-bucket
registry list-flows
registry create-flow
registry delete-flow
registry list-flow-versions
registry export-flow-version
registry import-flow-version
registry sync-flow-versions
registry transfer-flow-version
registry diff-flow-versions
registry upload-bundle
registry upload-bundles
registry list-bundle-groups
registry list-bundle-artifacts
registry list-bundle-versions
registry download-bundle
registry get-bundle-checksum
registry list-extension-tags
registry list-extensions
registry list-users
registry create-user
registry update-user
registry list-user-groups
registry create-user-group
registry update-user-group
registry get-policy
registry update-policy
registry update-bucket-policy You can get a description of each by executing: <path to>/cli.sh registry sync-flow-versions -h Since you are changing FlowPersistence providers and not trying to sync flows to a new NiFi-Registry, You really can't use the above "sync-flow-versions" function. Plus, I really don't see it even in that scenario being able to accomplish your goal because you would end up with new flow ids. When you create a bucket in NiFi-Registry it is assigned a bucket if (random uuid). When you version control a Process Group (PG) in NiFi, you choose an existing bucket and it first creates a new flow id (Random UUID assigned to the flow). Then the initial version 1 of that PG flow is created and assigned to that flow id in the NiFi-Registry. Since you cannot force the flow id assigned UUID, syncing flows from registry 1 to registry 2, would not track to your version controlled PGs in your NiFI because of change in flow id. In your scenario, you would need to export all your flows (version by version and it is important you keep rack of the version fo the flow you extract). So for a flow with ID XYZ you may have 6 versions. This means you would use: registry export-flow-version I'd suggest naming the produced json file using source flow id and flow version like XYZ_v1.json, XYZ_v2.json, etc... Example: ./cli.sh registry export-flow-version -ot json -u http://<nifi-registry hostname>:<port>/ -f c97fd570-e2ef-4001-98c9-8810244b6015 -fv 1 -o /tmp/c97fd570-e2ef-4001-98c9-8810244b6015_ver1.json You should then save off your original DB. Delete all existing flows so all you have are your original buckets Then you would need to take all these exported flows and import them back in to registry after switching to your new persistence provider. Now keep in mind before importing each flow version you must first create a new flow within the correct still existing buckets. Keep track of these newly assigned flow ids and which original flow id you are importing in to them (very important) Then you MUST import each new flow in exact version 1 to version x order. If you import version 5 of flow XYZ first it will become version 1 within that new flow Id. The version persisted in the output json is not used when importing, it is assigned the next incremental version in the new flow id. Once you are done here you have a bunch of new flow ids with all your versions imported. Now you need to go edit your flow.xml.gz in NiFi. For every version controlled PG in that flow.xml.gz you will find a section that looks like this: <versionControlInformation>
<registryId>912e8161-0176-1000-ffff-ffff98135aca</registryId>
<bucketId>0cab84ff-399b-4113-9767-687e8e33e48a</bucketId>
<bucketName>bucket-name</bucketName>
<flowId>136b3ba8-bc6f-46dd-afe5-235a80ef8cfe</flowId>
<flowName>flow-name</flowName>
<flowDescription/>
<version>5</version>
</versionControlInformation> Everything here should remain the same except fro the change in "flowId" This would allow you to do a global search and replace on "<flowId>original id</flowId>" to "<flowId>new id</flowId>". Make sure you stop all NiFi nodes, put same modified flow.xml.gz on all nodes (backup original), and start NiFi nodes again. Your PGs should now be tracking to your new flows imported in your registry now backed by the gitFlowPersistenceProvider. [1] https://nifi.apache.org/docs/nifi-docs/html/toolkit-guide.html#nifi_CLI Sorry there is no automated path for this. If you found this addressed your query, please take a moment to login and click "Accept" on those solutions which assisted you. Thanks, Matt
... View more
06-02-2021
06:17 AM
@Maqbool Can you provide some more detail on what you are trying to accomplish here? If we look at the NiFi Expression Language (NEL) example that does work you shared: ${category_mappings:jsonPath("$.catCode")} NEL evaluates this as follows: Finds NiFi variable with name "category_mappings" and return the value assigned to that variable (Depending on level of NEL support in the component property were this is used, it may search for that variable name in multiple places (FlowFile attributes, variable registry, java properties, environment variables). The returned value is then passed to the NEL function :jsonPath(). Now if we look at your parameter context example: #{category_mappings:jsonPath("$.catCode")} It is looking for the parameter context with string "category_mappings:jsonPath("$.catCode")" which it did not find in the "Parameters" assigned to the Process Group in which this processor resides. I suspect this is not what you really are trying to accomplish here. I suspect that the parameter context you have in your parameters on this process group is really only "category_mappings" and you still need to use NEL in order to execute a NEL function against that parameter's value. ${#{category_mappings}:jsonPath("$.catCode")} You'll notice the slight difference here. We now have a NEL statement that gets its subject from a parameter "category_mappings" instead of variable and passes that value to the NEL ":jsonPath()" function. If you found this addresses your query, please take a moment to login and click "Accept" on this solution. Thank you, Matt
... View more
06-01-2021
07:45 AM
Thanks for the great explanation Matt, much appreciated.
... View more
05-31-2021
08:13 PM
I managed to solve the problem. Below is an example of what I used to get the value of the attributes using the advanced part of the "UpdateAttribute". Hope this helps.
... View more