Member since
07-30-2019
3471
Posts
1642
Kudos Received
1020
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 134 | 06-03-2026 06:06 PM | |
| 451 | 05-06-2026 09:16 AM | |
| 807 | 05-04-2026 05:20 AM | |
| 485 | 05-01-2026 10:15 AM | |
| 616 | 03-23-2026 05:44 AM |
06-02-2021
06:22 AM
1 Kudo
@hegdemahendra The NiFi CLI toolkit [1] can help here to an extent. This toolkit provides the following NiFi-Registry capabilities: registry current-user
registry list-buckets
registry create-bucket
registry delete-bucket
registry list-flows
registry create-flow
registry delete-flow
registry list-flow-versions
registry export-flow-version
registry import-flow-version
registry sync-flow-versions
registry transfer-flow-version
registry diff-flow-versions
registry upload-bundle
registry upload-bundles
registry list-bundle-groups
registry list-bundle-artifacts
registry list-bundle-versions
registry download-bundle
registry get-bundle-checksum
registry list-extension-tags
registry list-extensions
registry list-users
registry create-user
registry update-user
registry list-user-groups
registry create-user-group
registry update-user-group
registry get-policy
registry update-policy
registry update-bucket-policy You can get a description of each by executing: <path to>/cli.sh registry sync-flow-versions -h Since you are changing FlowPersistence providers and not trying to sync flows to a new NiFi-Registry, You really can't use the above "sync-flow-versions" function. Plus, I really don't see it even in that scenario being able to accomplish your goal because you would end up with new flow ids. When you create a bucket in NiFi-Registry it is assigned a bucket if (random uuid). When you version control a Process Group (PG) in NiFi, you choose an existing bucket and it first creates a new flow id (Random UUID assigned to the flow). Then the initial version 1 of that PG flow is created and assigned to that flow id in the NiFi-Registry. Since you cannot force the flow id assigned UUID, syncing flows from registry 1 to registry 2, would not track to your version controlled PGs in your NiFI because of change in flow id. In your scenario, you would need to export all your flows (version by version and it is important you keep rack of the version fo the flow you extract). So for a flow with ID XYZ you may have 6 versions. This means you would use: registry export-flow-version I'd suggest naming the produced json file using source flow id and flow version like XYZ_v1.json, XYZ_v2.json, etc... Example: ./cli.sh registry export-flow-version -ot json -u http://<nifi-registry hostname>:<port>/ -f c97fd570-e2ef-4001-98c9-8810244b6015 -fv 1 -o /tmp/c97fd570-e2ef-4001-98c9-8810244b6015_ver1.json You should then save off your original DB. Delete all existing flows so all you have are your original buckets Then you would need to take all these exported flows and import them back in to registry after switching to your new persistence provider. Now keep in mind before importing each flow version you must first create a new flow within the correct still existing buckets. Keep track of these newly assigned flow ids and which original flow id you are importing in to them (very important) Then you MUST import each new flow in exact version 1 to version x order. If you import version 5 of flow XYZ first it will become version 1 within that new flow Id. The version persisted in the output json is not used when importing, it is assigned the next incremental version in the new flow id. Once you are done here you have a bunch of new flow ids with all your versions imported. Now you need to go edit your flow.xml.gz in NiFi. For every version controlled PG in that flow.xml.gz you will find a section that looks like this: <versionControlInformation>
<registryId>912e8161-0176-1000-ffff-ffff98135aca</registryId>
<bucketId>0cab84ff-399b-4113-9767-687e8e33e48a</bucketId>
<bucketName>bucket-name</bucketName>
<flowId>136b3ba8-bc6f-46dd-afe5-235a80ef8cfe</flowId>
<flowName>flow-name</flowName>
<flowDescription/>
<version>5</version>
</versionControlInformation> Everything here should remain the same except fro the change in "flowId" This would allow you to do a global search and replace on "<flowId>original id</flowId>" to "<flowId>new id</flowId>". Make sure you stop all NiFi nodes, put same modified flow.xml.gz on all nodes (backup original), and start NiFi nodes again. Your PGs should now be tracking to your new flows imported in your registry now backed by the gitFlowPersistenceProvider. [1] https://nifi.apache.org/docs/nifi-docs/html/toolkit-guide.html#nifi_CLI Sorry there is no automated path for this. If you found this addressed your query, please take a moment to login and click "Accept" on those solutions which assisted you. Thanks, Matt
... View more
06-01-2021
07:45 AM
Thanks for the great explanation Matt, much appreciated.
... View more
05-31-2021
08:13 PM
I managed to solve the problem. Below is an example of what I used to get the value of the attributes using the advanced part of the "UpdateAttribute". Hope this helps.
... View more
05-28-2021
09:23 AM
@Amn Where did you define the NiFi Attribute "${itemType}" (FlowFile Attribute on a source FlowFile driving execution of GetMongo processor or NiFi variable registry)? NiFi is also case sensitive, so for example "ItemType" and "itemType" would be considered different variables. If you found this helpful to yoru query, please take a moment to login and click "Accept" on this solution. Thank you, Matt
... View more
05-28-2021
08:23 AM
@Rupesh_Raghani I just want to make sure that when we are both talking about NiFi "Templates" we are talking about the same thing. https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#templates When you upload a NiFi template (xml file) to NiFi via the UI (Does not matter which node in a NiFi cluster you are accessing), that template will get uploaded and replicated to all nodes in the cluster. So all nodes will have that template in each nodes JVM heap and written to the flow.xml.gz on disk. This statement is not clear to me: "So if I have templates on every node in cluster then will the nifi load templates on each node as per templates available on their node or a parent node will load all templates from other cluster". What do you mean by "if I have templates on every node"? In a NiFi cluster, every node must have the same flow.xml.gz. If the flow loaded in to heap memory does not match between nodes, the nodes no matching the elected cluster flow will be disconnected from the cluster. Each node, while it has its own local copy of the flow, runs the exact same flow. Hope this helps, Matt
... View more
05-24-2021
11:56 AM
Some examples https://community.cloudera.com/t5/Support-Questions/How-Extract-text-from-a-multiline-flow-and-create-only-one/td-p/104706 https://nathanlabadie.com/recombining-multiline-logs-with/ https://github.com/tspannhw/EverythingApacheNiFi/blob/main/README.md
... View more
05-21-2021
01:46 PM
@SAMSAL Sounds like you are very close... The 403 has nothing to do with authentication, but rather authorization. This eludes to the fact that the client was able to be properly trusted through the mutual TLS handshake, but when that resulting client string was checked against the endpoint policy being checked it was not present, thus resulting in a 403 response. The DN is the DistinquishedName for your certificate. You can use the below command to get the verbose details on the certs added to a keystore: keytool -v -list -keystore <keystore.jks or truststore.jks> You will see that each certificate has either type "PrivateKeyEntry" or "TrustedCertEntry" For each certificate you will see an "Owner" and an "Issuer" For each of those it will show the complete DN which would look something like: CN=<some string>, OU=<some string>, DC=<some string> There are numerous parts that can be part (CN, OU, O, ST, DC, etc.) of a DN and various lengths The "Owner" DN for the PrivateKeyEntry is what is used as the client/user authenticated string after the successful mutual TLS handshake. The nifi.properties file has optional ideNtity.mapping properties that can be configured to trim and manipulated these identity strings (for example pulling our on the string from the CN). With nifi.remote.input.secure= set to false, FlowFiles send over S2S will not be over a TLS encrypted connection. The fact that you say it works when you have this set to false and you can still successfully obtain S2S details from the configured https:// secured NiFi Destination URL tells me that authentication is correct for "Retrieve site-to-site details" NiFi policy. That only leaves having incorrect policy setup for your remote input port. You should open a command prompt on each node and "tail -F /<path to>/nifi-user.log". Then enable your controller services and check the logs being tailed for the authorization exception. It should clearly show you the "string" being checked against the "receive data via site-to-site" endpoint which in the log would look like "/data-transfer/input-ports/<uuid of remote input port>". Also remember that you are NOT authorizing your user to any of these S2S policies, but rather the DN or identity mapped DN string. Really hope this helps you get fully secured here. Matt
... View more
05-21-2021
10:21 AM
@MattWho, Good afternoon. I changed my flow to make a counter by the processor "RouteOnAttribute 1.12.1" so every time I have a record to be sent to the API and have a return to store in Oracle, I do the LOOP through RouteOnAttribut. Thus sending one FlowFile at a time to the PROCEDURE call in Oracle by "ExecuteSQL 1.12.1" But what is happening at the moment is that from the second time it goes through the LOOP, it seems to me that the Flow to get the attributes "CODIGO" and "JSON_ORIGINAL" is not passing the values that I requested in the processor "EvaluateJsonPath 1.12.1" passing the parameters as null. Do you have any idea what may be happening in this second flow passage within the LOOP? This is the JSON ORIGINAL of the second stream that I must send to Oracle = This is OK with the CODE attribute as you can check. This is Split Configure Processor = This is EvaluateJsonPath 1.12.1 Configure Processor = This is MergeContent 1.12.1 Configure Processor = This is ExecuteSQL 1.12.1 Configure Processor =
... View more
05-20-2021
02:48 PM
1 Kudo
@leandrolinof NiFi Expression Language (NEL) [1] does not read content of the FlowFile. The RouteOnAttribute processor never looks at the content of a FlowFile. So verify your source FlowFile has attributes set with valid numeric only values. So your inbound FlowFile would need to have two attributes on it already: 1. cont 2. CONTADOR Note: NiFi is case sensitive as well. And both these attributes need to have assigned values to them. The NEL statement you have will return the value assigned to the FlowFile attribute "cont" and check to see if it is less than the value assigned to the FlowFile attribute "CONTADOR". If that resolves to "True", the FlowFile will be routed to the connection containing the new dynamically created "CONTINUE" relationship. Otherwise, it will route to the "unmatched" relationship which you appear to have auto-terminated. [1] https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html If you found this addressed your query, please take a moment to login and click accept on this solution. Thank you, Matt
... View more
05-20-2021
06:17 AM
@Kilynn The following property in the nifi.properties file controls when a swap file is created per connection. nifi.queue.swap.threshold=20000 This is per connection and not for all FlowFiles across all connections. A FlowFile swap file will always consist of 10000 FlowFiles. So if a connection reaches 20000 queued FlowFiles, a swap file will be created for 10000 of those. So if a connection queue reaches 40000, you would have 3 swap files of that connection. You can control individual connection queues by setting the "Back pressure Object Threshold" on a connection: Note: Threshold settings are soft limits And default for object threshold is 10000. So with these settings there should be very little to no swapping of FlowFiles to disk happening at all. Swap files would only happen if source processor to that connection output enough FlowFiles to connection at one time to trigger a swap file. For example: - Connection has 9000 queued FlowFiles, so back pressure is not being applied. - Source processor is thus allowed to execute - Source processor upon execution produces 12000 FlowFiles - now downstream connection has 21000 queued FlowFiles. One swap file is produced and back pressure is enabled until queue drops back below 10000 queued FlowFiles. FlowFiles consist of two parts (FlowFile attributes/metadata and FlowFileContent). The only portion of a FlowFile held in heap memory is the FlowFile attributes/Metadata. FlowFile content is never held in memory (Some processors may load content in to memory in order to perform their function only). FlowFile attributes/metadata is persisted to the flowfile repository and FlowFile content is written to the content repository. This important to avoid data loss if NiFi dies or is restarted while data still exists in connection queues. If you found this helped with your query, please take a moment to login and click accept in the solutions that helped. Thank you, Matt
... View more