Member since
07-25-2022
29
Posts
7
Kudos Received
4
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1744 | 11-06-2024 02:11 PM | |
1242 | 08-14-2024 01:31 PM | |
2818 | 08-14-2024 12:52 PM | |
1163 | 04-04-2024 08:45 AM |
06-06-2025
12:01 PM
Thanks @MattWho for your valuable inputs. This is very helpful for conducting the correct analysis on our end. I will check the possibility of customizing the processors that connect to the database.
... View more
05-02-2025
07:25 AM
@shiva239 There are numerous things happening when a node is disconnected. A disconnected node is different then a dropped node. A cluster node must be disconnected before it can be dropped. A node can become disconnected in two ways: Manually disconnected - User manually disconnects a node via the NiFi Cluster UI. A manually disconnected node will not attempt to auto-rejoin cluster. A user can manually reconnect the node from another node in the cluster via the same Cluster UI. A node becomes disconnected due to some issue. A node that is disconnected is still part of the cluster until it is dropped. Once dropped the cluster no longer considers it part of the cluster. This distinction matter when it comes to load balanced connections that use a load balance strategy other then Round Robin. Load balance connections use NiFi Site-To-Site protocol to move FlowFiles between nodes. Only Connected nodes are eligible to have FlowFiles sent over Site-To-Site. Even a disconnected node is still able to load-balance FlowFiles to other nodes still connected in the cluster. (so when you had 1 node disconnect from cluster, if you went to that nodes UI directly the load balanced connection would appear to processing all FlowFiles normally. This is because the two nodes where it sends some FlowFiles by attribute are still connected and thus it is allowed to send to them. The other FlowFiles by attribute destined for disconnected node never leave the node and get processed locally. Over on the still cluster connected nodes the story is different. They can only send to connected nodes and any FlowFiles destined for that disconnected node will begin to queue. Even if you stopped the dataflows on the disconnected node the FlowFiles would continue to queue for that node. So stopping the dataflow on a node that disconnects would still present same issue. A disconnected node is still aware of what node are part of the cluster and can still communicate with ZK to know which node is the elected cluster coordinator. Lets say a second node disconnects. The disconnected node would stop attempting to send to that now disconnected node and queue FlowFiles destined for that node. Only the round robin strategy will attempt to redistribute FlowFile to remaining connected nodes when a node becomes disconnected. The Partition by attribute and Single Node strategies are used when it is important that "like" FlowFiles end up on the same node for downstream processing (So once a like FlowFile, which in your case are FlowFiles with same value in the orderid FlowFile attribute, is marked for node 3, all FlowFiles with that same orderId will queue for node 3 as long as node three is still a member of the cluster. A disconnected node is still part of the cluster and will have some "like" FlowFiles already on it, so we would not want NiFi to start sending "Like" data to some other node all of a sudden. Now if manual user action was taken to drop the disconnected node, then the load-balanced connections would start using a different node for the FlowFiles original being allocated to the disconnected node. NiFi also offers an off-loading feature. This allows a user with proper authorization to off-load a disconnected node (IMPORTANT: only a reachable and running node can be offloaded successfully. Attempting Offloading of a down or unreachable node will not work). Once a node is disconnected a user can choose to offload the node this is typical if say a user want to decommission a node in the cluster. Initiating off-load will send a request to that disconnected node to stop, terminate all running components and then off load the queued FlowFiles to other nodes connected to the cluster. If cluster nodes where allowed to continue to load-balance to disconnected node(s), this capability would fail as you would end-up with a constant loop of FlowFiles back to disconnected node. Once offloading completes that disconnected node could be dropped and the FlowFiles that were offload would get load balanced to remaining nodes still members of the cluster. I think I covered all the basic behind the scenes functionality of load-balanced connection with regards to disconnected node behaviors. In your scenario, your node, the node became disconnected due to some issue when changing the version of a version controlled process group. I would recommend a new community question if you need help with that issue as it has no direct relationship with how load-balance connection function or disconnected nodes still running discussed here. Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
03-27-2025
03:22 PM
Thanks @Shelton for details!! I will try Option 3 in our pipeline shell script. Will let you know if any further issues.
... View more
02-28-2025
02:03 PM
Thank you @MattWho for details. As you mentioned, I will post my usecase in the jira. Thanks for your help!
... View more
11-06-2024
02:11 PM
@CommanderLaus Gitlab integration is available in latest version Nifi 2.0 (released few days ago). I was able to import and export flows directly from Gitlab repo. Note: Until Nifi 2.0M4, the file extension is "snapshot", but in 2.0 if you use Gitlab, flow config file extension should be ".json" We need to provide access token of the gitlab repository in Registry controller settings.
... View more
11-04-2024
06:09 AM
2 Kudos
@shiva239 You can create an apache "NiFi" jira in the Apache community to highlight this new feature and request modification to existing processor to support it. https://issues.apache.org/jira/ The more detail you provide in your jira the better chance someone might take this on in the community. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
09-09-2024
03:47 PM
Thanks @SAMSAL for the clarification and the alternatives for looping in Nifi. We will consider using DuplicateFlowFile processor. Do you think nifi documentation should be updated to explicitly mention that it is not only for load tests but also can be used in production flows when there is need to clone flow files ?
... View more
09-04-2024
02:51 PM
@AlokKumar I am not able to understand the context here. It would help to get to right solution if you can explain why the inserts into same table should be executed in certain order. However, Nifi Connection configuration settings have prioritizers. Please check screenshot below.. If you are able to set attribute "priority" with desired number, "PriorityAttributePrioritizer" can be used. Please note that these prioritizers operate on node level. You may have to test these if you have multi node nifi environment. There is a processor "EnforceOrder" which has more options to enforce execution order. You may check this processor and find out if it works. https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/nifi2.0-m4/org.apache.nifi.processors.standard.EnforceOrder/index.html
... View more
08-16-2024
08:43 AM
@Adyant001 Oracle has auto increment IDENTITY feature as below. Table definition takes care of setting primary key incremental value. No need to send this value in json payload I think all other major RDBMS have similar feature to assign auto increment value. CREATE TABLE roletab (id NUMBER GENERATED BY DEFAULT ON NULL AS IDENTITY (START WITH 1 INCREMENT BY 1 NOCYCLE), role_id NUMBER, role_name VARCHAR(100), PRIMARY KEY (id) ); INSERT INTO roletab (role_id, role_name) VALUES (10, 'Admin'); INSERT INTO roletab (role_id, role_name) VALUES (20, 'Developer'); commit; SELECT * FROM roletab; ID |ROLE_ID|ROLE_NAME| --+-------+---------+ 1 | 10 | Admin | 2 | 20 |Developer|
... View more
08-14-2024
01:31 PM
1 Kudo
@Former Member LookupRecord processor can be used in this case. Please check if this method works for you. Sample CSV input id,brand,phone 10,Samsung,1 20,Apple,2 Expected output (codes 1 and 2 are replaced with names) id,brand,phone 10,Samsung,Note 20,Apple,iPhone I tried nifi flow as below Lookup record configuration CSV Reader CSV Recordset Writer Didnt make any change. Used default settings Simple KeyValue Lookup service. There are several lookup services available in Nifi. You may use right service as per your requirement.
... View more