About MattWho

MattWho · ‎07-01-2022

@Brenigan Are you running your dataflow on a standalone NiFi install or a NiFi cluster install? If a multi node NiFi cluster, are all 200 FlowFiles on the same NiFi node? Does your partition_number start at 0? Do you see your FlowFiles getting routes to the overtook relationship after 10 minutes? Assuming all the following: 1. All FlowFiles are on same NiFi node 2. partition_number starts at "0" and "increments consistently by "1" 3. All FlowFiles have same filename 4. Wait relationship is route via a connection back to the EnforceOrder processor. You should be seeing: 1. All FlowFiles routed to the "wait" relationship until a FlowFile with attribute "partition_number" equal to "0" is processed which will result in that FlowFile routing to success. 2. Other FlowFiles meeting above 4 criteria will continue to loop through wait until "partition_number" attribute with value "1" is seen and routed to success. 3. If a FlowFile in incremental order is missing, all FlowFiles with a partition_number higher than the next expected integer will continue to route to wait relationship. 4. after the configured "wait timeout" any FlowFile that has been waiting this long will be routed to the "overtook" relationship. You can right click on a connection holding the FlowFiles and list the queue. From there you can select the "view details" icon to the far left to examine the FlowFiles current attributes. You should see a new attribute "EnforceOrder.expectedOrder" that contains the next expected integer value that the group this FlowFile belongs to is waiting for. You will also find your "partition_number" which will have the current integer for this FlowFile. If you have your FlowFiles distributed across multiple nodes in a NiFi cluster, you will need to get all FlowFiles with the same "group identifier" moved to the same NiFi node in order to enforce order (you can not enforce order across different nodes in a NiFi cluster). You can accomplish this by editing the connection feeding your enforceOrder processor and under settings select a "Load Balancing Strategy" of "Partition by Attribute" using the "filename" attribute that you are using as your group identifier in the Enforce Order processor. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt

MattWho · ‎06-30-2022

@pandav You can not offload a NiFi node that is down. Can you clarify what you mean by "down"? Was the NiFi service not running on the nodes you attempted to offload? The offload option from the cluster UI sends a request to the disconnected (not down) node to offload its queued FlowFiles to nodes still connected to the cluster. If your nodes are down, you'll need to start the service on those nodes again. On startup (assuming no issues), these nodes will rejoin your cluster. If you plan to decomission a node later, you can use the NiFi cluster UI to manually disconnect a node and then offload that nodes FlowFiles. Once the FlowFiles have been successfully offloaded, the node can be deleted from the cluster using the NiFi cluster UI. Note: restarting a node that has been dropped/deleted from the cluster will trigger that node to start heartbeating to the cluster and thus reconnect unless you edit the configuration of the node so it does not use the same zookeeper znode as the current cluster (nifi.zookeeper.root.node property in nifi.properties file). https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#basic-cluster-setup As far as your nodes going down on a configuration change, you'll want to inspect the NiFi logs for an exceptions or timeouts that may have occurred. Network issues, long Garbage Collection (GC) pauses, and resource congestion/exhaustion can lead to nodes not responding or receiving the replicated change request. As a result a node can get disconnected. In the scenarios like this if you are using the latest Apache NiFi release, those nodes should automatically reconnect. Upon reconnect, if the nodes flow does not match the cluster flow, the node will automatically take the clusters flow and join. In order release a flow mismatch would between connecting node and cluster flow, would require manual intervention (copying the flow.xml.gz from a node still in the cluster to the node not connecting). If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt

rafy · ‎06-29-2022

Thank you Sir for the reply. Yes i have looked into them. The issue i have is that my query statement has multiple join conditions. And also the table is incremental. So, someone proposed i should convert the query to a database table view, then i can now supply the view name to any of the processors you mentioned above. Please, do you have a better approach. Thank you sir.

rafy · ‎06-29-2022

Thank you so much Mr @MattWho for comprehensive explanation on what Mr @SAMSAL earlier proposed. I really learn alot & appreciate.

MattWho · ‎06-29-2022

@Tryfan You mention this file comes in daily. You also mention that this file arrives through a load-balancer so you don't know which node will receive it. This means you can't configure your source processor for "primary node" only execution as you have done in your shared sample flow with the ListFile. As Primary Node only, the elected primary node will be the only node that executes that processor. So if the source file lands on any other node, it would not get listed. You could handle this flow in the following manor: GetFile ---> (7 success relationships) PostHTTP or InvokeHTTP (7 of these with one configured for each node in your cluster cluster) ListenHTTP --> UpdateAttribute --> PutFile So in this flow, no matter which node receives your source file, the GetFile will consume it. It will then get cloned 6 times (7 copies then exist) with one copy of the FlowFile getting routed to 7 unique PostHttp processors. Each of these targets the ListenHTTP processor listening on each node in your cluster. That ListenHTTP processor will receive all 7 copies (one copy per node) of the original source file. Then use the UpdateAttribute to set your username and location info before the putFile which place each copy in the desired location on the source node. If you add or remove nodes from your cluster, you would need to modify this flow accordingly which is a major downside to such a design. Thus the best solution is still one where the source file is placed somewhere all nodes can retrieve it from so it scales automatically. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt

DianaTorres · ‎06-27-2022

@yagoaparecidoti Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. Thanks

araujo · ‎06-22-2022

@nada , Please check this solution: https://community.cloudera.com/t5/Community-Articles/Decompressing-nested-ZIP-files-in-NiFi/ta-p/346169 Cheers, André

arshahin · ‎06-22-2022

Dear @MattWho Thanks a lot for the response. I installed Java SDK 11 and that helps to run NiFi on my laptop. I also change the environmental variables. I add JAVA_HOME under user variables and defined it like %JAVA_HOME%\bin in the path and now it is working fine. For my desktop I did the same thing and I notice NI applications (LabView) installed on my desktop was also using port 8080, so I fixed that one as well. NiFi is working fine on my desktop as well. I also found the following link relevant and useful for any other community members getting the same error: https://stackoverflow.com/questions/69830860/nifi-will-run-but-wont-open-in-browser-possible-error-with-java All the best Alireza

MattWho · ‎06-21-2022

@ThongPham Sounds like you may making a lot of unnecessary rest-api calls that could impact your NiFi's overall performance. Have you maybe looked at using the SiteToSiteBulletingReportingTask? This reporting task will send a FlowFile to a remote input port upon execution of bulletin(s) are produced. That Remote Input Port could then be built into a dataflow that makes notifications via putEmail. So instead of constantly calling the rest-api to see if something happened in the last 5 minutes, the flow will simply send something out when it happens only. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt

EldarB · ‎06-16-2022

Just add this ports to firewalld

Online	Online
Last Visited	‎02-03-2026 06:48 PM

Member Since	‎07-30-2019 10:41 AM
Last Visited	‎02-03-2026 06:48 PM
Posts	3,434
Kudos received	1628

Cloudera Community

Re: Setting TTL per key when writing to redis

Re: Best Practice for configuring registry flows

Re: Nifi 2.7.2 Start Problem

Re: Error importing NiFi workflow template from ve...

Re: nifi 2.6 registry security scan results

Re: EnforceOrder processor doesn't work.

Re: Nifi balancing cause loss of data

Re: Data migration: Which processor to use?

Re: Can NIFI nodes access different records on a D...

Re: How to replicate a file in NiFi

Re: NIFI - Unknown user with identity 'user'. Cont...

Re: Decompress .json.gz inside folder in NIFI

Re: An unexpected error has occurred. Please check...

Re: Call REST API to access a secured NiFi cluster

Re: Nifi : Unable to elect cluster coordinator