Member since
07-30-2019
3406
Posts
1622
Kudos Received
1008
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 188 | 12-17-2025 05:55 AM | |
| 249 | 12-15-2025 01:29 PM | |
| 183 | 12-15-2025 06:50 AM | |
| 279 | 12-05-2025 08:25 AM | |
| 467 | 12-03-2025 10:21 AM |
05-20-2021
06:17 AM
@Kilynn The following property in the nifi.properties file controls when a swap file is created per connection. nifi.queue.swap.threshold=20000 This is per connection and not for all FlowFiles across all connections. A FlowFile swap file will always consist of 10000 FlowFiles. So if a connection reaches 20000 queued FlowFiles, a swap file will be created for 10000 of those. So if a connection queue reaches 40000, you would have 3 swap files of that connection. You can control individual connection queues by setting the "Back pressure Object Threshold" on a connection: Note: Threshold settings are soft limits And default for object threshold is 10000. So with these settings there should be very little to no swapping of FlowFiles to disk happening at all. Swap files would only happen if source processor to that connection output enough FlowFiles to connection at one time to trigger a swap file. For example: - Connection has 9000 queued FlowFiles, so back pressure is not being applied. - Source processor is thus allowed to execute - Source processor upon execution produces 12000 FlowFiles - now downstream connection has 21000 queued FlowFiles. One swap file is produced and back pressure is enabled until queue drops back below 10000 queued FlowFiles. FlowFiles consist of two parts (FlowFile attributes/metadata and FlowFileContent). The only portion of a FlowFile held in heap memory is the FlowFile attributes/Metadata. FlowFile content is never held in memory (Some processors may load content in to memory in order to perform their function only). FlowFile attributes/metadata is persisted to the flowfile repository and FlowFile content is written to the content repository. This important to avoid data loss if NiFi dies or is restarted while data still exists in connection queues. If you found this helped with your query, please take a moment to login and click accept in the solutions that helped. Thank you, Matt
... View more
05-17-2021
05:52 AM
@leandrolinof The exception indicates "unable to unmarshal JSON to an object" which is commonly thrown when there is an issue with the source FlowFile content. For examples some unescaped Unicode characters like whitespace. I'd inspect you source content. I see in your dataflow that you have a convertAvrotoJson processor that took in a 2.08 KB content source FlowFile and output only a 2 byte content FlowFile. Then later a ReplaceText processor that takes that 2 byte content and output 78 bytes. So you'll want to inspect the content output there just before your JoltTransformJson processor. NiFi processor bulletins which you have shared will not return stack traces that may be produced by processor. Always best to check the nifi-app.log when you see a bulletin to see if maybe there is additional log output that can be helpful as well. Hope this helps, Matt
... View more
05-17-2021
05:34 AM
@rahul_ars You can use the "UpdateCounter" [1] processor to create counters and update the count on them (up and down). The "Counter Name" property accepts NiFi EL, so you can use attribute son FlowFiles to dynamically update a counter uniquely by FlowFile traversing same dataflow path. This will allow you to consolidate within your dataflow(s) to reduce the number of processors needed. Fewer processors can lead to better performance through reduced resource utilization. The "Counters" UI is found under the NiFi Global menu in the upper right corner of the NiFi UI. [1] http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.13.2/org.apache.nifi.processors.standard.UpdateCounter/index.html If you found this help with your query, please take a moment to login and click accept on this solution. Thanks, Matt
... View more
05-17-2021
05:06 AM
@techNerd I don't know anything about the Wildfly endpoint service, so I can only assume your URL is correct. Using the IP as you have should not be an issue as long as the NiFi host can reach that network address. What you are seeing is a timeout which indicates the endpoint you are trying to reach did not respond to you post request. There could be numerous reasons for this such as network issues, incorrect or missing headers on post request, bad endpoint URL, too many concurrent connections to the endpoint at time of this request, failed authentication, using http for a https endpoint, bad or no client authentication, etc... I would suggest you monitor the logs on Wildfly when the post request is made by NiFi to see if: 1. Wildfly acknowledges receiving the request. 2. Wildfly does not throw and exception about the request. Since the endpoint URL you shared had multipart in it, I assumed that perhaps it only accepts multi-part form data and thus may be expecting the proper header fo this type of data. And if it is multi-part form data, you are going to have better success using the postHTTP processor instead of the InvokeHTTP processor. Hope this helps you, Matt
... View more
05-14-2021
11:12 AM
@Kilynn Can you provide more detail around "nodes randomly restarting"? What do you see in the nifi-app.log and nifi-bootstrap.log when the node(s) restart? Do you see anything in /var/log/messages? Maybe OOM Killer? I see Max heap shows 330 GB configured. With 12 nodes that is ~27GB set per node. Any particular reason why you set heap so high for your NiFi nodes? Larger heaps mean longer stop-the-world GC pauses. What version of Java are you using? Thanks, Matt
... View more
05-14-2021
06:53 AM
@techNerd I am not sure how large your multi-part form data is that your are trying to post to you endpoint is, but your timeout values are very aggressive at 5 sec and 15 sec. Have you tried sending these values to more common 30 sec? Have you tried reaching this endpoint from command line on the host where NiFi is running? How is the rest of yoru processor configured? Try using the PostHTTP [1] processor instead. Make sure you set the "Content-Type" configuration property to match the content that is actually being sent. For example: Content-Type = multipart/form-data; boundary=<boundary id for multipart content> Replace boundary sting with whatever your source content uses in its multipart form data. [1] http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.13.2/org.apache.nifi.processors.standard.PostHTTP/index.html If you found this helped with yoru query, please take a moment to login and click accept on this solution. Thank you, Matt
... View more
05-14-2021
06:01 AM
@Seedy Sharing screenshots of the steps followed and observations seen would be helpful. So you have created a Process Group (PG) and created a new Parameter Context which is then assigned to that PG. Is this a nested PG? The blank canvas you see when NiFi is first accessed is the root level PG. Any PG you add to canvas is a nested/sub-PG to that root PG. You can also have PGs nested even deeper. As the user who is having issues, sharing screenshots may help in seeing what the issue may be. Please include screenshots showing following while authenticated as user having issues: 1. Actions attempting to be performed 2. NiFi Parameter contexts UI (found under the NiFi global menu in upper right corner --> Parameter Contexts) 3. On the PG on which the parameter context has been assigned, show the configuration for that PG including the Controller Services tab. Example: 4. Configuration of the assigned parameter context. click through each existing parameter in the context, do any show "unauthorized referencing components"? Thanks, Matt
... View more
05-13-2021
10:36 AM
@sangee The "RouteText" [1] processor is used to route individual lines within a multi-line content FlowFile to different relationships (essentially splitting up a multi-line FlowFile). What you are trying to accomplish can however be done using the "RouteOnContent" [2] processor. Your sample shows content with two lines and only one of those lines containing the string "Merging completed". So I assume the other FlowFIle's content will have only one line with "Merging complete"? If so, the "RouteOnContent" can be configured as follows: The dynamic property I added becomes a new relationship on the processor. So any FlowFile where content matches exactly "Merging complete" will be sent to this relationship while all other FlowFiles will be routed to the pre-existing "unmatched" relationship. [1] http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.13.2/org.apache.nifi.processors.standard.RouteText/index.html [2] http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.13.2/org.apache.nifi.processors.standard.RouteOnContent/index.html If you found this help with your query, please take a moment to login and click accept on this solution. Thank you, Matt
... View more
05-13-2021
08:33 AM
@ThangND-2210 Thank you for filing Apache Jira detailing your observations here: https://issues.apache.org/jira/browse/NIFI-8541 I see that Apache NiFi community committers are looking in to your Jira. Thank you, Matt
... View more
05-12-2021
12:02 PM
1 Kudo
@kkau Sorry to hear that the embedded documentation was not accurate for the port property. I filed an apache jira to address that issue: https://issues.apache.org/jira/browse/NIFI-8536 Best path forward is following NIFi recommended best practice where each NiFi node in your cluster is on a unique host. This would allow each node to bind this reporting task to the same port. Having multiple hosts on same server will present issue/challenges with any component (processor, controller service, reporting tasks, etc) that create a listener port. While some may you may be able to work around with Variable Registry suggestion I shared, others like this use case will not. Thanks, Matt
... View more