Support Questions
Find answers, ask questions, and share your expertise

Nifi Cannot swap in FlowFiles from location

New Contributor
107641-nifiqueueswapthreshold.png


107651-swap.png


When 43,508 flow files are sent to SplitJson, they are stacked in the queue and processed within the value set in nifi.queue.swap.threshold(= 20000).

After that, a swap file was created and there are 23,508 stacks in the queue

but it is not running any more and is stopped.


nifi-app.log seems to be in progress because of an error in swap in.

2019-04-05 16:31:59,534 WARN [Timer-Driven Process Thread-6] o.a.n.controller.FileSystemSwapManager Cannot swap in FlowFiles from location D:\download\NIFI-1~1.0-B\NIFI-1~1.0\.\flowfile_repository\swap\1554449490032-01691059-3464-1fc9-0a1a-9def8b12c89e-15293fe8-c2ea-4f9d-a4fb-fd992716903d.swap because the FlowFile Repository does not know about this Swap Location. This file should be manually removed. This typically occurs when a Swap File is written but the FlowFile Repository is not updated yet to reflect this. This is generally not a cause for concern, but may be indicative of a failure to update the FlowFile Repository.


Changing nifi.queue.swap.threshold = 40,000 will process up to 40,000 and stop equally.



Thank you for your help.

5 REPLIES 5

Explorer

We have the same issue, has anyone found any solution to this case?

New Contributor

This seems to happen since 1.9, I reverted back to 1.8.0 and it runs again.

Before I did this, I turned off the caching for any repo in the nifi.properties. Had no effect on this issue, only made Nifi extremely slow (as expected)

New Contributor

Breakdown down your problem into layers.after splitjson try ingesting into Kafka/HDFS then you understand why its not swapping.


Explorer

After read the content of the link below a made some changes in both nifi.properties e boostrap.conf files. That changes solved the issue, for now.

https://community.hortonworks.com/articles/7882/hdfnifi-best-practices-for-setting-up-a-high-perfo.h...


nifi.properties file:

<nifi.queue.swap.threshold=100000/>


Bootstrap.conf file:

<# JVM memory settings

java.arg.2=-Xms4g

java.arg.3=-Xmx4g/>


coment Garbage collector usage

<# java.arg.13=-XX:+UseG1GC/>


Master Guru

@dhhan74 

 

The attached screenshot shows your running version as Apache NiFi 1.9.0.
There is a known issue in that released that results in the ERROR condition you have encountered:

https://issues.apache.org/jira/browse/NIFI-6285

 

This issue was addressed as of Apache NiFi 1.10.

 

Hope this helps,

Matt

; ;