Member since
03-09-2018
2
Posts
3
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
3389 | 08-05-2020 06:09 AM |
08-05-2020
06:09 AM
2 Kudos
Hi SAMSAL, I believe we're hitting a similar issue here to this stackoverflow thread. As in, the third party utility that creates the csv files adds this non-breaking space character to the file that displays as a red dot. You mentioned that it's not trivial to change the input files but maybe this is a minor adjustment that the vendor can make.
... View more
07-29-2020
03:08 AM
Hi @rmtonkin, I think I understood the problem and the solution is currently working for me pretty well. The culprit property here is Scroll Duration in the processor ScrollElasticSearchHttp. The property default is 1m(minute) and by definition it means How long each search context is kept in memory (of ES itself). In my configuration, Page Size is 100, means 100 records per flow file. Problem(for example): 1. For some reason, fetching a page(*current cycle) took more than 1 minute. 2. 1 minute passed means, since Scroll Duration is set to 1 minute, ES removed that ScrollId(in View State) from memory and no longer recognises it. 3. Nifi processor uses the ScrollId to fetch the next page after completing the *current cycle. 4. Gets 404 because there's no such ScrollId in ES memory. Solution(working for me): Increase the value of Scroll Duration to 5m and hope each cycle completes within 5 mins(which IMO is a pretty good time). NOTE: Don't increase the value much or it might be a huge pressure on the ES and might stall it(haven't tried, just a theory). Hope this solves your problem too. You can try and let me know. Best of luck. For all others, please understand this is NOT a solution to the original question. This is about how to avoid the scenario mentioned in it. Cheers.
... View more