Hi @rmtonkin, I think I understood the problem and the solution is currently working for me pretty well. The culprit property here is Scroll Duration in the processor ScrollElasticSearchHttp. The property default is 1m(minute) and by definition it means How long each search context is kept in memory (of ES itself). In my configuration, Page Size is 100, means 100 records per flow file. Problem(for example): 1. For some reason, fetching a page(*current cycle) took more than 1 minute. 2. 1 minute passed means, since Scroll Duration is set to 1 minute, ES removed that ScrollId(in View State) from memory and no longer recognises it. 3. Nifi processor uses the ScrollId to fetch the next page after completing the *current cycle. 4. Gets 404 because there's no such ScrollId in ES memory. Solution(working for me): Increase the value of Scroll Duration to 5m and hope each cycle completes within 5 mins(which IMO is a pretty good time). NOTE: Don't increase the value much or it might be a huge pressure on the ES and might stall it(haven't tried, just a theory). Hope this solves your problem too. You can try and let me know. Best of luck. For all others, please understand this is NOT a solution to the original question. This is about how to avoid the scenario mentioned in it. Cheers.
... View more
