I have an interesting
issue with Nifi concurrency. I have a dataflow with two connected ExecuteStreamCommand
processors (each with concurrent tasks=1), but when I set the number of
threads of the whole instance to 1, the two processors still manage to
somehow run concurrently although I expect them to run sequentially. Each processor makes a call to python code as part of its execution.
The first processor takes on average 2.5 seconds per flowfile and the
second processors takes on average 4.5 seconds per flowfile.. I gave the
dataflow 100 flowlfiles and I was expecting all flowfiles will finish
in around 700 seconds (i.e., sequential execution) but they manage
to finish in 480 seconds which suggests that each processor is using a
separate thread and they do not wait on each other. Am I missing
something here ? Note: The command that I run in the ExecuteStreamCommand processor is a call to python code that does busy wait. For example, the following is a snippet for busy wait for 2.5 seconds: import time
current_time = time.time()
while (time.time() < current_time+sleep_time):
pass
... View more
