NiFi Wait/Notify flow not working as expected

[#1]

We're using a Wait/Notify pattern in one of our flows to overcome a race condition problem (object x is referencing object y, object x will wait until object y is created. Creation of object y will generate the notification).

This basically works for most of our flow files. Files are waiting in queue and are released when notified as expected. However, for some flow files the notify- signal for object y is sent but object x will still be stuck in waiting state, not reacting on it.

I have enabled persistence on the cache and verified that the notify- signals are really added to it. I have juggled the different parameters of the wait/notify- processors without any success (it should require minimal configuration though since we only have a 1 to 1 relation between wait and notify signals).

Any ideas?

We're using NiFi 1.6.0.

[#2]

I created a script to get all the keys (using DistributedCacheMapService) from the cache and it seems that the keys we're waiting for are actually not in it. However, looking at the snapshot file for the cache on disk it seems that the keys are there.

Not sure about the life cycle for cached entries. When are they in the journal file and when are they in the snapshot file?

[#3]

Another observation. We're running a small cluster with two NiFi instances. Depending on which instance that runs I get different results from the cache extraction explained above (the DistributedCacheMapService points at cache at localhost). Is the cache bound to each NiFi instance? I expected it to be distributed?