Member since
10-21-2022
3
Posts
0
Kudos Received
0
Solutions
12-19-2022
03:49 AM
Hello All, I have encountered a rather weird bug with NiFi's SplitJson processor. I am using NiFi 1.18 and it's a three node cluster. I think the best way of explaining it will be to use as example, so here is how to recreate this bug: I created this flow with the following sequence of steps: Create a GenerateFlowFile and set the custom text to: { "test": [ { "A": 1 } ], "test2": [ { "B": 1 } ] } Create the top SplitJson and set the path to: $.test.* Create all connections for the top SplitJson (original is terminated), start it and run GenerateFlowFile once -> The output queue should show a single flow file with this content: { "A": 1 } Now select the top SplitJson, Crtl+C, Crtl+V, move the copy down create its connections and set the path to: $.test2.* Start the bottom SplitJson, and run GenerateFlowFile once again Both output queues have a new flow file and the contents of both flow files are: { "A": 1 }, which is correct for the top SplitJson, but for the bottom one the value should be { "B": 1 } It seems that, for some reason, the copied SplitJson continues operating using the path that it had at the moment of copying and not the one that is present in the settings of the processor. I've checked that after refreshing the view or even the page, the settings of the bottom SplitJson show $.test2.* (so it is not a issue of my browser client not conveying the change to the cluster successfully, as the value stays) and yet the output of the copied SplitJson continuous to be { "A": 1 } instead of { "B": 1 }. Is this a known issue? If so is there a fix for it? Perhaps someone could quickly check if this issue is still present on 1.19, we could update our cluster if it was fixed. Best regards, Aleksander
... View more
Labels:
- Labels:
-
Apache NiFi
11-24-2022
02:14 AM
Hello everyone, Recently, we've decided to try out the load balancing feature in our NiFi 1.15 cluster and we've run into the following issue. 16 flow files got stuck in a queue that had load balancing switched on. On the connection, I can see that there are 16 flow files and about 70 MB of data. However, when I try to list that queue, it says that it's empty and when I try to empty it, it says that 0 out of 16 flow files were removed. What's more, event after switching the load balancing feature off everywhere, it still continuously prints the following two errors in the log: Failed to communicate with Peer node001:8443 when load balancing data for Connection with ID a3f7a255-0184-1000-ffff-fffffe4ef3af due to java.io.EOFException: Encountered End-of-File when trying to verify with Peer node001:8443 whether or not space is available in Connection a3f7a255-0184-1000-ffff-fffffe4ef3af Failed to receive FlowFiles for Load Balancing due to org.apache.nifi.controller.queue.clustered.server.TransactionAbortedException: Attempted to receive FlowFiles from Peer node001 for Connection with ID a3f7a255-0184-1000-ffff-fffffe4ef3af but no Connection exists with that ID At this point, we have decided that we are not going to use load balancing until the cluster gets updated to 1.17, however, we would very much like to get rid of those problematic flow files and continuous error messages. Is this a know issue?
... View more
Labels:
- Labels:
-
Apache NiFi
10-21-2022
02:09 AM
The issue was not related to the size of the queue, but instead to the fact that when to destination that we are sending those logs to had connectivity issues (which was causing quite a lot of timeout errors), NiFi was using a lot of a significant number of threads to retry sending the logs to the InvokeHTTP destination. This high usage of threads was causing ListenSyslog to not function properly - it was dropping the logs that it was supposed to receive.
... View more