NIFI 1.18.0 is not cleaning flow file from queue

[#1]

HI ,

Hi,
I was using NIFI 1.12.0 version and now shiting a job into NIFI 1.18.0 version because of some security concerns. I am facing challenges in that the NIFI queue is not getting empty after successfully data transfer to EventHub. Below is the NIFI 

[#2]

@MKothari, Welcome to our community! To help you get the best possible answer, I have tagged in our NiFi experts @cotopaul @SAMSAL @steven-matison @ckumar @MattWho  who may be able to assist you further.

Please feel free to provide any additional information or details about your query, and we hope that you will find a satisfactory solution to your question.

[#3]

Thanks Viidya

[#4]

hi @MKothari,

Are you certain that your messages get transferred successfully? Based on your screenshot, I can see that no FlowFile got IN in the past 5 mins, meaning that your flow is not running, even though everything is green and you have no errors in your GUI.
I assume that the flowfile format matches whatever you have configured in your kafka. Have a look within PublishKafka and see if you do not have a strange scheduling set, causing you to see the number increasing instead of decreasing. Another thing you might try is stop both EvaluateJsonPath and PublishKafka and right click on your matched queue and open List Queue and see if there are any flowfile there and if so, on which node. Next, you can go on that node and further check the log file for possible error messages.

[#5]

Thanks ....let me validate the proposed one and update

[#6]

Hi Cotopauul,

I verified but did not saw any error inside the the log.Please find below for your reference.

1. node2 log

ponse took 2 millis; receiving full response took 2 millis; total time was 96 millis
2023-04-24 05:38:15,287 INFO [Clustering Tasks Thread-2] o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2023-04-24 05:38:15,189 and
sent to dafct-con1-d-euwe-cdp-nifi-vm3.r2k.ct2.atos.net:13443 at 2023-04-24 05:38:15,287; determining Cluster Coordinator took 2 millis; DNS
lookup for coordinator took 0 millis; connecting to coordinator took 2 millis; sending heartbeat took 90 millis; receiving first byte from res
ponse took 3 millis; receiving full response took 3 millis; total time was 97 millis
2023-04-24 05:38:20,387 INFO [Clustering Tasks Thread-2] o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2023-04-24 05:38:20,287 and
sent to dafct-con1-d-euwe-cdp-nifi-vm3.r2k.ct2.atos.net:13443 at 2023-04-24 05:38:20,387; determining Cluster Coordinator took 3 millis; DNS
lookup for coordinator took 0 millis; connecting to coordinator took 2 millis; sending heartbeat took 91 millis; receiving first byte from res
ponse took 2 millis; receiving full response took 3 millis; total time was 99 millis
2023-04-24 05:38:25,490 INFO [Clustering Tasks Thread-2] o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2023-04-24 05:38:25,388 and
sent to dafct-con1-d-euwe-cdp-nifi-vm3.r2k.ct2.atos.net:13443 at 2023-04-24 05:38:25,490; determining Cluster Coordinator took 2 millis; DNS
lookup for coordinator took 0 millis; connecting to coordinator took 3 millis; sending heartbeat took 94 millis; receiving first byte from res
ponse took 2 millis; receiving full response took 3 millis; total time was 102 millis
2023-04-24 05:38:27,839 INFO [pool-10-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile Repository
2023-04-24 05:38:27,839 INFO [pool-10-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with 1 re
cords in 0 milliseconds
2023-04-24 05:38:30,598 INFO [Clustering Tasks Thread-2] o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2023-04-24 05:38:30,491 and
sent to dafct-con1-d-euwe-cdp-nifi-vm3.r2k.ct2.atos.net:13443 at 2023-04-24 05:38:30,598; determining Cluster Coordinator took 2 millis; DNS
lookup for coordinator took 6 millis; connecting to coordinator took 2 millis; sending heartbeat took 93 millis; receiving first byte from res
ponse took 2 millis; receiving full response took 3 millis; total time was 106 millis
2023-04-24 05:38:35,695 INFO [Clustering Tasks Thread-2] o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2023-04-24 05:38:35,598 and
sent to dafct-con1-d-euwe-cdp-nifi-vm3.r2k.ct2.atos.net:13443 at 2023-04-24 05:38:35,695; determining Cluster Coordinator took 2 millis; DNS
lookup for coordinator took 0 millis; connecting to coordinator took 2 millis; sending heartbeat took 90 millis; receiving first byte from res
ponse took 2 millis; receiving full response took 2 millis; total time was 96 millis
2023-04-24 05:38:39,684 INFO [Cleanup Archive for default] o.a.n.c.repository.FileSystemRepository Successfully deleted 0 files (0 bytes) from
archive
2023-04-24 05:38:39,685 INFO [Cleanup Archive for default] o.a.n.c.repository.FileSystemRepository Archive cleanup completed for container def
ault; will now allow writing to this container. Bytes used = 14.14 GB, bytes free = 14.87 GB, capacity = 29.01 GB
2023-04-24 05:38:40,795 INFO [Clustering Tasks Thread-2] o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2023-04-24 05:38:40,695 and
sent to dafct-con1-d-euwe-cdp-nifi-vm3.r2k.ct2.atos.net:13443 at 2023-04-24 05:38:40,795; determining Cluster Coordinator took 4 millis; DNS
lookup for coordinator took 0 millis; connecting to coordinator took 1 millis; sending heartbeat took 91 millis; receiving first byte from res
ponse took 2 millis; receiving full response took 3 millis; total time was 100 millis

node3 log

2023-04-24 05:33:39,931 INFO [Process Cluster Protocol Request-45] o.a.n.c.p.impl.SocketProtocolListener Finished processing request dd33a8e7-
de3c-4894-9497-632b6cfba73f (type=HEARTBEAT, length=4615 bytes) from dafct-con1-d-euwe-cdp-nifi-vm2.r2k.ct2.atos.net:9090 in 93 millis
2023-04-24 05:33:40,208 INFO [Heartbeat Monitor Thread-1] o.a.n.c.c.h.AbstractHeartbeatMonitor Finished processing 3 heartbeats in 3500 nanos
2023-04-24 05:33:40,931 INFO [Process Cluster Protocol Request-5] o.a.n.c.p.impl.SocketProtocolListener Finished processing request b5ecae2f-c
ffa-47d9-ac92-b9782a25eb48 (type=HEARTBEAT, length=4619 bytes) from dafct-con1-d-euwe-cdp-nifi-vm3.r2k.ct2.atos.net:9090 in 89 millis
2023-04-24 05:33:40,934 INFO [Clustering Tasks Thread-2] o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2023-04-24 05:33:40,836 and
sent to dafct-con1-d-euwe-cdp-nifi-vm3.r2k.ct2.atos.net:13443 at 2023-04-24 05:33:40,934; determining Cluster Coordinator took 1 millis; DNS
lookup for coordinator took 3 millis; connecting to coordinator took 1 millis; sending heartbeat took 87 millis; receiving first byte from res
ponse took 2 millis; receiving full response took 4 millis; total time was 98 millis
2023-04-24 05:33:41,888 INFO [Cleanup Archive for default] o.a.n.c.repository.FileSystemRepository Successfully deleted 0 files (0 bytes) from
archive
2023-04-24 05:33:41,888 INFO [Cleanup Archive for default] o.a.n.c.repository.FileSystemRepository Archive cleanup completed for container def
ault; will now allow writing to this container. Bytes used = 13.05 GB, bytes free = 15.95 GB, capacity = 29.01 GB
2023-04-24 05:33:43,656 INFO [Process Cluster Protocol Request-9] o.a.n.c.p.impl.SocketProtocolListener Finished processing request 297e9082-f
9f8-4798-ab4d-cbdc2cbc52e0 (type=HEARTBEAT, length=4611 bytes) from dafct-con1-d-euwe-cdp-nifi-vm1.r2k.ct2.atos.net:9090 in 93 millis
2023-04-24 05:33:45,028 INFO [Process Cluster Protocol Request-26] o.a.n.c.p.impl.SocketProtocolListener Finished processing request 345f6585-
fe2b-4aaf-8c28-2768f652887d (type=HEARTBEAT, length=4615 bytes) from dafct-con1-d-euwe-cdp-nifi-vm2.r2k.ct2.atos.net:9090 in 91 millis
2023-04-24 05:33:45,208 INFO [Heartbeat Monitor Thread-1] o.a.n.c.c.h.AbstractHeartbeatMonitor Finished processing 3 heartbeats in 5800 nanos
2023-04-24 05:33:46,026 INFO [Process Cluster Protocol Request-2] o.a.n.c.p.impl.SocketProtocolListener Finished processing request 334b594d-8
0e9-4a98-8e2d-90ecb3adfa7c (type=HEARTBEAT, length=4619 bytes) from dafct-con1-d-euwe-cdp-nifi-vm3.r2k.ct2.atos.net:9090 in 88 millis
2023-04-24 05:33:46,028 INFO [Clustering Tasks Thread-2] o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2023-04-24 05:33:45,934 and
sent to dafct-con1-d-euwe-cdp-nifi-vm3.r2k.ct2.atos.net:13443 at 2023-04-24 05:33:46,028; determining Cluster Coordinator took 2 millis; DNS
lookup for coordinator took 0 millis; connecting to coordinator took 1 millis; sending heartbeat took 87 millis; receiving first byte from res
ponse took 2 millis; receiving full response took 4 millis; total time was 94 millis
2023-04-24 05:33:48,757 INFO [Process Cluster Protocol Request-37] o.a.n.c.p.impl.SocketProtocolListener Finished processing request be474378-
37fb-41a7-9be8-619c56ccfe47 (type=HEARTBEAT, length=4611 bytes) from dafct-con1-d-euwe-cdp-nifi-vm1.r2k.ct2.atos.net:9090 in 91 millis

[#7 · Accepted solution]

There was some issue with PublishKafka_2.6.After changing to PublishKafka_2.0 problem got resolved.