@plapla  The consumeKafka processor should not be reading the same message twice.  The processor should be maintaining local state (since you are not clustered) in NiFi's local state directory. Make sure that you are not having disk space or permissions issues that may prevent the writing of that local state.  You can write click on the ConsumeKafka processor to view the current stored state. The consumeKafka processor creates a consumer group using the GroupID configured in the processor, so make sure you do not have multiple consumeKafka processor consuming from the same Kafka Topic using the same Group ID.  For optimal performance the number of concurrent tasks configured on the consumeKafka processor should match the number fo partitions on target topic. Do you see any Kafka rebalance going on?   Will happen when you have more consumers than partitions in a the consumer group that is consuming from that topic.  A rebalance can affect the commit of the offset resulting in possible data duplication. Thanks, Matt ... View more

@PriyankaMondal  Just to add to What @ckumar provided, the NiFi repositories are not locked to the specific node.  What i mean by that is that they can be moved to a new node, withe "new" being the key word there.  A typical prod NiFi setup will use protected storage for its flowfile_repository and content_repository(s) which hold all the FlowFile metadata and FlowFile content for all actively queued and archived FlowFiles on a node.  To prevent loss of data, these repositories should be protected through the use of RAID storage or some other equivalent protected storage.  The data stored in these repositories is tightly coupled to the flow.xml.gz/flow.json.gz that cluster is running on every node.  Let's say you have hardware failure, it may be faster to standup a new server then repair the existing hardware failure.  You can simple move or copy the protected repositories to the new node before starting it.  When the node starts and joins your existing cluster it will inherit the cluster's flow.xml.gz/flow.json.gz and then begin loading the FlowFiles from those moved repositories in to the connection queues.  Processing will continue exactly where it left off on the old node.  There is no way to merge repositories together, so you can not add the contents of one nodes repositories to the already existing repositories of another node. The provenance_repository holds lineage data, and the database_repository holds flow configuration history and some node specific info.  Neither of these are needed to preserve the actual FlowFiles. Hope this helps, Matt ... View more

@plapla  Since Apache NiFi does not have a ConsumeKafka processor build with the Kafka 2.6 client, i would recommend going with the client closest to but not newer than the Kafka server version you are using.  In this case the ConsumeKafka_2_0 processor. 1. Is your NiFi a standalone NiFi instance install or a multi-node NiFi cluster setup? 2. How are your distributedMapCacheClient and DistributedMapCacheServer controller services configured? 3. What is the rate of FlowFiles being produced by your consumeKafka processor? I see that you configured the DetectDuplicate processor with an age off of 420 days; however, DistributedMapCache server has a configured max cache entries with a default of 10,000.  So possibly due to volume cache entries are being removed from the cache server resulting in issues detecting duplicates.  The DistributedMapCache also holds all cache entries in NiFi heap memory and is not a good cache server to use in high volume caches (because of heap usage).  DistributedMapCache also offers no high availability.    This is becomes even more of an issue with a NiFi cluster.  You would be better off using an external map cache server. If you are using a NiFi cluster, make sure your DistributedMapCacheClient is configured to connect to one specific DistirbutedMapCacheServer.  I have seen misuse here where individuals configured it to connect to local host or each NiFi node to its own host's map cache server.  The Map cache servers running on each node do not share data. Hope this helps you... If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@sukanta  I am not completely clear on what you are asking for here.  Can you provide more detail? What do you mean by "custom error page in NiFi"? Are you referring to the NiFi bulletin Board? Only users who are authorized for the component producing the bulletin message woudl be able to view the bulletin text via the component or bulletin board. Thanks, Matt   ... View more

@plapla  Couple things here: 1. You should be using the ConsumeKafka processor that matches your Kafka server version.  If your KafKa server is 2.6, you should be using the ConsumeKafka_2_6 processor. 2. In your detectDuplicate processor you are using ${key}.  How is this Attribute beig created on each FlowFile and where is it's value derived from? Thanks, Matt ... View more

Big shout out to all the amazing contributors to this community both via question solution assistance and valuable articles!!!   Together we enable great people to accomplish great things. ... View more

@Sartha  You are still not using the correct URL in your postHTTP as I stated in last response: It needs to be based on what you shared in yoru last response: http://10.73.121.84:5026/contentListner  The ListenHTTP processor defines the "Base Path" on which it is listening for inbound connections.  The default value is "contentListener".  So the PostHTTP needs to post to that Base Path. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more