- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
Nifi consume duplicate message problem from Kafka
- Labels:
-
Apache NiFi
Created ‎01-22-2024 11:48 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Nifi version is 1.23.2 and kafka version 2.6.I use ConsumeKafka_1_0 1.23.2 to read messages from kafka.DetectDuplicate 1.23.2 processor is available, which deduplicates stream files on the same stream.However, stream read the same recording twice and DetectDuplicate processor could not deduplicate the same message.I understand that it reads the same message because the offset id value is same in two message.What could be the reason why the ConsumeKafka_1_0 processor reads the same message and DetectDuplicate processor cannot detect same message?
Created ‎02-15-2024 03:20 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I forgot to answer the question.If flowfile drop on failure and retry transition,I rotates flowFile same putElasticsearchHttp processor.

- « Previous
-
- 1
- 2
- Next »