I just re-run above case again. This time, I found that for those duplicated topics., exception is found in Kafka logs: ERROR [Replica Manager on Broker 0]: Error processing append operation on partition MY_TOPIC-0 (kafka.server.ReplicaManager) org.apache.kafka.common.errors.ProducerFencedException: Producer's epoch is no longer valid. There is probably another producer with a newer epoch. 0 (request epoch), 1 (server epoch) Do I configure anything wrong in my Kafka server? ... View more

The above case is run on a standalone nifi. I can confirm that it is not twice the amount, as I set failure queue to be 'auto terminated' ... View more

@Bryan Bende My flow is simple: ListFile > FetchFile > PublishKafka_0_11 (with transaction=true, message demarcator=\n) To test whether it is duplicate, I : 1. Check offset of the topic bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:49092 --topic MY_TOPIC --time -1 2. Run kafka-console-consumer.sh with param "--from-beginning" and compare the result with the source file. My source file contains 9.26 million rows. Using method 1, I got offset=18460643, far more than row count. Using method 2, I got ~18million messages consumed. p.s. The topic is newly created when running the flow. There's no error occurred on the flow that I found duplicate. But in general, I occasionally encounter ProducerFencedException from kafka when using the PublishKafka producer. FYI, from consumer side, if I use consumeKafka processor with 'Honor Transaction=true', I cannot get any flowfile. Only if 'Honor Transaction=false', I can get that 18 million duplicated messages. ... View more

Thanks all. One more question: How does PutParquet Processor connect to HDFS? By WebHDFS , HttpFS or Native ? ... View more

@Shu Thanks for the reply, however using UpdateAttribute processor will have multiple parquet files output. What I want to achieve is: N files under directory > 1 .parquet file ... View more