question What is the difference between publishKafka and publishKafkaRecord processors in Apache NiFi in Support Questions

What is the difference between publishKafka and publishKafkaRecord processors in Apache NiFi

Rishi_DE — Tue, 01 Jun 2021 18:56:58 GMT

Re: What is the difference between publishKafka and publishKafkaRecord processors in Apache NiFi

aakulov — Tue, 01 Jun 2021 19:31:21 GMT

PublishKafka sends the entire flow file to Kafka as a single message. PublishKafkaRecord splits each "record" (e.g. row) in the flow file and sends each record as a message.

References:

1. PublishKafka: https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-kafka-0-9-nar/1.6.0/org.apache.nifi.processors.kafka.pubsub.PublishKafka/

2. PublishKafkaRecord: https://nifi.apache.org/docs/nifi-docs/components/nifi-docs/components/org.apache.nifi/nifi-kafka-2-0-nar/1.9.0/org.apache.nifi.processors.kafka.pubsub.PublishKafkaRecord_2_0/index.html

Re: What is the difference between publishKafka and publishKafkaRecord processors in Apache NiFi

Rishi_DE — Tue, 01 Jun 2021 19:44:32 GMT

So if I have used JSON transformations on my data and each time 100 flow file are generated so I guess both will be equivalent ...

Re: What is the difference between publishKafka and publishKafkaRecord processors in Apache NiFi

aakulov — Tue, 01 Jun 2021 19:56:55 GMT

If your files contain one record each, then yes, these NiFi processors will be equivalent.