Support Questions

Find answers, ask questions, and share your expertise

What is the difference between publishKafka and publishKafkaRecord processors in Apache NiFi

avatar
Frequent Visitor

What is the difference between using the PublishKafka processor versus the PublishKafkaRecord processor when sending data from Apache NiFi to Kafka?

1 ACCEPTED SOLUTION

avatar
Master Collaborator

PublishKafka sends the entire flow file to Kafka as a single message. PublishKafkaRecord splits each "record" (e.g. row) in the flow file and sends each record as a message. 

 

References:

1. PublishKafka: https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-kafka-0-9-nar/1.6.0/org.apach...

2. PublishKafkaRecord: https://nifi.apache.org/docs/nifi-docs/components/nifi-docs/components/org.apache.nifi/nifi-kafka-2-...

 

View solution in original post

3 REPLIES 3

avatar
Master Collaborator

PublishKafka sends the entire flow file to Kafka as a single message. PublishKafkaRecord splits each "record" (e.g. row) in the flow file and sends each record as a message. 

 

References:

1. PublishKafka: https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-kafka-0-9-nar/1.6.0/org.apach...

2. PublishKafkaRecord: https://nifi.apache.org/docs/nifi-docs/components/nifi-docs/components/org.apache.nifi/nifi-kafka-2-...

 

avatar
Frequent Visitor

So if I have used JSON transformations on my data and each time 100 flow file are generated so I guess both will be equivalent ...

avatar
Master Collaborator

If your files contain one record each, then yes, these NiFi processors will be equivalent.