Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

What is the difference between publishKafka and publishKafkaRecord processors in Apache NiFi

avatar
New Contributor
 
1 ACCEPTED SOLUTION

avatar
Master Collaborator

PublishKafka sends the entire flow file to Kafka as a single message. PublishKafkaRecord splits each "record" (e.g. row) in the flow file and sends each record as a message. 

 

References:

1. PublishKafka: https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-kafka-0-9-nar/1.6.0/org.apach...

2. PublishKafkaRecord: https://nifi.apache.org/docs/nifi-docs/components/nifi-docs/components/org.apache.nifi/nifi-kafka-2-...

 

View solution in original post

3 REPLIES 3

avatar
Master Collaborator

PublishKafka sends the entire flow file to Kafka as a single message. PublishKafkaRecord splits each "record" (e.g. row) in the flow file and sends each record as a message. 

 

References:

1. PublishKafka: https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-kafka-0-9-nar/1.6.0/org.apach...

2. PublishKafkaRecord: https://nifi.apache.org/docs/nifi-docs/components/nifi-docs/components/org.apache.nifi/nifi-kafka-2-...

 

avatar
New Contributor

So if I have used JSON transformations on my data and each time 100 flow file are generated so I guess both will be equivalent ...

avatar
Master Collaborator

If your files contain one record each, then yes, these NiFi processors will be equivalent.