Support Questions

Find answers, ask questions, and share your expertise

What is the difference between publishKafka and publishKafkaRecord processors in Apache NiFi

avatar
New Contributor
 
1 ACCEPTED SOLUTION

avatar
Master Collaborator

PublishKafka sends the entire flow file to Kafka as a single message. PublishKafkaRecord splits each "record" (e.g. row) in the flow file and sends each record as a message. 

 

References:

1. PublishKafka: https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-kafka-0-9-nar/1.6.0/org.apach...

2. PublishKafkaRecord: https://nifi.apache.org/docs/nifi-docs/components/nifi-docs/components/org.apache.nifi/nifi-kafka-2-...

 

View solution in original post

3 REPLIES 3

avatar
Master Collaborator

PublishKafka sends the entire flow file to Kafka as a single message. PublishKafkaRecord splits each "record" (e.g. row) in the flow file and sends each record as a message. 

 

References:

1. PublishKafka: https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-kafka-0-9-nar/1.6.0/org.apach...

2. PublishKafkaRecord: https://nifi.apache.org/docs/nifi-docs/components/nifi-docs/components/org.apache.nifi/nifi-kafka-2-...

 

avatar
New Contributor

So if I have used JSON transformations on my data and each time 100 flow file are generated so I guess both will be equivalent ...

avatar
Master Collaborator

If your files contain one record each, then yes, these NiFi processors will be equivalent.