Support Questions
Find answers, ask questions, and share your expertise

What is the difference between publishKafka and publishKafkaRecord processors in Apache NiFi

Solved Go to solution

What is the difference between publishKafka and publishKafkaRecord processors in Apache NiFi

New Contributor
 
1 ACCEPTED SOLUTION

Accepted Solutions

Re: What is the difference between publishKafka and publishKafkaRecord processors in Apache NiFi

Expert Contributor

PublishKafka sends the entire flow file to Kafka as a single message. PublishKafkaRecord splits each "record" (e.g. row) in the flow file and sends each record as a message. 

 

References:

1. PublishKafka: https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-kafka-0-9-nar/1.6.0/org.apach...

2. PublishKafkaRecord: https://nifi.apache.org/docs/nifi-docs/components/nifi-docs/components/org.apache.nifi/nifi-kafka-2-...

 

View solution in original post

3 REPLIES 3

Re: What is the difference between publishKafka and publishKafkaRecord processors in Apache NiFi

Expert Contributor

PublishKafka sends the entire flow file to Kafka as a single message. PublishKafkaRecord splits each "record" (e.g. row) in the flow file and sends each record as a message. 

 

References:

1. PublishKafka: https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-kafka-0-9-nar/1.6.0/org.apach...

2. PublishKafkaRecord: https://nifi.apache.org/docs/nifi-docs/components/nifi-docs/components/org.apache.nifi/nifi-kafka-2-...

 

View solution in original post

Re: What is the difference between publishKafka and publishKafkaRecord processors in Apache NiFi

New Contributor

So if I have used JSON transformations on my data and each time 100 flow file are generated so I guess both will be equivalent ...

Re: What is the difference between publishKafka and publishKafkaRecord processors in Apache NiFi

Expert Contributor

If your files contain one record each, then yes, these NiFi processors will be equivalent.