question Re: How to avoid record reader/writer when publishing to kafka in NiFi in Archives of Support Questions (Read Only)

How to avoid record reader/writer when publishing to kafka in NiFi

praveen_bathala — Fri, 27 Apr 2018 05:16:30 GMT

I am trying to fetch files (very small files, but millions of them) from S3 and put it into kafka and Kafka consumer will read each file from Kafka and process the data inside the file.

I noticed that publish kafka 0_11 processor forces to use record reader/writer, but for data i am trying to put into Kafka, I am not sure what will be my schema. Can I just avoid this and NiFi not try to parse this data and just pass through the flow file content and publish to kafka?

Re: How to avoid record reader/writer when publishing to kafka in NiFi

Shu_ashu — Fri, 27 Apr 2018 06:15:58 GMT

@Praveen Bathala

Could you try considering PublishKafka_0_10 (or) PublishKafka processors which are not going to force you to specify Record Reader/Writer.

Re: How to avoid record reader/writer when publishing to kafka in NiFi

praveen_bathala — Fri, 27 Apr 2018 06:36:39 GMT

@Shu Never noticed there is another publish processor for Kafka, PublishKafka_0_11 seems to be working fine. Thanks for the answer!