One way of doing this is by using publish kafka record processor which can read the incoming flowfile data and writes the required fileds into topic.
Fork the success relation from GetFile processor and then use two Publish Kafka Record processors to Publish messages into Kafka topic1 and kafka topic2.
Flow:-
First PublishKafkaRecord processor configure the Record reader as CsvReader to read your incoming file(with 36 fields) and Record Writer as CsvSetWriter to write only to required four fields to KafkaTopic1.
Second PublishKafkaRecord processor configure the Record reader as Csv Reader same as above to read your 36 fields and Record Writer as CsvSetWriter to write only to required thirty fields to KafkaTopic2.
By using this way we are splitting the fields and publishing them to the desired kafka topics.
(or)
if you want to publish all the fields into one topic using publishkafka_0_11 processor then want to split into two, then
Use ConsumeKafka Processor to consume the 36 fileds message and use two PublishKafkaRecord_0_11 processors with Same CsvReader as record reader and different CSVSetWriters with 4fields and 30fileds.
Now we are Consuming the messages and writing them to two different kafka topics that are configured with different csvsetwriter controller services.
Flow:-
(or)
if you want to publish all the fields into one topic using publishkafka_0_11 processor then want to split into two, then
Once you publish the messages to Kafka topic by using Publish Kafka 0_11 processor then use
Consume kafka processor to consume the published kafka messages from the topic.
Then use Two Convert record processors in parallel with same csv reader controller service and two different Csv Set Writer controller services because we need to write 4 fields to kafka1 topic and 30 fields to kafka2 topic.
then use two PublishKafka_0_11 processors in parallel to publish the prepared messages from the two convertrecord processors.
Flow:-
As you can choose which is the best fit for your case from the above three approaches.
Please refer to below links to configure/use the Record Reader and Record writer properties
One way of doing this is by using publish kafka record processor which can read the incoming flowfile data and writes the required fileds into topic.
Fork the success relation from GetFile processor and then use two Publish Kafka Record processors to Publish messages into Kafka topic1 and kafka topic2.
Flow:-
First PublishKafkaRecord processor configure the Record reader as CsvReader to read your incoming file(with 36 fields) and Record Writer as CsvSetWriter to write only to required four fields to KafkaTopic1.
Second PublishKafkaRecord processor configure the Record reader as Csv Reader same as above to read your 36 fields and Record Writer as CsvSetWriter to write only to required thirty fields to KafkaTopic2.
By using this way we are splitting the fields and publishing them to the desired kafka topics.
(or)
if you want to publish all the fields into one topic using publishkafka_0_11 processor then want to split into two, then
Use ConsumeKafka Processor to consume the 36 fileds message and use two PublishKafkaRecord_0_11 processors with Same CsvReader as record reader and different CSVSetWriters with 4fields and 30fileds.
Now we are Consuming the messages and writing them to two different kafka topics that are configured with different csvsetwriter controller services.
Flow:-
(or)
if you want to publish all the fields into one topic using publishkafka_0_11 processor then want to split into two, then
Once you publish the messages to Kafka topic by using Publish Kafka 0_11 processor then use
Consume kafka processor to consume the published kafka messages from the topic.
Then use Two Convert record processors in parallel with same csv reader controller service and two different Csv Set Writer controller services because we need to write 4 fields to kafka1 topic and 30 fields to kafka2 topic.
then use two PublishKafka_0_11 processors in parallel to publish the prepared messages from the two convertrecord processors.
Flow:-
As you can choose which is the best fit for your case from the above three approaches.
Please refer to below links to configure/use the Record Reader and Record writer properties
