using nifi as a kafka streaming- real-time stream processing

[#1]

Hi , 

This is my current NIFI flow where i am consuming data in NIFI using ConsumeKafka_1_0 and publishing using PublishKafka_1_0 . Now 

HiveMQ (MQTT) → Kafka Topic A → NiFi consumes → NiFi publishes → Kafka Topic B

My requirement is send data in real time , i want to do real time data streams like kafka streams .

How to achieve that using NIFI . Is kafka streams available in NIFI ?

iam not doing and data transformation or any other operation to data. Iam just simply consuming and publishing data , but now i want to do it in real time .

[#2]

@Rohit1997jio 

For real-time streaming, Apache NiFi may not going to be the best option.  NiFi works with files at rest.  So you have an ingest processor that consumes from Kafka (consumed content is written to NiFi's content_repository and associated to a NiFi FlowFile that get moved to the outbound success relationship.  ConsumeKafkaRecord processor likely to give better throughput).  So even if you connect that success directly to a PublishKafka/PublishKafkaRecord processor, you'll still have a delay of some minimal amount of time as the Publish processor will get scheduled to execute and publish a FlowFiles message or record with multiple messages to the target cluster.

Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt