question Re: terminate kafka connection if publish kafka processor is idle for 5 min and start when data received in Support Questions

terminate kafka connection if publish kafka processor is idle for 5 min and start when data received

Rohit1997jio — Mon, 20 Nov 2023 08:10:26 GMT

Hi ,

In Apache NIFI I am consuming data from one Kafka topic and publishing that data to customers Kafka topic . For this i am using ConsumeKafka and PublishKafka processor respectively.

If outgoing dataflow is idle for more than 5 minutes, the Kafka connection with customer should be terminated and reconnected when there is any new messages consumed .

how to do this with publish kafka processor ?? how to terminate and start the connection??

Re: terminate kafka connection if publish kafka processor is idle for 5 min and start when data received

joseomjr — Tue, 21 Nov 2023 21:50:04 GMT

Why do you need to stop/terminate the connection? Sort of defeats how Kafka is meant to function.

Re: terminate kafka connection if publish kafka processor is idle for 5 min and start when data received

cotopaul — Wed, 22 Nov 2023 08:10:14 GMT

@Rohit1997jio,

As @joseomjr already pointed out, by doing this, you defeat the single purpose of using Kafka. As you already know, Kafka is a stream-processing platform and basically function in a very basic way as a queue of messages.

Integrating Kafka with NiFi, especially using the processor ConsumeKafka, you basically create a bucket at the end of the queue. As long as messages are present in the queue (Kafka in your case), you will have messages arriving in your bucket (you NiFi processing layer in your case). When you do not have any messages in the kafka system, your ConsumeKafka processor will be in a let's call it idle state, meaning that it will not waste resources in vain - it will however use some resources to check whether new messages arrived or not.

That being said, I see no point in trying to kill a connection which is NOT affecting the involved systems in any way and basically defeats the entire purpose of using NiFi and Kafka.

However, if you still want to achieve this, you will need to put some extra effort in doing this. First of all, you need to create a flow which checks the state of the desired processor using the NiFi's REST API - achievable in many ways, like InvokeHTTP, ExecuteStreamCommand, etc. If nothing has been done in the past 5 minutes (displayed in the JSON received as response from REST API) you will activate an InvokeHTTP in which you call again the REST API for STOPPING the ConsumeKafka Processor.

Re: terminate kafka connection if publish kafka processor is idle for 5 min and start when data received

Rohit1997jio — Wed, 13 Dec 2023 07:04:18 GMT

consuming from nifi is fine that i want to be continuous , i want to terminate connection for publish kafka processor , when there is no new message consumed by consumeKafka processor for more then 5 minutes , i want to terminate connection with publish kafka as my consumer kafka and producer kafka are different .

as soon as my consumer kafka consumer some message , i want connection to be established again with producer kafka

Re: terminate kafka connection if publish kafka processor is idle for 5 min and start when data received

Rohit1997jio — Wed, 13 Dec 2023 07:06:05 GMT

disconnect and reconnect again this functionality i want to achieve with Publish kafka processor .

ConsumerKafka will consumer continuously .