Support Questions

Find answers, ask questions, and share your expertise

NiFi:ConsumeKafka with dynamic topic name list

Explorer

Using ConsumeKafka processor in NiFi - can the Topic Name(s) property list be dynamically updated (without a processor restart)?

I have a need to expose a web service trough NiFi that can retrieve metadata on which topic names to consume/ingest. So when receiving a new one topic consume request, I need to update the Topic Name(s) property list of the ConsumeKafka processor (adding the new topic to the list). How do I achieve that? Does it afterwards require a restart of the ConsumeKafka processor (through the NiFi REST API)?

The topic list is planned to be persisted somewhere. Could simply be a flat file or in RDBMS, HBASE or just about anywhere. Any preferences recommended?

1 REPLY 1

The processor has a pool of consumers that is created when the processor is started so you would have to use the REST API to stop the processor, change the value of the topic names property, and then start the processor again.