Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

NiFi:ConsumeKafka with dynamic topic name list

NiFi:ConsumeKafka with dynamic topic name list

Explorer

Using ConsumeKafka processor in NiFi - can the Topic Name(s) property list be dynamically updated (without a processor restart)?

I have a need to expose a web service trough NiFi that can retrieve metadata on which topic names to consume/ingest. So when receiving a new one topic consume request, I need to update the Topic Name(s) property list of the ConsumeKafka processor (adding the new topic to the list). How do I achieve that? Does it afterwards require a restart of the ConsumeKafka processor (through the NiFi REST API)?

The topic list is planned to be persisted somewhere. Could simply be a flat file or in RDBMS, HBASE or just about anywhere. Any preferences recommended?

1 REPLY 1
Highlighted

Re: NiFi:ConsumeKafka with dynamic topic name list

The processor has a pool of consumers that is created when the processor is started so you would have to use the REST API to stop the processor, change the value of the topic names property, and then start the processor again.

Don't have an account?
Coming from Hortonworks? Activate your account here