Consume Kafka topics using wildcard

[#1]

Hi,

I am using apache nifi's consumeKafka processor to consume IOT data.

It is working fine when explicitly specifying the topics. However, I am looking for something more dynamic so I don't have to add topics manually in nifi each time we connect a new device.

Is it possible to have wildcards in the kafka topic name field?

My topics name look like domain.iot_device_name so I need something like domain.*

[#2]

@dieden9 

NiFi provides a number of Kafka processors based off the Kafka Client they are using.  The original ConsumeKafka processor (no number) used the old Kafka 0.8 client.  The 0.8 client processor does not offer the ability to specify a regex for the topic names.  You should be using the Kafka client version processors that match the Kafka server version you are consuming from.

From ConsumeKafka_0_10 [1] on, you have the ability to configure the processor to use "names" or "pattern" for the topic name(s).

The pattern is a java regular expression.

[1] http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-kafka-1-0-nar/1.13.2/org.apach...

If you found this help with your query, please take a moment to login and click accept on this solution.

Thank you,

Matt