Consume Kafka topics using wildcard

dieden9 — Tue, 27 Apr 2021 10:52:23 GMT

Hi,

I am using apache nifi's consumeKafka processor to consume IOT data.

It is working fine when explicitly specifying the topics. However, I am looking for something more dynamic so I don't have to add topics manually in nifi each time we connect a new device.

Is it possible to have wildcards in the kafka topic name field?

My topics name look like domain.iot_device_name so I need something like domain.*

Re: Consume Kafka topics using wildcard

MattWho — Tue, 11 May 2021 12:35:46 GMT

@dieden9

NiFi provides a number of Kafka processors based off the Kafka Client they are using. The original ConsumeKafka processor (no number) used the old Kafka 0.8 client. The 0.8 client processor does not offer the ability to specify a regex for the topic names. You should be using the Kafka client version processors that match the Kafka server version you are consuming from.

From ConsumeKafka_0_10 [1] on, you have the ability to configure the processor to use "names" or "pattern" for the topic name(s).

The pattern is a java regular expression.

[1] http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-kafka-1-0-nar/1.13.2/org.apache.nifi.processors.kafka.pubsub.ConsumeKafka_1_0/index.html

If you found this help with your query, please take a moment to login and click accept on this solution.

Thank you,

Matt

question Consume Kafka topics using wildcard in Support Questions

Consume Kafka topics using wildcard

Re: Consume Kafka topics using wildcard