question Re: flow strategy when 200 types of command or events in Support Questions

flow strategy when 200 types of command or events

tpatel — Sat, 08 May 2021 03:32:18 GMT

Hi,

I am new to Nifi and three words to say. "I love it". To move forward, one needs to make a serious decision and trying to solve one puzzle. The flow I need to design is reading from Kafka messages from 1000 different topics (each topic is customer, need to keep them separate) and each topic has 100 different types of command-type or event-type.

Every event type has JSON data with different fields and values which need to transform. My worry is what is the best way to handle this?

How to route and handle 100 different command/events along with 1000 diff topics?

Re: flow strategy when 200 types of command or events

tpatel — Tue, 25 May 2021 16:37:03 GMT

Please anyone, any documentation or links to start with?

Re: flow strategy when 200 types of command or events

dyadav1 — Wed, 26 May 2021 09:05:46 GMT

If your use case is to retrieve data from multiple kafka topic using nifi then you can refer the below articles that can help you in this case:

https://community.cloudera.com/t5/Support-Questions/Retrieving-from-multiple-Kafka-topics-through-Nifi-causes/td-p/225277

https://stackoverflow.com/questions/51701582/multiple-kafka-topics-in-publishkafka-processor-in-apache-nifi

Re: flow strategy when 200 types of command or events

tpatel — Wed, 26 May 2021 19:27:30 GMT

Hi @dyadav1

Thanks for pointing out the direction for Kafka topic but I need little more help to determine the strategy, as I mentioned if every topic is putting 100 different types of command and I need to transform each type of command into new type (assuming Jolt transformation and have jolt spec for 100 different command types), how do I store those spec and make generalize flow so that new command types get added I just need to update Spec and flow didn;t require any changes. In more terms need to dynamically figure out which command type coming in the flow file and transform it based on spec, now I feel I need to use lookup and if I store all spec in database but then for each transformation I need to do database trip which I want to avoid.