flow strategy when 200 types of command or events

[#1]

Hi,

I am new to Nifi and three words to say. "I love it". To move forward, one needs to make a serious decision and trying to solve one puzzle.  The flow I need to design is reading from Kafka messages from 1000 different topics (each topic is customer, need to keep them separate) and each topic has 100 different types of command-type or event-type. 

Every event type has JSON data with different fields and values which need to transform.  My worry is what is the best way to handle this?

How to route and handle 100 different command/events along with 1000 diff topics?

[#2]

Please anyone, any documentation or links to start with? 

[#3]

Hi

If your use case is to retrieve data from multiple kafka topic using nifi then you can refer the below articles that can help you in this case:

https://community.cloudera.com/t5/Support-Questions/Retrieving-from-multiple-Kafka-topics-through-Ni...

https://stackoverflow.com/questions/51701582/multiple-kafka-topics-in-publishkafka-processor-in-apac...

[#4]

Hi @dyadav1

Thanks for pointing out the direction for Kafka topic but I need little more help to determine the strategy, as I mentioned if every topic is putting 100 different types of command and I need to transform each type of command into new type (assuming Jolt transformation and have jolt spec for 100 different command types), how do I store those spec and make generalize flow so that new command types get added I just need to update Spec and flow didn;t require any changes. In more terms need to dynamically figure out which command type coming in the flow file and transform it based on spec, now I feel I need to use lookup and if I store all spec in database but then for each transformation I need to do database trip which I want to avoid.