Skip to main content
tpatel
tpatel
Silver ⭐️⭐️
May 8, 2021
Question

flow strategy when 200 types of command or events

Hi,

 

I am new to Nifi and three words to say. "I love it". To move forward, one needs to make a serious decision and trying to solve one puzzle.  The flow I need to design is reading from Kafka messages from 1000 different topics (each topic is customer, need to keep them separate) and each topic has 100 different types of command-type or event-type. 

Every event type has JSON data with different fields and values which need to transform.  My worry is what is the best way to handle this?

How to route and handle 100 different command/events along with 1000 diff topics?

3 replies

tpatel
tpatelAuthor
Silver ⭐️⭐️
May 25, 2021

Please anyone, any documentation or links to start with? 

Translate
dyadav1
Gold ⭐️
dyadav1
Gold ⭐️
May 26, 2021

Hi

 

If your use case is to retrieve data from multiple kafka topic using nifi then you can refer the below articles that can help you in this case:

 

https://community.cloudera.com/t5/Support-Questions/Retrieving-from-multiple-Kafka-topics-through-Nifi-causes/td-p/225277

https://stackoverflow.com/questions/51701582/multiple-kafka-topics-in-publishkafka-processor-in-apache-nifi

Translate
tpatel
tpatelAuthor
Silver ⭐️⭐️
May 26, 2021

Hi @dyadav1

 

Thanks for pointing out the direction for Kafka topic but I need little more help to determine the strategy, as I mentioned if every topic is putting 100 different types of command and I need to transform each type of command into new type (assuming Jolt transformation and have jolt spec for 100 different command types), how do I store those spec and make generalize flow so that new command types get added I just need to update Spec and flow didn;t require any changes. In more terms need to dynamically figure out which command type coming in the flow file and transform it based on spec, now I feel I need to use lookup and if I store all spec in database but then for each transformation I need to do database trip which I want to avoid.

Translate
Terms & ConditionsAccessibility statement