Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

flow strategy when 200 types of command or events

avatar
Explorer

Hi,

 

I am new to Nifi and three words to say. "I love it". To move forward, one needs to make a serious decision and trying to solve one puzzle.  The flow I need to design is reading from Kafka messages from 1000 different topics (each topic is customer, need to keep them separate) and each topic has 100 different types of command-type or event-type. 

Every event type has JSON data with different fields and values which need to transform.  My worry is what is the best way to handle this?

How to route and handle 100 different command/events along with 1000 diff topics?

3 REPLIES 3

avatar
Explorer

Please anyone, any documentation or links to start with? 

avatar
Expert Contributor

Hi

 

If your use case is to retrieve data from multiple kafka topic using nifi then you can refer the below articles that can help you in this case:

 

https://community.cloudera.com/t5/Support-Questions/Retrieving-from-multiple-Kafka-topics-through-Ni...

https://stackoverflow.com/questions/51701582/multiple-kafka-topics-in-publishkafka-processor-in-apac...

avatar
Explorer

Hi @dyadav1

 

Thanks for pointing out the direction for Kafka topic but I need little more help to determine the strategy, as I mentioned if every topic is putting 100 different types of command and I need to transform each type of command into new type (assuming Jolt transformation and have jolt spec for 100 different command types), how do I store those spec and make generalize flow so that new command types get added I just need to update Spec and flow didn;t require any changes. In more terms need to dynamically figure out which command type coming in the flow file and transform it based on spec, now I feel I need to use lookup and if I store all spec in database but then for each transformation I need to do database trip which I want to avoid.