Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Enforce ordering on kafka messages in nifi

Enforce ordering on kafka messages in nifi

New Contributor

Hi, We are consuming kafka messages from different consumer groups and then distributing it on the cluster using RPG. Each flow file takes a different route before finally getting to PutHbasejson processor. We need to make sure that they get inserted in Hbase depending on the order they were consumed e.g Update should only be inserted after Insert etc. But since they go through different routes they often reach out of order to hbase. I have tried using the timestamp feature to version rows in Hbase but often times both update/delete have the same time stamp (which is technically incorrect behavior). Anyway the only other option left is to make sure to enforce the order. I have tried understanding the enforce order processor but its a little confusing PFB
1.As I understand Enforce order will only order the flow files on a single node and not the whole cluster? If so,How do i merge all the kafka messages on a single node?

2. There is property called Group identifier in the processor, we have different consumer ids in the consume kafka processor but i think consumer id cannot be accessed as attribute so how do i ensure that it only enforces order from messages from a specific consumer?

I would ideally want to version the rows in hbase based on the timestamp(maybe concatenated with Primary keys) but i haven't been able to find any example of that. Please share if you have worked with something similar.

Don't have an account?
Coming from Hortonworks? Activate your account here