Hi there! I’ve just heard about Apache Nifi through word of mouth and wondering if somebody could point me in the right direction with my use case - my team’s recently been thrown into the deep end with some requirements and would really appreciate the help. Problem: Our end game is to build a federated search of customers over a variety of large separate datasets which hold varying degrees of differing data about individuals, so it’s primarily an entity resolution problem. I was thinking Nifi could help query our various databases, merge the result, deduplicate the entries via an external tool and then push this result to an Elasticsearch instance for our applications querying. Roughly speaking something like this (haven’t tried implementing this flow yet!):- pasted-graphic-2.png So, for examples sake the following data in the result database from the first flow :- first.png Then run https://github.com/dedupeio/dedupe over this database table which will add cluster ids to aid the record linkage, e.g.:- second.png Second flow would then feed this result into Elasticsearch instance for use by the API and front-end querying. Questions: Does this approach sound feasible? How would I trigger dedupe to run to ultimately cluster the duplicates after the merged content was pushed to the database? The corollary question - how would the second flow know when to fetch results for pushing into Elasticsearch? Periodic polling? Thanks for any insight anyone can give me regarding this, I’d be happy to consider any other bits of tech stack people might have if there was an entirely better way to approach it as I’d like this to be as robust as possible. I appreciate this isn’t primarily an Nifi question and I haven’t considered any CDC process here to capture updates to the datasets so I’d imagine this would get even more complicated… P.S. I’ve watched the HortonWorks talk here https://youtu.be/fblkgr1PJ0o?t=3149 which I found helpful and mentioned these community forums. Cheers, Gavin. ... View more