Support Questions
Find answers, ask questions, and share your expertise
Announcements
Check out our newest addition to the community, the Cloudera Innovation Accelerator group hub.

How to integrate NiFi with Atlas, for metadata, lineage, etc.

Rising Star

Hello,

I'm looking for info/ideas/examples on how to integrate NiFi with Atlas, to store metadata and lineage information from NiFi in Atlas. I found couple of custom processor examples (https://community.hortonworks.com/repos/39432/nifi-atlas-lineage-reporter.html, https://community.hortonworks.com/repos/66014/nifi-atlas-bridge.html) here in HCC, which may not be Production ready, so I am trying to find solutions that can be implemented in a Production environment.

Thanks in advance.

1 ACCEPTED SOLUTION

Master Guru

@Raj B

There are no existing NiFi reporting tasks that are part of any current NiFi release for sending any information to Atlas. So if you are looking for something that has been tested and accepted by the Apache community, it does not exist yet.

Thanks,

Matt

View solution in original post

4 REPLIES 4

Rising Star

Master Guru

@Raj B

There are no existing NiFi reporting tasks that are part of any current NiFi release for sending any information to Atlas. So if you are looking for something that has been tested and accepted by the Apache community, it does not exist yet.

Thanks,

Matt

Rising Star

Ok great, thanks for letting me know @Matt Clarke

New Contributor

While there are no official NiFi tasks (processor, controller services, reporting tasks, etc.) users can integrate NiFi and Atlas for data lineage using processors that come out of the box with NiFi. UpdateAttribute, ReplaceTExt, InokeHTTP, EvaluateJsonPath can be stitched together to invoke the Atlas API to populate lineage information.

Wrapping this up in a template/progress group, one can productionise the flow.

That said, Vadim's solution is a much more elegant implementation that can be extended. I'm sure this will be the approach taken by the NiFi development team in their implementation of the NiFi-Atlas bridge.