How to integrate NiFi with Atlas, for metadata, lineage, etc.

[#1]

Hello,

I'm looking for info/ideas/examples on how to integrate NiFi with Atlas, to store metadata and lineage information from NiFi in Atlas. I found couple of custom processor examples (https://community.hortonworks.com/repos/39432/nifi-atlas-lineage-reporter.html, https://community.hortonworks.com/repos/66014/nifi-atlas-bridge.html) here in HCC, which may not be Production ready, so I am trying to find solutions that can be implemented in a Production environment.

Thanks in advance.

[#2]

Any takers ? @Bryan Bende @Timothy Spann @Matt Clarke @Matt Burgess @Andrew Grande

[#3 · Accepted solution]

@Raj B

There are no existing NiFi reporting tasks that are part of any current NiFi release for sending any information to Atlas. So if you are looking for something that has been tested and accepted by the Apache community, it does not exist yet.

Thanks,

Matt

[#4]

Ok great, thanks for letting me know @Matt Clarke

[#5]

While there are no official NiFi tasks (processor, controller services, reporting tasks, etc.) users can integrate NiFi and Atlas for data lineage using processors that come out of the box with NiFi. UpdateAttribute, ReplaceTExt, InokeHTTP, EvaluateJsonPath can be stitched together to invoke the Atlas API to populate lineage information.

Wrapping this up in a template/progress group, one can productionise the flow.

That said, Vadim's solution is a much more elegant implementation that can be extended. I'm sure this will be the approach taken by the NiFi development team in their implementation of the NiFi-Atlas bridge.