question Re: How to integrate NiFi with Atlas, for metadata, lineage, etc. in Archives of Support Questions (Read Only)

How to integrate NiFi with Atlas, for metadata, lineage, etc.

Raj_B — Fri, 17 Feb 2017 04:51:49 GMT

Hello,

I'm looking for info/ideas/examples on how to integrate NiFi with Atlas, to store metadata and lineage information from NiFi in Atlas. I found couple of custom processor examples (https://community.hortonworks.com/repos/39432/nifi-atlas-lineage-reporter.html, https://community.hortonworks.com/repos/66014/nifi-atlas-bridge.html) here in HCC, which may not be Production ready, so I am trying to find solutions that can be implemented in a Production environment.

Thanks in advance.

Re: How to integrate NiFi with Atlas, for metadata, lineage, etc.

Raj_B — Wed, 01 Mar 2017 22:08:37 GMT

Any takers ? @Bryan Bende @Timothy Spann @Matt Clarke @Matt Burgess @Andrew Grande

Re: How to integrate NiFi with Atlas, for metadata, lineage, etc.

MattWho — Thu, 02 Mar 2017 03:21:48 GMT

@Raj B

There are no existing NiFi reporting tasks that are part of any current NiFi release for sending any information to Atlas. So if you are looking for something that has been tested and accepted by the Apache community, it does not exist yet.

Thanks,

Matt

Re: How to integrate NiFi with Atlas, for metadata, lineage, etc.

Raj_B — Thu, 02 Mar 2017 03:58:40 GMT

Ok great, thanks for letting me know @Matt Clarke

Re: How to integrate NiFi with Atlas, for metadata, lineage, etc.

grant1 — Tue, 07 Mar 2017 08:56:42 GMT

While there are no official NiFi tasks (processor, controller services, reporting tasks, etc.) users can integrate NiFi and Atlas for data lineage using processors that come out of the box with NiFi. UpdateAttribute, ReplaceTExt, InokeHTTP, EvaluateJsonPath can be stitched together to invoke the Atlas API to populate lineage information.

Wrapping this up in a template/progress group, one can productionise the flow.

That said, Vadim's solution is a much more elegant implementation that can be extended. I'm sure this will be the approach taken by the NiFi development team in their implementation of the NiFi-Atlas bridge.