Support Questions
Find answers, ask questions, and share your expertise

How to integrate NiFi with Atlas, for metadata, lineage, etc.

Solved Go to solution
Highlighted

How to integrate NiFi with Atlas, for metadata, lineage, etc.

Rising Star

Hello,

I'm looking for info/ideas/examples on how to integrate NiFi with Atlas, to store metadata and lineage information from NiFi in Atlas. I found couple of custom processor examples (https://community.hortonworks.com/repos/39432/nifi-atlas-lineage-reporter.html, https://community.hortonworks.com/repos/66014/nifi-atlas-bridge.html) here in HCC, which may not be Production ready, so I am trying to find solutions that can be implemented in a Production environment.

Thanks in advance.

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: How to integrate NiFi with Atlas, for metadata, lineage, etc.

Master Guru

@Raj B

There are no existing NiFi reporting tasks that are part of any current NiFi release for sending any information to Atlas. So if you are looking for something that has been tested and accepted by the Apache community, it does not exist yet.

Thanks,

Matt

View solution in original post

4 REPLIES 4
Highlighted

Re: How to integrate NiFi with Atlas, for metadata, lineage, etc.

Rising Star
Highlighted

Re: How to integrate NiFi with Atlas, for metadata, lineage, etc.

Master Guru

@Raj B

There are no existing NiFi reporting tasks that are part of any current NiFi release for sending any information to Atlas. So if you are looking for something that has been tested and accepted by the Apache community, it does not exist yet.

Thanks,

Matt

View solution in original post

Highlighted

Re: How to integrate NiFi with Atlas, for metadata, lineage, etc.

Rising Star

Ok great, thanks for letting me know @Matt Clarke

Highlighted

Re: How to integrate NiFi with Atlas, for metadata, lineage, etc.

New Contributor

While there are no official NiFi tasks (processor, controller services, reporting tasks, etc.) users can integrate NiFi and Atlas for data lineage using processors that come out of the box with NiFi. UpdateAttribute, ReplaceTExt, InokeHTTP, EvaluateJsonPath can be stitched together to invoke the Atlas API to populate lineage information.

Wrapping this up in a template/progress group, one can productionise the flow.

That said, Vadim's solution is a much more elegant implementation that can be extended. I'm sure this will be the approach taken by the NiFi development team in their implementation of the NiFi-Atlas bridge.