How to use NIFI & Atlas or Spark & Atlas

[#1]

Hi,

currently I am trying to make lineage between nifi and atlas. I put it in Directory /etc/atlas/conf/atlas-application.properties However I have the following error as soon as I start the ReportLineageToAtlas service [StandardProcessScheduler Thread-3] o.a.n.a.re.reporting.ReportLineageToAtlas ReportLineageToAtlas [id = efcd4518-0164-xxxx-ffff-xxxxxxxx] Failed to invoke @OnEnabled method due to org.apache.nifi.processor.exception.ProcessException: Could not find /atlas-application.properties in classpath. Atlas Configuration Directory a directory containing Atlas properties file, or enable Atlas Configuration File to generate it: {} org.apache.nifi.processor.exception.ProcessException: Could not find /atlas-application.properties in classpath. Please add it to classpath, or specify Atlas Configuration Directory a directory containing Atlas properties file, or enable Atlas Configuration File to generate it. at org.apache.nifi.atlas.reporting.ReportLineageToAtlas.initAtlasProperties (ReportLineageToAtlas.java:457) at org.apache.nifi.atlas.reporting.ReportLineageToAtlas.setup (ReportLineageToAtlas.java:403) at sun.reflect.GeneratedMethodAccessor857.invoke (Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43) 2) I have a second question please, is it possible to do lineage using spark and Atlas and / or java with Atlas?

Cordialement,

Koly

50430-screen-shot-2018-01-16-at-103742-pm.png

[#2]

Hi @Koly SALL,

For your first question:

the property Atlas Configuration Directory is the "Directory path" that contains 'atlas-application.properties' file. If not specified and 'Create Atlas Configuration File' is disabled, then, 'atlas-application.properties' file under root classpath is used.

since you have provided the file path include the filename and not choose to create config file option thus resulted to opt for root class path "/".

as a resolution you can remove the filename (by just keeping the directory name) should resolve that error.

to your second question :

from spark 2.3 on wards you can keep track of the leaning of hive and hdfs paths atleast by enabling the spark external listeners.

following repository developed by HWX suports to post the leaning information from spark to Atlas.

https://github.com/hortonworks-spark/spark-atlas-connector

Hope this helps !!

[#3]

Hi bkosaraju,

I have remove the filename and diseable the create Atlas Configuration but i have the same error.

You find attatched the screenshot.

Best,

Koly

captureatlas.png