Support Questions
Find answers, ask questions, and share your expertise

Atlas Elasticsearch Hook

Atlas Elasticsearch Hook

Contributor

Hi

I'm looking to capture data linage in Atlas for our Java mapreduce jobs loading data from HDFS into Elasticsearch. Even an API example. We can track it from sqoop to hdfs but not the next step. App

2 REPLIES 2
Highlighted

Re: Atlas Elasticsearch Hook

Expert Contributor

@David Sheard, Atlas lineage are generated using native types DataSet and Process (e.g. DataSet -> Process -> DataSet). If you can represent your pipeline of loading data from HDFS into elastic search using Process and DataSet types, Alas will help you generate the lineage for the above pipeline (there is no out of box solution for the above transformation pipeline). You can view an example about creating lineage for custom types here.

Hope this helps. If this answers your question please accept and up-vote the answer.

Highlighted

Re: Atlas Elasticsearch Hook

Contributor

Excellent, thanks for the link

Appreciated