question Re: Metadata sources availability in Atlas in Archives of Support Questions (Read Only)

Metadata sources availability in Atlas

divyasree175 — Mon, 20 Feb 2017 18:07:25 GMT

What are the metedata sources available in Atlas. Currently i am using Atlas 0.7.

We were able to see hbase_table and kafka_topic in DSL search. And we could also see kafka in data lineages . So is kafka one of the metadata sources ?

Is that means hbase and kafka one of the metadata sources. if not so , when could we expect them in Atlas as one of the metadata sources.?

Re: Metadata sources availability in Atlas

apathan — Mon, 20 Feb 2017 18:21:52 GMT

@DIVYASREE TAMILMANI

Currently, Atlas supports ingesting and managing metadata from the following sources:

Hive
Sqoop
Storm/Kafka (limited support)
Falcon (limited support)

For more details, please refer to this documentation: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/bk_data-governance/content/ch_hdp_data_governance_overview.html

Re: Metadata sources availability in Atlas

divyasree175 — Mon, 20 Feb 2017 19:51:29 GMT

Thank you so much for your quick reply...

Re: Metadata sources availability in Atlas

divyasree175 — Mon, 20 Feb 2017 19:55:14 GMT

Thank you ayub...

I have referred that document. In that they have mention limited support. Can you elaborate on that?

And can we able access kafka topic in atlas?

Re: Metadata sources availability in Atlas

apathan — Tue, 21 Feb 2017 00:52:47 GMT

@DIVYASREE TAMILMANI

(In the context of HDP-2.5 release)

As part of storm data model, kafka topic and hbase table are modeled as datasets. What it means is, whenever a storm topology is submitted, StormHook captures and posts all the metadata of spouts, bolts and endpoints including data sets(like kafka_topic and hbase_table) to Atlas.

Please note there is no hook/plugin specifically for kafka which would ingest metadata of kafka topics into atlas. For more detailed information, please take a look at the below documentation.

http://atlas.incubator.apache.org/StormAtlasHook.html

http://atlas.incubator.apache.org/Bridge-Falcon.html