Metadata sources availability in Atlas

[#1]

What are the metedata sources available in Atlas. Currently i am using Atlas 0.7.

We were able to see hbase_table and kafka_topic in DSL search. And we could also see kafka in data lineages . So is kafka one of the metadata sources ?

Is that means hbase and kafka one of the metadata sources. if not so , when could we expect them in Atlas as one of the metadata sources.?

[#2 · Accepted solution]

@DIVYASREE TAMILMANI

Currently, Atlas supports ingesting and managing metadata from the following sources:

• Hive
• Sqoop
• Storm/Kafka (limited support)
• Falcon (limited support)

For more details, please refer to this documentation: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/bk_data-governance/content/ch_hdp_data_gove...

[#3]

Thank you so much for your quick reply...

[#4]

@DIVYASREE TAMILMANI

(In the context of HDP-2.5 release)

As part of storm data model, kafka topic and hbase table are modeled as datasets. What it means is, whenever a storm topology is submitted, StormHook captures and posts all the metadata of spouts, bolts and endpoints including data sets(like kafka_topic and hbase_table) to Atlas.

Please note there is no hook/plugin specifically for kafka which would ingest metadata of kafka topics into atlas. For more detailed information, please take a look at the below documentation.

http://atlas.incubator.apache.org/StormAtlasHook.html

http://atlas.incubator.apache.org/Bridge-Falcon.html

[#5]

Thank you ayub...

I have referred that document. In that they have mention limited support. Can you elaborate on that?

And can we able access kafka topic in atlas?