How to use Atlas metadata with DynamoDB

[#1]

Hi, I have created 'BerkeleyDB and Elastic search' build for Atlas which are used by Titan as back-end

Is there any way to integrate/export this data with DynamoDB or MySQL type of databases?

[#2 · Accepted solution]

@Muhammad Imran Tariq

No, Atlas requires Titan Graph Database which only supports BerkeleyDB, HBase and Cassandra as storage backends.

http://atlas.apache.org/Architecture.html

http://titan.thinkaurelius.com/

[#3]

@Eyad Garelnabi

I want to use metadata stored in Atlas outside of Atlas. What could be the possible way?

[#4]

How are you trying to use the metadata?

In most of our implementations we use the Atlas REST API (http://atlas.apache.org/api/v2/index.html ) for metadata/lineage import and export. Have you considered using that?

Please note that I have linked above to the new API, the legacy API ( http://atlas.apache.org/api/rest.html ) has been deprecated with HDP 2.6 (Atlas 0.8) and will be removed in a future version.

[#5]

@Eyad Garelnabi Yes I checked this API. I export metadata and this API export this data in the form of different files. Yes I am trying to use metadata outside of Atlas.

[#6]

@Eyad Garelnabi

According to Amazon it also support DynamoDB

My purpose of this question is that I also want my metadata to be queried and search in any other system/tool other than Atlas. Currently BerkeleyDB did not allow me to connect and search data directly into it without using Atlas.

[#7]

Please share a link to the source. There might be some confusion.

[#8]

https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Tools.TitanDB.html

[#9]

When you follow the links to the Github repo you'll see that AWS has built custom adapters to allow integration with JanusGraph not Titan. JanusGraph is a fork of the Titan project and has some differences.

[#10]

Oh I see. This means in order to use DynamoDB, we have to use Janusgraph in Atlas instead of Titan. Right?

[#11]

No, Atlas only works with Titan, not Janusgraph. So you cannot use DynamoDB as a datastore for Atlas.

[#12]

In order to ingest the metadata for a SQL like source, you can use Sqoop which integrates with Atlas.

If you're looking for using DynamoDB/MySQL as datastore for Atlas, that not currently possible with Titan 0.5.4 (only Hbase, cassandra and berkeleyDB are supported).

[#13]

@anaik

Can we use Sqoop export to export the metadata of non-hive entities(e.g. entities of MySQL tables)?

[#14]

Hi @Muhammad Imran Tariq,

If you're looking for bringing in data FROM MySQL into Atlas, that can be done using sqoop.

[#15]

@anaik

I am looking for; from Atlas to MySQL or Atlas to Hive. Can this be done using Sqoop?