Support Questions

Find answers, ask questions, and share your expertise

How to use Atlas metadata with DynamoDB

Hi, I have created 'BerkeleyDB and Elastic search' build for Atlas which are used by Titan as back-end

Is there any way to integrate/export this data with DynamoDB or MySQL type of databases?

1 ACCEPTED SOLUTION

@Muhammad Imran Tariq

No, Atlas requires Titan Graph Database which only supports BerkeleyDB, HBase and Cassandra as storage backends.

http://atlas.apache.org/Architecture.html

http://titan.thinkaurelius.com/

View solution in original post

14 REPLIES 14

@Muhammad Imran Tariq

No, Atlas requires Titan Graph Database which only supports BerkeleyDB, HBase and Cassandra as storage backends.

http://atlas.apache.org/Architecture.html

http://titan.thinkaurelius.com/

@Eyad Garelnabi

I want to use metadata stored in Atlas outside of Atlas. What could be the possible way?

How are you trying to use the metadata?

In most of our implementations we use the Atlas REST API (http://atlas.apache.org/api/v2/index.html ) for metadata/lineage import and export. Have you considered using that?

Please note that I have linked above to the new API, the legacy API ( http://atlas.apache.org/api/rest.html ) has been deprecated with HDP 2.6 (Atlas 0.8) and will be removed in a future version.

@Eyad Garelnabi Yes I checked this API. I export metadata and this API export this data in the form of different files. Yes I am trying to use metadata outside of Atlas.

@Eyad Garelnabi

According to Amazon it also support DynamoDB

My purpose of this question is that I also want my metadata to be queried and search in any other system/tool other than Atlas. Currently BerkeleyDB did not allow me to connect and search data directly into it without using Atlas.

Please share a link to the source. There might be some confusion.

When you follow the links to the Github repo you'll see that AWS has built custom adapters to allow integration with JanusGraph not Titan. JanusGraph is a fork of the Titan project and has some differences.

Oh I see. This means in order to use DynamoDB, we have to use Janusgraph in Atlas instead of Titan. Right?

No, Atlas only works with Titan, not Janusgraph. So you cannot use DynamoDB as a datastore for Atlas.

Rising Star

In order to ingest the metadata for a SQL like source, you can use Sqoop which integrates with Atlas.

If you're looking for using DynamoDB/MySQL as datastore for Atlas, that not currently possible with Titan 0.5.4 (only Hbase, cassandra and berkeleyDB are supported).

@anaik

Can we use Sqoop export to export the metadata of non-hive entities(e.g. entities of MySQL tables)?

Rising Star

Hi @Muhammad Imran Tariq,

If you're looking for bringing in data FROM MySQL into Atlas, that can be done using sqoop.

@anaik

I am looking for; from Atlas to MySQL or Atlas to Hive. Can this be done using Sqoop?