question Re: How to use Atlas metadata with DynamoDB in Support Questions

How to use Atlas metadata with DynamoDB

m_imran_tariq — Fri, 16 Sep 2022 12:01:18 GMT

Hi, I have created 'BerkeleyDB and Elastic search' build for Atlas which are used by Titan as back-end

Is there any way to integrate/export this data with DynamoDB or MySQL type of databases?

Re: How to use Atlas metadata with DynamoDB

egarelnabi — Mon, 31 Jul 2017 23:20:22 GMT

@Muhammad Imran Tariq

No, Atlas requires Titan Graph Database which only supports BerkeleyDB, HBase and Cassandra as storage backends.

http://atlas.apache.org/Architecture.html

http://titan.thinkaurelius.com/

Re: How to use Atlas metadata with DynamoDB

m_imran_tariq — Tue, 01 Aug 2017 13:21:31 GMT

@Eyad Garelnabi

According to Amazon it also support DynamoDB

My purpose of this question is that I also want my metadata to be queried and search in any other system/tool other than Atlas. Currently BerkeleyDB did not allow me to connect and search data directly into it without using Atlas.

Re: How to use Atlas metadata with DynamoDB

m_imran_tariq — Tue, 01 Aug 2017 13:22:56 GMT

@Eyad Garelnabi

I want to use metadata stored in Atlas outside of Atlas. What could be the possible way?

Re: How to use Atlas metadata with DynamoDB

egarelnabi — Wed, 02 Aug 2017 00:00:41 GMT

Please share a link to the source. There might be some confusion.

Re: How to use Atlas metadata with DynamoDB

egarelnabi — Wed, 02 Aug 2017 00:05:41 GMT

How are you trying to use the metadata?

In most of our implementations we use the Atlas REST API (http://atlas.apache.org/api/v2/index.html ) for metadata/lineage import and export. Have you considered using that?

Please note that I have linked above to the new API, the legacy API ( http://atlas.apache.org/api/rest.html ) has been deprecated with HDP 2.6 (Atlas 0.8) and will be removed in a future version.

Re: How to use Atlas metadata with DynamoDB

m_imran_tariq — Wed, 02 Aug 2017 13:23:46 GMT

https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Tools.TitanDB.html

Re: How to use Atlas metadata with DynamoDB

m_imran_tariq — Wed, 02 Aug 2017 13:25:56 GMT

@Eyad Garelnabi Yes I checked this API. I export metadata and this API export this data in the form of different files. Yes I am trying to use metadata outside of Atlas.

Re: How to use Atlas metadata with DynamoDB

egarelnabi — Wed, 02 Aug 2017 19:49:08 GMT

When you follow the links to the Github repo you'll see that AWS has built custom adapters to allow integration with JanusGraph not Titan. JanusGraph is a fork of the Titan project and has some differences.

Re: How to use Atlas metadata with DynamoDB

m_imran_tariq — Wed, 02 Aug 2017 20:42:57 GMT

Oh I see. This means in order to use DynamoDB, we have to use Janusgraph in Atlas instead of Titan. Right?

Re: How to use Atlas metadata with DynamoDB

anaik — Thu, 03 Aug 2017 02:24:21 GMT

In order to ingest the metadata for a SQL like source, you can use Sqoop which integrates with Atlas.

If you're looking for using DynamoDB/MySQL as datastore for Atlas, that not currently possible with Titan 0.5.4 (only Hbase, cassandra and berkeleyDB are supported).

Re: How to use Atlas metadata with DynamoDB

egarelnabi — Thu, 03 Aug 2017 02:28:18 GMT

No, Atlas only works with Titan, not Janusgraph. So you cannot use DynamoDB as a datastore for Atlas.

Re: How to use Atlas metadata with DynamoDB

m_imran_tariq — Thu, 03 Aug 2017 13:53:23 GMT

@anaik

Can we use Sqoop export to export the metadata of non-hive entities(e.g. entities of MySQL tables)?

Re: How to use Atlas metadata with DynamoDB

anaik — Thu, 03 Aug 2017 22:45:58 GMT

Hi @Muhammad Imran Tariq,

If you're looking for bringing in data FROM MySQL into Atlas, that can be done using sqoop.

Re: How to use Atlas metadata with DynamoDB

m_imran_tariq — Fri, 04 Aug 2017 07:22:13 GMT

@anaik

I am looking for; from Atlas to MySQL or Atlas to Hive. Can this be done using Sqoop?