Created on 07-31-2017 06:59 AM - edited 09-16-2022 05:01 AM
Hi, I have created 'BerkeleyDB and Elastic search' build for Atlas which are used by Titan as back-end
Is there any way to integrate/export this data with DynamoDB or MySQL type of databases?
Created 07-31-2017 04:20 PM
No, Atlas requires Titan Graph Database which only supports BerkeleyDB, HBase and Cassandra as storage backends.
Created 07-31-2017 04:20 PM
No, Atlas requires Titan Graph Database which only supports BerkeleyDB, HBase and Cassandra as storage backends.
Created 08-01-2017 06:22 AM
I want to use metadata stored in Atlas outside of Atlas. What could be the possible way?
Created 08-01-2017 05:05 PM
How are you trying to use the metadata?
In most of our implementations we use the Atlas REST API (http://atlas.apache.org/api/v2/index.html ) for metadata/lineage import and export. Have you considered using that?
Please note that I have linked above to the new API, the legacy API ( http://atlas.apache.org/api/rest.html ) has been deprecated with HDP 2.6 (Atlas 0.8) and will be removed in a future version.
Created 08-02-2017 06:25 AM
@Eyad Garelnabi Yes I checked this API. I export metadata and this API export this data in the form of different files. Yes I am trying to use metadata outside of Atlas.
Created 08-01-2017 06:21 AM
According to Amazon it also support DynamoDB
My purpose of this question is that I also want my metadata to be queried and search in any other system/tool other than Atlas. Currently BerkeleyDB did not allow me to connect and search data directly into it without using Atlas.
Created 08-01-2017 05:00 PM
Please share a link to the source. There might be some confusion.
Created 08-02-2017 06:23 AM
Created 08-02-2017 12:49 PM
When you follow the links to the Github repo you'll see that AWS has built custom adapters to allow integration with JanusGraph not Titan. JanusGraph is a fork of the Titan project and has some differences.
Created 08-02-2017 01:42 PM
Oh I see. This means in order to use DynamoDB, we have to use Janusgraph in Atlas instead of Titan. Right?
Created 08-02-2017 07:28 PM
No, Atlas only works with Titan, not Janusgraph. So you cannot use DynamoDB as a datastore for Atlas.
Created 08-02-2017 07:24 PM
In order to ingest the metadata for a SQL like source, you can use Sqoop which integrates with Atlas.
If you're looking for using DynamoDB/MySQL as datastore for Atlas, that not currently possible with Titan 0.5.4 (only Hbase, cassandra and berkeleyDB are supported).
Created 08-03-2017 06:53 AM
Can we use Sqoop export to export the metadata of non-hive entities(e.g. entities of MySQL tables)?
Created 08-03-2017 03:45 PM
If you're looking for bringing in data FROM MySQL into Atlas, that can be done using sqoop.
Created 08-04-2017 12:22 AM
I am looking for; from Atlas to MySQL or Atlas to Hive. Can this be done using Sqoop?