question Re: Does Atlas support Pig, Sqoop in Support Questions

Does Atlas support Pig, Sqoop

yashwant_chandr — Mon, 19 Aug 2019 12:19:42 GMT

Does Atlas support the tags for Spark, Pig and Sqoop?

Where does Atlas store the metadata?

Does Atlas provide the feature to fetch metadata for other tools (e.g. Informatica) and integrate it with its own metadata?

Re: Does Atlas support Pig, Sqoop

aervits — Wed, 06 Jan 2016 22:29:30 GMT

you can always check the release notes for what's supported and added but so far I don't see support for Spark and Pig, there is work in integrating Sqoop with Atlas. Here are the latest Release Notes

http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-...

Re: Does Atlas support Pig, Sqoop

nsabharwal — Wed, 06 Jan 2016 22:40:20 GMT

@Yashwant Chandrakar

You can add as many tags as you want.

Re: DB http://atlas.incubator.apache.org/Configuration.ht...

Also , see this on data model http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-...

Re: Does Atlas support Pig, Sqoop

SQLShaw — Wed, 06 Jan 2016 22:42:10 GMT

Atlas only supports the sqoop bridge for hiveimport. http://atlas.incubator.apache.org/Bridge-Sqoop.html. Currently I don't see any Pig integration. The metadata is stored in a Titan\HBase repository. This is a graph database. Here is the link to the architecture: http://atlas.incubator.apache.org/Architecture.html. I have not heard recently of 3rd party integration with Atlas but I suspect its on the roadmap.

Re: Does Atlas support Pig, Sqoop

sdutta — Wed, 13 Jan 2016 02:52:15 GMT

The support for sqoop instrumentation is present as part of eerie tech preview. If you get the eerie tech preview you can make it work. In Atlas there is a tutorial where I have used the REST API to model this. https://github.com/shivajid/atlas/tree/master/tutorial.

Re: Does Atlas support Pig, Sqoop

yashwant_chandr — Wed, 13 Jan 2016 14:18:46 GMT

Thanks all of you for your answers. Below are answers that i got from Apache atlas developer

Apache atlas supports integration with hive. limited integration with storm, kafka, sqoop and falcon is available in 0.6 Atlas

metadata is stored in titan graph

Atlas doesn’t support metadata exchange with informatica currently. Its in the roadmap