Support Questions
Find answers, ask questions, and share your expertise

Does Atlas support Pig, Sqoop

Solved Go to solution

Does Atlas support Pig, Sqoop

New Contributor

1207-pig.png

1208-atlas.png

  • Does Atlas support the tags for Spark, Pig and Sqoop?
  • Where does Atlas store the metadata?
  • Does Atlas provide the feature to fetch metadata for other tools (e.g. Informatica) and integrate it with its own metadata?
1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Does Atlas support Pig, Sqoop

New Contributor

Thanks all of you for your answers. Below are answers that i got from Apache atlas developer

Apache atlas supports integration with hive. limited integration with storm, kafka, sqoop and falcon is available in 0.6 Atlas

metadata is stored in titan graph

Atlas doesn’t support metadata exchange with informatica currently. Its in the roadmap

View solution in original post

5 REPLIES 5
Highlighted

Re: Does Atlas support Pig, Sqoop

Mentor

you can always check the release notes for what's supported and added but so far I don't see support for Spark and Pig, there is work in integrating Sqoop with Atlas. Here are the latest Release Notes

http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-...

Highlighted

Re: Does Atlas support Pig, Sqoop

Highlighted

Re: Does Atlas support Pig, Sqoop

Atlas only supports the sqoop bridge for hiveimport. http://atlas.incubator.apache.org/Bridge-Sqoop.html. Currently I don't see any Pig integration. The metadata is stored in a Titan\HBase repository. This is a graph database. Here is the link to the architecture: http://atlas.incubator.apache.org/Architecture.html. I have not heard recently of 3rd party integration with Atlas but I suspect its on the roadmap.

Highlighted

Re: Does Atlas support Pig, Sqoop

The support for sqoop instrumentation is present as part of eerie tech preview. If you get the eerie tech preview you can make it work. In Atlas there is a tutorial where I have used the REST API to model this. https://github.com/shivajid/atlas/tree/master/tutorial.

Highlighted

Re: Does Atlas support Pig, Sqoop

New Contributor

Thanks all of you for your answers. Below are answers that i got from Apache atlas developer

Apache atlas supports integration with hive. limited integration with storm, kafka, sqoop and falcon is available in 0.6 Atlas

metadata is stored in titan graph

Atlas doesn’t support metadata exchange with informatica currently. Its in the roadmap

View solution in original post