Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Does Atlas support Pig, Sqoop

avatar
New Contributor

1207-pig.png

1208-atlas.png

  • Does Atlas support the tags for Spark, Pig and Sqoop?
  • Where does Atlas store the metadata?
  • Does Atlas provide the feature to fetch metadata for other tools (e.g. Informatica) and integrate it with its own metadata?
1 ACCEPTED SOLUTION

avatar
New Contributor

Thanks all of you for your answers. Below are answers that i got from Apache atlas developer

Apache atlas supports integration with hive. limited integration with storm, kafka, sqoop and falcon is available in 0.6 Atlas

metadata is stored in titan graph

Atlas doesn’t support metadata exchange with informatica currently. Its in the roadmap

View solution in original post

5 REPLIES 5

avatar
Master Mentor

you can always check the release notes for what's supported and added but so far I don't see support for Spark and Pig, there is work in integrating Sqoop with Atlas. Here are the latest Release Notes

http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-...

avatar
Master Mentor

avatar

Atlas only supports the sqoop bridge for hiveimport. http://atlas.incubator.apache.org/Bridge-Sqoop.html. Currently I don't see any Pig integration. The metadata is stored in a Titan\HBase repository. This is a graph database. Here is the link to the architecture: http://atlas.incubator.apache.org/Architecture.html. I have not heard recently of 3rd party integration with Atlas but I suspect its on the roadmap.

avatar

The support for sqoop instrumentation is present as part of eerie tech preview. If you get the eerie tech preview you can make it work. In Atlas there is a tutorial where I have used the REST API to model this. https://github.com/shivajid/atlas/tree/master/tutorial.

avatar
New Contributor

Thanks all of you for your answers. Below are answers that i got from Apache atlas developer

Apache atlas supports integration with hive. limited integration with storm, kafka, sqoop and falcon is available in 0.6 Atlas

metadata is stored in titan graph

Atlas doesn’t support metadata exchange with informatica currently. Its in the roadmap