Created 12-11-2015 03:38 AM
We know that Atlas works well within Hadoop ecosystem components. Is there any existing or future possibility as per the Atlas roadmap to use it as an Enterprise Metadata repository with end-to-end lineage including RDBMS and conventional ETL tools?
If we could have just one metadata data tool, what are some of the existing Tools which are more universal and work with all data sources within an enterprise and capture end-to-end data lineage?
Thanks in advance for your response...
Created 12-11-2015 06:45 PM
Atlas has ton of exciting features in the roadmap and definitely plans for two way metadata exchange with other metadata management tools. As of right now (& this may change), the plan is to exchange the lineage information with other tools too, to be able to provide an end-to-end lineage of data from source system, all the way to the final destination. With that said, it seems very unlikely that in a large enterprise setting you would replace all other metadata tools with one magical tool.
Typically speaking the governance tools are expected to tap into the data processes automatically and non-intrusively to gather lineage information and this require native hooks into those data processes. Atlas has and will continue to expand, when it comes to native hooks for processing that takes place in a Hadoop cluster but I doubt there is any interest in tapping natively into the processes going on into other systems like data warehousing system, transactional, operational and reporting systems. For those pieces (metadata and lineage) from external systems, Atlas will continue to rely on and integrate with other metadata tools.
Just like Hadoop, other components in overall data architecture have their roles and place so they will continue to exist and so will the governance tools for those components. Vendors need to and (most likely) will work together to provide a seamless experience to the customers.
If you havent watched this presentation from Andrew Ahn, PM for Governance Tools at HWX, I would highly recommend it to understand better where Atlas is going - https://www.youtube.com/watch?time_continue=3&v=LZ...
Hope this helps. Let me know if you have any follow up question.
Created 12-11-2015 05:51 AM
Created 12-11-2015 06:45 PM
Atlas has ton of exciting features in the roadmap and definitely plans for two way metadata exchange with other metadata management tools. As of right now (& this may change), the plan is to exchange the lineage information with other tools too, to be able to provide an end-to-end lineage of data from source system, all the way to the final destination. With that said, it seems very unlikely that in a large enterprise setting you would replace all other metadata tools with one magical tool.
Typically speaking the governance tools are expected to tap into the data processes automatically and non-intrusively to gather lineage information and this require native hooks into those data processes. Atlas has and will continue to expand, when it comes to native hooks for processing that takes place in a Hadoop cluster but I doubt there is any interest in tapping natively into the processes going on into other systems like data warehousing system, transactional, operational and reporting systems. For those pieces (metadata and lineage) from external systems, Atlas will continue to rely on and integrate with other metadata tools.
Just like Hadoop, other components in overall data architecture have their roles and place so they will continue to exist and so will the governance tools for those components. Vendors need to and (most likely) will work together to provide a seamless experience to the customers.
If you havent watched this presentation from Andrew Ahn, PM for Governance Tools at HWX, I would highly recommend it to understand better where Atlas is going - https://www.youtube.com/watch?time_continue=3&v=LZ...
Hope this helps. Let me know if you have any follow up question.
Created 12-12-2015 12:53 PM
Created 12-22-2015 10:12 AM
Hello,
@bsaini
The video dated May 5. Where Alas is on the roadmap ?
Do you have some recent video demonstration of the product Atlas ?
I saw in the HDP2.3 release that Atlas is on version 0.50, is it a Beta Version ?
Thank you for your response.
Created 12-22-2015 01:56 PM
@Robin GOYON Tagging @Anderw Ahn