Atlas HDFS hook/bridge

otto_day — Wed, 16 Nov 2016 05:39:18 GMT

Hi, When files are ingested into HDFS (eg. let's say manually by hdfs CLI, not by a job), we would like to have the file's technical metadata added into Atlas near real-time (for any Create/Uupdate/Delete operations on the file). So the file's metadata will be in Atlas and some time later other jobs can look for it and to create lineage. I also noticed Atlas has already defined some types for files/directories (i.e., fs_path, hdfs_path, etc..) I'm wondering if it make sense to create an Atlas hdfs hook, and if so, would this something that can be consider adding into Atlas' road map? Thanks.

Re: Atlas HDFS hook/bridge

myoung — Wed, 16 Nov 2016 05:54:17 GMT

@Otto

The Atlas team continues to work on adding more hooks for other components of the stack. I believe that HDFS is roadmap, but I can't say when it is likely to be released. Here is some work related to that: https://issues.apache.org/jira/browse/ATLAS-599

question Re: Atlas HDFS hook/bridge in Archives of Support Questions (Read Only)

Atlas HDFS hook/bridge

Re: Atlas HDFS hook/bridge