Atlas HDFS hook/bridge

[#1]

Hi, When files are ingested into HDFS (eg. let's say manually by hdfs CLI, not by a job), we would like to have the file's technical metadata added into Atlas near real-time (for any Create/Uupdate/Delete operations on the file). So the file's metadata will be in Atlas and some time later other jobs can look for it and to create lineage. I also noticed Atlas has already defined some types for files/directories (i.e., fs_path, hdfs_path, etc..) I'm wondering if it make sense to create an Atlas hdfs hook, and if so, would this something that can be consider adding into Atlas' road map? Thanks.

[#2 · Accepted solution]

@Otto

The Atlas team continues to work on adding more hooks for other components of the stack. I believe that HDFS is roadmap, but I can't say when it is likely to be released. Here is some work related to that: https://issues.apache.org/jira/browse/ATLAS-599