Member since
03-18-2016
1
Post
0
Kudos Received
0
Solutions
11-15-2016
09:39 PM
Hi,
When files are ingested into HDFS (eg. let's say manually by hdfs CLI, not by a job), we would like to have the file's technical metadata added into Atlas near real-time (for any Create/Uupdate/Delete operations on the file).
So the file's metadata will be in Atlas and some time later other jobs can look for it and to create lineage.
I also noticed Atlas has already defined some types for files/directories (i.e., fs_path, hdfs_path, etc..)
I'm wondering if it make sense to create an Atlas hdfs hook, and if so, would this something that can be consider adding into Atlas' road map?
Thanks.
... View more
Labels:
- Labels:
-
Apache Atlas
-
Apache Hadoop