We have files being pushed into HDFS using curl and WebHDFS interface. Files mostly contain structured data and there is lots of metadata available around fields, data type, file description etc. at the time of ingestion. There is no specific requirement to add these files to Hive. As per my understanding Atlas is more aligned with Sqoop/Hive based data ingestion. Can we add the file specific metadata in Atlas, some how at the time ingestion using curl/WebHDFS? Example: HDFS location, file name, fields, datatype etc. An alternate might be to use Falcon and using free form tags to the metadata but that would require changes to the way data is currently ingested, unless we can schedule a curl script in Falcon. Any ideas or suggestions... Thanks! ... View more

We know that Atlas works well within Hadoop ecosystem components. Is there any existing or future possibility as per the Atlas roadmap to use it as an Enterprise Metadata repository with end-to-end lineage including RDBMS and conventional ETL tools? If we could have just one metadata data tool, what are some of the existing Tools which are more universal and work with all data sources within an enterprise and capture end-to-end data lineage? Thanks in advance for your response... ... View more