Apache Atlas - Unable to collect HDFS metadata

Question

I want to collect the metadata from HDFS. But when i searched , it looks like there is no atlas hook for HDFS available like it is there for kafka , hive and other data sources. Can I get the HDFS hook  install steps or the steps to create a custom hook.

pkr · Accepted Answer

@Nigal ,

Currently , When you create hive/sqoop/falcon/storm entity which has an association to HDFS path, it shows up in Atlas.

Otherwise , any file/folder created in HDFS doesn't show up in Atlas.

For example, when you create a directory in HDFS , Atlas doesn't ingest it .

But when you create a hive table like :

"CREATE EXTERNAL TABLE test_table ( id int,value string) LOCATION '/user/cloudera/text' "

Atlas creates a lineage graph which shows relationship between the hive table and the HDFS path.

You can see the HDFS directories by searching "hdfs_path" and the hive tables by searching "hive_table".

pkr · Answer

Hello @Nigal ,

Yes right. There is no 'HDFS hook' pre-defined in Atlas.

Atlas mainly collects information from Hive - Spark - Hbase - Impala

https://docs.cloudera.com/cdp-private-cloud-base/7.1.6/cdp-governance-overview/topics/atlas-metadata-collection-overview.html

hdfs_path is synced only if this belongs to a Hive table's lineage (as is explained in https://issues.apache.org/jira/browse/ATLAS-599). By default, Atlas won't fetch HDFS paths.

Unlike HIVE entities, HDFS entities within Atlas are created manually using the Create Entity link within the Atlas Web UI.

Please check out the list of available 'hooks' in Atlas:

https://atlas.apache.org

Here's a document on creating hdfs_path manually in Atlas:

https://atlas.apache.org/2.0.0/Export-HDFS-API.html

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.