I'm trying to understand how Atlas is working. I know that there is no hook for HDFS in Atlas (yet!?). I understood that all metadata is stored by the Atlas service in HBase and Solr. So if the HDFS hook is implemented, does it mean that all the metadata for all the files stored in HDFS will be stored in HBase too, and not alongside the file in HDFS? If so, I fail to understand how this can scale: the HDFS Ranger plugin wil need to retrieve metadata from (the remote service) Atlas for every file access!
I feel I'm missing something here... Could you please explain this use case to me?