question Re: Discovering existing Hive tables in Atlas in Support Questions

Discovering existing Hive tables in Atlas

marcel-jan — Fri, 25 Jan 2019 00:02:43 GMT

Is there a way to let Atlas discover existing Hive tables (in HDP 2.6.5)? I've got a couple of Hive external tables that existed before Atlas was enabled. It didn't pick them up, but Atlas does find new tables.

I'm trying to get one Hive table in Atlas, but it's not exactly easy to do that for tables, columns, storagedesc and references. There are very few (simple) examples out there. I'm getting one ObjectId is not valid AtlasObjectId error after the other.

Re: Discovering existing Hive tables in Atlas

marcel-jan — Mon, 28 Jan 2019 23:29:58 GMT

I've been trying this now for one and a half day. I've been trying to approach this from all kinds of directions (writing the JSON from scratch, modifying exported JSON data from another table, trying the old REST API version). I've been searching the web for any solution. I'm about to give up. Anyone else ideas?

Re: Discovering existing Hive tables in Atlas

sandyy006 — Mon, 28 Jan 2019 23:50:11 GMT

@Marcel-Jan Krijgsman Do run the /usr/hdp/current/atlas-server/hook-bin/import-hive.sh utility which imports the existing hive tables into atlas.

Re: Discovering existing Hive tables in Atlas

marcel-jan — Tue, 29 Jan 2019 00:01:14 GMT

Thanks, @Sandeep Nemuri. I'll try that.

Re: Discovering existing Hive tables in Atlas

Shelton — Tue, 29 Jan 2019 00:07:22 GMT

@Marcel-Jan Krijgsman

So frustrating indeed have you tried running the hive import from /usr/hdp/2.6.5.0-292/atlas/hook-bin ? The output should look like below

# ./import-hive.sh
Using Hive configuration directory [/etc/hive/conf]
Log file for import is /usr/hdp/current/atlas-server/logs/import-hive.log
log4j:WARN No such property [maxFileSize] in org.apache.log4j.PatternLayout.
log4j:WARN No such property [maxBackupIndex] in org.apache.log4j.PatternLayout.
Enter username for atlas :- admin
Enter password for atlas :-
Hive Meta Data imported successfully!!!

After running successfully you should be able to see your tables in Atlas

Re: Discovering existing Hive tables in Atlas

sandyy006 — Tue, 29 Jan 2019 00:33:31 GMT

Seems to be the same script which i mentioned above. Isn't it?

Re: Discovering existing Hive tables in Atlas

doonaks — Tue, 29 Jan 2019 04:10:31 GMT

HTH:

https://stackoverflow.com/questions/48068756/how-do-i-update-apache-atlas-metadata?rq=1

Re: Discovering existing Hive tables in Atlas

marcel-jan — Tue, 29 Jan 2019 16:21:01 GMT

Thanks for the responses everyone. I saw I needed to have the Atlas admin account to run this script. I don't have it, but I've asked the people who do (a company that maintains the Hadoop cluster for us) to run it for me.

While we're waiting, it might be interesting to share the one thing that I did get working. So I put this in the JSON:

{
  "entity": {
    "typeName": "hive_table",
    "attributes": {
      "description": null,
      "name": "tablename",
      "owner": "degierf",
      "qualifiedName": "test.tablename@clustername",
      "replicatedFrom": [],
      "replicatedTo": [],
      "aliases": [],
      "columns": [],
      "comment": null,
      "createTime": 1494949241,
      "db": {
        "guid": "557c073c-da51-461c-8bba-3594e004db63",
        "typeName": "hive_db"
      },
      "lastAccessTime": 1494949241,
      "partitionKeys": [],
      "retention": null,
      "sd": null,
      "tableType": null,
      "temporary": null,
      "viewExpandedText": null,
      "viewOriginalText": null
    },
    "guid": -1
  },
  "referredEntities": {}
}

Then I ran it with the entity endpoint:

curl -u myaccount -i -H "Content-Type: application/json" -X POST -d @upload_hive_table_def.json https://atlasnode:21000/api/atlas/v2/entity

What you get is that Atlas but wasn't the desired result : I managed to get Atlas to recognise there's a Hive table. Just not the columns, references, etc.. I was hoping to add them one by one after this, but that didn't work.

Re: Discovering existing Hive tables in Atlas

marcel-jan — Fri, 01 Feb 2019 01:56:45 GMT

import-hive.sh was indeed the solution.

We only got it working when admin had read/execute access in Ranger to protected data in HDFS.

Now all Hive tables have been discovered.

Re: Discovering existing Hive tables in Atlas

Shelton — Fri, 01 Feb 2019 23:30:10 GMT

@Sandeep Nemuri

I think we responded at almost the same time, when some is clicking submit, there is no logic that checks whether a similar answer has already been give 🙂 Maybe you should have added that he needs to run the script as Atlas admin user as illustrated which he wasn't aware of 🙂