Atlas hive hook fails with table not found error ?

[#1]

Following are hiveserver2 logs:

RROR HiveHook.java:205 - Atlas hook failed due to error java.lang.reflect.UndeclaredThrowableException at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1884) at org.apache.atlas.hive.hook.HiveHook$2.run(HiveHook.java:195) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.atlas.hook.AtlasHookException: HiveHook.registerProcess() failed. at org.apache.atlas.hive.hook.HiveHook.registerProcess(HiveHook.java:701) at org.apache.atlas.hive.hook.HiveHook.collect(HiveHook.java:268) at org.apache.atlas.hive.hook.HiveHook.access$200(HiveHook.java:83) at org.apache.atlas.hive.hook.HiveHook$2$1.run(HiveHook.java:198) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866) ... 6 more Caused by: org.apache.atlas.hook.AtlasHookException: HiveHook.processHiveEntity() failed. at org.apache.atlas.hive.hook.HiveHook.processHiveEntity(HiveHook.java:731) at org.apache.atlas.hive.hook.HiveHook.registerProcess(HiveHook.java:668) ... 12 more Caused by: org.apache.atlas.hook.AtlasHookException: HiveHook.createOrUpdateEntities() failed. at org.apache.atlas.hive.hook.HiveHook.createOrUpdateEntities(HiveHook.java:597) at org.apache.atlas.hive.hook.HiveHook.processHiveEntity(HiveHook.java:711) ... 13 more Caused by: org.apache.atlas.hook.AtlasHookException: HiveHook.createOrUpdateEntities() failed. at org.apache.atlas.hive.hook.HiveHook.createOrUpdateEntities(HiveHook.java:589) at org.apache.atlas.hive.hook.HiveHook.createOrUpdateEntities(HiveHook.java:595) ... 14 more Caused by: org.apache.hadoop.hive.ql.metadata.InvalidTableException: Table not found values__tmp__table__1 at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1213) at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1183) at org.apache.atlas.hive.hook.HiveHook.createOrUpdateEntities(HiveHook.java:568)

Any thought on how to solve this ?

[#2]

@btandel

Have you tried running the import-hive.sh?

/usr/hdp/current/atlas-server/hook-bin/import-hive.sh

See docmentation

[#3]

@Geoffrey Shelton Okot I did run that script and also it was successful. But still i see following error
ERROR AtlasHook.java:141 - Failed to send notification - attempt #2; error=java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.RecordTooLargeException: The message is 1464396 bytes when serialized which is larger than the maximum request size you have configured with the max.request.size configuration.

[#4]

@btandel

You have an issue with max.request.size the records imported was larger than the configured threshold. Can you now see the tables in ATLAS?

[#5]

i could see the tables but when i do some changes again in hive like dropping table then Atlas is not capturing.

Regarding max.request.size, do you have any more information.

[#6]

@btandel

Did you get a message like this at the end of the import? Hive Data Model imported successfully!!!

Can you create any table using CTAS from an existing table as see the lineage in atlas?

The following hive operations are captured by hive hook currently

• create database
• create table/view, create table as select
• load, import, export
• DMLs (insert)
• alter database
• alter table (skewed table information, stored as, protection is not supported)
• alter view

[#7]

Hey @Geoffrey Shelton Okot I did get message Hive Data Model imported successfully!!!.

However, i am not able to capture new lineage.

I get following error,

Atlas hook failed due to error java.lang.reflect.UndeclaredThrowableException at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1884) at org.apache.atlas.hive.hook.HiveHook$2.run(HiveHook.java:195) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.atlas.hook.AtlasHookException: HiveHook.registerProcess() failed. at org.apache.atlas.hive.hook.HiveHook.registerProcess(HiveHook.java:701) at org.apache.atlas.hive.hook.HiveHook.collect(HiveHook.java:268) at org.apache.atlas.hive.hook.HiveHook.access$200(HiveHook.java:83) at org.apache.atlas.hive.hook.HiveHook$2$1.run(HiveHook.java:198) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866) ... 6 more Caused by: org.apache.atlas.hook.AtlasHookException: HiveHook.processHiveEntity() failed. at org.apache.atlas.hive.hook.HiveHook.processHiveEntity(HiveHook.java:731) at org.apache.atlas.hive.hook.HiveHook.registerProcess(HiveHook.java:668) ... 12 more Caused by: org.apache.atlas.hook.AtlasHookException: HiveHook.createOrUpdateEntities() failed. at org.apache.atlas.hive.hook.HiveHook.createOrUpdateEntities(HiveHook.java:597) at org.apache.atlas.hive.hook.HiveHook.processHiveEntity(HiveHook.java:711) ... 13 more Caused by: org.apache.atlas.hook.AtlasHookException: HiveHook.createOrUpdateEntities() failed. at org.apache.atlas.hive.hook.HiveHook.createOrUpdateEntities(HiveHook.java:589) at org.apache.atlas.hive.hook.HiveHook.createOrUpdateEntities(HiveHook.java:595) ... 14 more Caused by: org.apache.hadoop.hive.ql.metadata.InvalidTableException: Table not found values__tmp__table__8 at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1213) at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1183)

[#8]

@btandel

Good to know you got that message.

Did you "Create Table As Select (CTAS) " from 2 different tables to see the lineage?