Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Atlas hive hook fails with table not found error ?

avatar
Explorer

Following are hiveserver2 logs:

RROR HiveHook.java:205 - Atlas hook failed due to error java.lang.reflect.UndeclaredThrowableException at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1884) at org.apache.atlas.hive.hook.HiveHook$2.run(HiveHook.java:195) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.atlas.hook.AtlasHookException: HiveHook.registerProcess() failed. at org.apache.atlas.hive.hook.HiveHook.registerProcess(HiveHook.java:701) at org.apache.atlas.hive.hook.HiveHook.collect(HiveHook.java:268) at org.apache.atlas.hive.hook.HiveHook.access$200(HiveHook.java:83) at org.apache.atlas.hive.hook.HiveHook$2$1.run(HiveHook.java:198) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866) ... 6 more Caused by: org.apache.atlas.hook.AtlasHookException: HiveHook.processHiveEntity() failed. at org.apache.atlas.hive.hook.HiveHook.processHiveEntity(HiveHook.java:731) at org.apache.atlas.hive.hook.HiveHook.registerProcess(HiveHook.java:668) ... 12 more Caused by: org.apache.atlas.hook.AtlasHookException: HiveHook.createOrUpdateEntities() failed. at org.apache.atlas.hive.hook.HiveHook.createOrUpdateEntities(HiveHook.java:597) at org.apache.atlas.hive.hook.HiveHook.processHiveEntity(HiveHook.java:711) ... 13 more Caused by: org.apache.atlas.hook.AtlasHookException: HiveHook.createOrUpdateEntities() failed. at org.apache.atlas.hive.hook.HiveHook.createOrUpdateEntities(HiveHook.java:589) at org.apache.atlas.hive.hook.HiveHook.createOrUpdateEntities(HiveHook.java:595) ... 14 more Caused by: org.apache.hadoop.hive.ql.metadata.InvalidTableException: Table not found values__tmp__table__1 at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1213) at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1183) at org.apache.atlas.hive.hook.HiveHook.createOrUpdateEntities(HiveHook.java:568)

Any thought on how to solve this ?

7 REPLIES 7

avatar
Master Mentor

@btandel

Have you tried running the import-hive.sh?

/usr/hdp/current/atlas-server/hook-bin/import-hive.sh

See docmentation

avatar
Explorer

@Geoffrey Shelton Okot I did run that script and also it was successful. But still i see following error
ERROR AtlasHook.java:141 - Failed to send notification - attempt #2; error=java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.RecordTooLargeException: The message is 1464396 bytes when serialized which is larger than the maximum request size you have configured with the max.request.size configuration.

avatar
Master Mentor

@btandel

You have an issue with max.request.size the records imported was larger than the configured threshold. Can you now see the tables in ATLAS?

avatar
Explorer

i could see the tables but when i do some changes again in hive like dropping table then Atlas is not capturing.

Regarding max.request.size, do you have any more information.

avatar
Master Mentor

@btandel

Did you get a message like this at the end of the import? Hive Data Model imported successfully!!!

Can you create any table using CTAS from an existing table as see the lineage in atlas?

The following hive operations are captured by hive hook currently

  • create database
  • create table/view, create table as select
  • load, import, export
  • DMLs (insert)
  • alter database
  • alter table (skewed table information, stored as, protection is not supported)
  • alter view

avatar
Explorer

Hey @Geoffrey Shelton Okot I did get message Hive Data Model imported successfully!!!.

However, i am not able to capture new lineage.

I get following error,

Atlas hook failed due to error java.lang.reflect.UndeclaredThrowableException at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1884) at org.apache.atlas.hive.hook.HiveHook$2.run(HiveHook.java:195) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.atlas.hook.AtlasHookException: HiveHook.registerProcess() failed. at org.apache.atlas.hive.hook.HiveHook.registerProcess(HiveHook.java:701) at org.apache.atlas.hive.hook.HiveHook.collect(HiveHook.java:268) at org.apache.atlas.hive.hook.HiveHook.access$200(HiveHook.java:83) at org.apache.atlas.hive.hook.HiveHook$2$1.run(HiveHook.java:198) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866) ... 6 more Caused by: org.apache.atlas.hook.AtlasHookException: HiveHook.processHiveEntity() failed. at org.apache.atlas.hive.hook.HiveHook.processHiveEntity(HiveHook.java:731) at org.apache.atlas.hive.hook.HiveHook.registerProcess(HiveHook.java:668) ... 12 more Caused by: org.apache.atlas.hook.AtlasHookException: HiveHook.createOrUpdateEntities() failed. at org.apache.atlas.hive.hook.HiveHook.createOrUpdateEntities(HiveHook.java:597) at org.apache.atlas.hive.hook.HiveHook.processHiveEntity(HiveHook.java:711) ... 13 more Caused by: org.apache.atlas.hook.AtlasHookException: HiveHook.createOrUpdateEntities() failed. at org.apache.atlas.hive.hook.HiveHook.createOrUpdateEntities(HiveHook.java:589) at org.apache.atlas.hive.hook.HiveHook.createOrUpdateEntities(HiveHook.java:595) ... 14 more Caused by: org.apache.hadoop.hive.ql.metadata.InvalidTableException: Table not found values__tmp__table__8 at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1213) at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1183)

avatar
Master Mentor

@btandel

Good to know you got that message.

Did you "Create Table As Select (CTAS) " from 2 different tables to see the lineage?