Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Atlas hive hook fails with table not found error ?

Atlas hive hook fails with table not found error ?

Explorer

Following are hiveserver2 logs:

RROR HiveHook.java:205 - Atlas hook failed due to error java.lang.reflect.UndeclaredThrowableException at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1884) at org.apache.atlas.hive.hook.HiveHook$2.run(HiveHook.java:195) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.atlas.hook.AtlasHookException: HiveHook.registerProcess() failed. at org.apache.atlas.hive.hook.HiveHook.registerProcess(HiveHook.java:701) at org.apache.atlas.hive.hook.HiveHook.collect(HiveHook.java:268) at org.apache.atlas.hive.hook.HiveHook.access$200(HiveHook.java:83) at org.apache.atlas.hive.hook.HiveHook$2$1.run(HiveHook.java:198) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866) ... 6 more Caused by: org.apache.atlas.hook.AtlasHookException: HiveHook.processHiveEntity() failed. at org.apache.atlas.hive.hook.HiveHook.processHiveEntity(HiveHook.java:731) at org.apache.atlas.hive.hook.HiveHook.registerProcess(HiveHook.java:668) ... 12 more Caused by: org.apache.atlas.hook.AtlasHookException: HiveHook.createOrUpdateEntities() failed. at org.apache.atlas.hive.hook.HiveHook.createOrUpdateEntities(HiveHook.java:597) at org.apache.atlas.hive.hook.HiveHook.processHiveEntity(HiveHook.java:711) ... 13 more Caused by: org.apache.atlas.hook.AtlasHookException: HiveHook.createOrUpdateEntities() failed. at org.apache.atlas.hive.hook.HiveHook.createOrUpdateEntities(HiveHook.java:589) at org.apache.atlas.hive.hook.HiveHook.createOrUpdateEntities(HiveHook.java:595) ... 14 more Caused by: org.apache.hadoop.hive.ql.metadata.InvalidTableException: Table not found values__tmp__table__1 at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1213) at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1183) at org.apache.atlas.hive.hook.HiveHook.createOrUpdateEntities(HiveHook.java:568)

Any thought on how to solve this ?

7 REPLIES 7
Highlighted

Re: Atlas hive hook fails with table not found error ?

Mentor

@btandel

Have you tried running the import-hive.sh?

/usr/hdp/current/atlas-server/hook-bin/import-hive.sh

See docmentation

Highlighted

Re: Atlas hive hook fails with table not found error ?

Explorer

@Geoffrey Shelton Okot I did run that script and also it was successful. But still i see following error
ERROR AtlasHook.java:141 - Failed to send notification - attempt #2; error=java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.RecordTooLargeException: The message is 1464396 bytes when serialized which is larger than the maximum request size you have configured with the max.request.size configuration.

Highlighted

Re: Atlas hive hook fails with table not found error ?

Mentor

@btandel

You have an issue with max.request.size the records imported was larger than the configured threshold. Can you now see the tables in ATLAS?

Highlighted

Re: Atlas hive hook fails with table not found error ?

Explorer

i could see the tables but when i do some changes again in hive like dropping table then Atlas is not capturing.

Regarding max.request.size, do you have any more information.

Highlighted

Re: Atlas hive hook fails with table not found error ?

Mentor

@btandel

Did you get a message like this at the end of the import? Hive Data Model imported successfully!!!

Can you create any table using CTAS from an existing table as see the lineage in atlas?

The following hive operations are captured by hive hook currently

  • create database
  • create table/view, create table as select
  • load, import, export
  • DMLs (insert)
  • alter database
  • alter table (skewed table information, stored as, protection is not supported)
  • alter view
Highlighted

Re: Atlas hive hook fails with table not found error ?

Explorer

Hey @Geoffrey Shelton Okot I did get message Hive Data Model imported successfully!!!.

However, i am not able to capture new lineage.

I get following error,

Atlas hook failed due to error java.lang.reflect.UndeclaredThrowableException at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1884) at org.apache.atlas.hive.hook.HiveHook$2.run(HiveHook.java:195) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.atlas.hook.AtlasHookException: HiveHook.registerProcess() failed. at org.apache.atlas.hive.hook.HiveHook.registerProcess(HiveHook.java:701) at org.apache.atlas.hive.hook.HiveHook.collect(HiveHook.java:268) at org.apache.atlas.hive.hook.HiveHook.access$200(HiveHook.java:83) at org.apache.atlas.hive.hook.HiveHook$2$1.run(HiveHook.java:198) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866) ... 6 more Caused by: org.apache.atlas.hook.AtlasHookException: HiveHook.processHiveEntity() failed. at org.apache.atlas.hive.hook.HiveHook.processHiveEntity(HiveHook.java:731) at org.apache.atlas.hive.hook.HiveHook.registerProcess(HiveHook.java:668) ... 12 more Caused by: org.apache.atlas.hook.AtlasHookException: HiveHook.createOrUpdateEntities() failed. at org.apache.atlas.hive.hook.HiveHook.createOrUpdateEntities(HiveHook.java:597) at org.apache.atlas.hive.hook.HiveHook.processHiveEntity(HiveHook.java:711) ... 13 more Caused by: org.apache.atlas.hook.AtlasHookException: HiveHook.createOrUpdateEntities() failed. at org.apache.atlas.hive.hook.HiveHook.createOrUpdateEntities(HiveHook.java:589) at org.apache.atlas.hive.hook.HiveHook.createOrUpdateEntities(HiveHook.java:595) ... 14 more Caused by: org.apache.hadoop.hive.ql.metadata.InvalidTableException: Table not found values__tmp__table__8 at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1213) at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1183)

Highlighted

Re: Atlas hive hook fails with table not found error ?

Mentor

@btandel

Good to know you got that message.

Did you "Create Table As Select (CTAS) " from 2 different tables to see the lineage?

Don't have an account?
Coming from Hortonworks? Activate your account here