INVALIDATE METADATA suddenly caused: javax.jdo.JDOObjectNotFoundException: No such database row

sergey.sheypak566881637 — Fri, 16 Sep 2022 08:49:00 GMT

Hi, we have partitioned table and IMPALA 1.1.1-1.p0.17

We have issue INVALIDATE METADATA on impalad instance. Application ehich uses partitioned table does send SQL queries to that imapald.

Suddenly we got an exception while executing impalad (we've implemented it as java action which connects via jdbc to target imapald)

Failing Oozie Launcher, Main class [mrjob.oozieaction.InvalidateMetadataAction], main() threw exception, Failed to perform Hive operation
mrjob.oozieaction.HiveOperationException: Failed to perform Hive operation
	at mrjob.oozieaction.HiveOperationBase.execute(HiveOperationBase.java:49)
	at mrjob.oozieaction.InvalidateMetadataAction.main(InvalidateMetadataAction.java:9)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:495)
	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:417)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
	at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
	at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.sql.SQLException: IllegalStateException: MetaException(message:Got exception: org.apache.hadoop.hive.metastore.api.MetaException javax.jdo.JDOObjectNotFoundException: No such database row
FailedObject:17687[OID]org.apache.hadoop.hive.metastore.model.MSerDeInfo
NestedThrowables:
org.datanucleus.exceptions.NucleusObjectNotFoundException: No such database row)
CAUSED BY: MetaException: Got exception: org.apache.hadoop.hive.metastore.api.MetaException javax.jdo.JDOObjectNotFoundException: No such database row
FailedObject:17687[OID]org.apache.hadoop.hive.metastore.model.MSerDeInfo
NestedThrowables:
org.datanucleus.exceptions.NucleusObjectNotFoundException: No such database row
	at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:159)
	at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:147)
	at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:182)
	at mrjob.oozieaction.InvalidateMetadata.doExecute(InvalidateMetadata.java:24)
	at mrjob.oozieaction.HiveOperationBase.execute(HiveOperationBase.java:45)
	... 14 more

It was working for a week without any problem.

What went wrong?

We do add partition each hour so it happened one ducring 7days*24 hours, now 10 hours passed and there are no problems.

Re: INVALIDATE METADATA suddenly caused: javax.jdo.JDOObjectNotFoundException: No such database row

alex.behm — Wed, 16 Oct 2013 19:05:49 GMT

Hi Sergey,

the Hive metastore uses Datanucleus for object-relational mapping. Datanucleus has a bunch of known issues, in particular, related to concurrency (e.g., HIVE-3826, HIVE-5457, HIVE-5181). From your description, I got the impression that there may be concurrent things happening while you invalidate metadata. I know it's not ideal, but I'd encourage you to avoid that, and also have a look at those JIRAs to see if you think they might apply.

I'm happy to investigate further, but for now, I'm not exactly sure what the problem is. If it is indeed a concurrencey problem, I believe simply retrying invalidate may do the trick (again not ideal, sorry).

Re: INVALIDATE METADATA suddenly caused: javax.jdo.JDOObjectNotFoundException: No such database row

sergey.sheypak566881637 — Thu, 17 Oct 2013 06:20:14 GMT

It's unpredictable. It works for 24 times a day for a whole week and once it fails.
Resubmitting helps.
We'll wait for new version. Thanks.

question Re: INVALIDATE METADATA suddenly caused: javax.jdo.JDOObjectNotFoundException: No such database row in Archives of Support Questions (Read Only)

INVALIDATE METADATA suddenly caused: javax.jdo.JDOObjectNotFoundException: No such database row

Re: INVALIDATE METADATA suddenly caused: javax.jdo.JDOObjectNotFoundException: No such database row

Re: INVALIDATE METADATA suddenly caused: javax.jdo.JDOObjectNotFoundException: No such database row