INVALIDATE METADATA suddenly caused: javax.jdo.JDOObjectNotFoundException: No such database row

[#1]

Hi, we have partitioned table and IMPALA 1.1.1-1.p0.17

We have issue INVALIDATE METADATA on impalad instance. Application ehich uses partitioned table does send SQL queries to that imapald.

Suddenly we got an exception while executing impalad (we've implemented it as java action which connects via jdbc to target imapald)

Failing Oozie Launcher, Main class [mrjob.oozieaction.InvalidateMetadataAction], main() threw exception, Failed to perform Hive operation
mrjob.oozieaction.HiveOperationException: Failed to perform Hive operation
	at mrjob.oozieaction.HiveOperationBase.execute(HiveOperationBase.java:49)
	at mrjob.oozieaction.InvalidateMetadataAction.main(InvalidateMetadataAction.java:9)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:495)
	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:417)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
	at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
	at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.sql.SQLException: IllegalStateException: MetaException(message:Got exception: org.apache.hadoop.hive.metastore.api.MetaException javax.jdo.JDOObjectNotFoundException: No such database row
FailedObject:17687[OID]org.apache.hadoop.hive.metastore.model.MSerDeInfo
NestedThrowables:
org.datanucleus.exceptions.NucleusObjectNotFoundException: No such database row)
CAUSED BY: MetaException: Got exception: org.apache.hadoop.hive.metastore.api.MetaException javax.jdo.JDOObjectNotFoundException: No such database row
FailedObject:17687[OID]org.apache.hadoop.hive.metastore.model.MSerDeInfo
NestedThrowables:
org.datanucleus.exceptions.NucleusObjectNotFoundException: No such database row
	at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:159)
	at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:147)
	at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:182)
	at mrjob.oozieaction.InvalidateMetadata.doExecute(InvalidateMetadata.java:24)
	at mrjob.oozieaction.HiveOperationBase.execute(HiveOperationBase.java:45)
	... 14 more

---

It was working for a week without any problem.

What went wrong?

We do add partition each hour so it happened one ducring 7days*24 hours, now 10 hours passed and there are no problems.

[#2 · Accepted solution]

Hi Sergey,

the Hive metastore uses Datanucleus for object-relational mapping. Datanucleus has a bunch of known issues, in particular, related to concurrency (e.g., HIVE-3826, HIVE-5457, HIVE-5181). From your description, I got the impression that there may be concurrent things happening while you invalidate metadata. I know it's not ideal, but I'd encourage you to avoid that, and also have a look at those JIRAs to see if you think they might apply.

I'm happy to investigate further, but for now, I'm not exactly sure what the problem is. If it is indeed a concurrencey problem, I believe simply retrying invalidate may do the trick (again not ideal, sorry).

[#3]

It's unpredictable. It works for 24 times a day for a whole week and once it fails.
Resubmitting helps.
We'll wait for new version. Thanks.