Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

INVALIDATE METADATA suddenly caused: javax.jdo.JDOObjectNotFoundException: No such database row

avatar
Expert Contributor

Hi, we have partitioned table and IMPALA 1.1.1-1.p0.17

 

We have issue INVALIDATE METADATA on impalad instance. Application ehich uses partitioned table does send SQL queries to that imapald.

 

Suddenly we got an exception while executing impalad (we've implemented it as java action which connects via jdbc to target imapald)

 

 

 

Failing Oozie Launcher, Main class [mrjob.oozieaction.InvalidateMetadataAction], main() threw exception, Failed to perform Hive operation
mrjob.oozieaction.HiveOperationException: Failed to perform Hive operation
	at mrjob.oozieaction.HiveOperationBase.execute(HiveOperationBase.java:49)
	at mrjob.oozieaction.InvalidateMetadataAction.main(InvalidateMetadataAction.java:9)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:495)
	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:417)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
	at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
	at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.sql.SQLException: IllegalStateException: MetaException(message:Got exception: org.apache.hadoop.hive.metastore.api.MetaException javax.jdo.JDOObjectNotFoundException: No such database row
FailedObject:17687[OID]org.apache.hadoop.hive.metastore.model.MSerDeInfo
NestedThrowables:
org.datanucleus.exceptions.NucleusObjectNotFoundException: No such database row)
CAUSED BY: MetaException: Got exception: org.apache.hadoop.hive.metastore.api.MetaException javax.jdo.JDOObjectNotFoundException: No such database row
FailedObject:17687[OID]org.apache.hadoop.hive.metastore.model.MSerDeInfo
NestedThrowables:
org.datanucleus.exceptions.NucleusObjectNotFoundException: No such database row
	at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:159)
	at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:147)
	at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:182)
	at mrjob.oozieaction.InvalidateMetadata.doExecute(InvalidateMetadata.java:24)
	at mrjob.oozieaction.HiveOperationBase.execute(HiveOperationBase.java:45)
	... 14 more

 


 

It was working for a week without any problem.

What went wrong?

We do add partition each hour so it happened one ducring 7days*24 hours, now 10 hours passed and there are no problems.

1 ACCEPTED SOLUTION

avatar

Hi Sergey,

 

the Hive metastore uses Datanucleus for object-relational mapping. Datanucleus has a bunch of known issues, in particular, related to concurrency (e.g., HIVE-3826, HIVE-5457, HIVE-5181). From your description, I got the impression that there may be concurrent things happening while you invalidate metadata. I know it's not ideal, but I'd encourage you to avoid that, and also have a look at those JIRAs to see if you think they might apply.

I'm happy to investigate further, but for now, I'm not exactly sure what the problem is. If it is indeed a concurrencey problem, I believe simply retrying invalidate may do the trick (again not ideal, sorry).

View solution in original post

2 REPLIES 2

avatar

Hi Sergey,

 

the Hive metastore uses Datanucleus for object-relational mapping. Datanucleus has a bunch of known issues, in particular, related to concurrency (e.g., HIVE-3826, HIVE-5457, HIVE-5181). From your description, I got the impression that there may be concurrent things happening while you invalidate metadata. I know it's not ideal, but I'd encourage you to avoid that, and also have a look at those JIRAs to see if you think they might apply.

I'm happy to investigate further, but for now, I'm not exactly sure what the problem is. If it is indeed a concurrencey problem, I believe simply retrying invalidate may do the trick (again not ideal, sorry).

avatar
Expert Contributor
It's unpredictable. It works for 24 times a day for a whole week and once it fails.
Resubmitting helps.
We'll wait for new version. Thanks.