Reply
Expert Contributor
Posts: 162
Registered: ‎07-29-2013
Accepted Solution

INVALIDATE METADATA suddenly caused: javax.jdo.JDOObjectNotFoundException: No such database row

Hi, we have partitioned table and IMPALA 1.1.1-1.p0.17

 

We have issue INVALIDATE METADATA on impalad instance. Application ehich uses partitioned table does send SQL queries to that imapald.

 

Suddenly we got an exception while executing impalad (we've implemented it as java action which connects via jdbc to target imapald)

 

 

 

Failing Oozie Launcher, Main class [mrjob.oozieaction.InvalidateMetadataAction], main() threw exception, Failed to perform Hive operation
mrjob.oozieaction.HiveOperationException: Failed to perform Hive operation
	at mrjob.oozieaction.HiveOperationBase.execute(HiveOperationBase.java:49)
	at mrjob.oozieaction.InvalidateMetadataAction.main(InvalidateMetadataAction.java:9)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:495)
	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:417)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
	at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
	at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.sql.SQLException: IllegalStateException: MetaException(message:Got exception: org.apache.hadoop.hive.metastore.api.MetaException javax.jdo.JDOObjectNotFoundException: No such database row
FailedObject:17687[OID]org.apache.hadoop.hive.metastore.model.MSerDeInfo
NestedThrowables:
org.datanucleus.exceptions.NucleusObjectNotFoundException: No such database row)
CAUSED BY: MetaException: Got exception: org.apache.hadoop.hive.metastore.api.MetaException javax.jdo.JDOObjectNotFoundException: No such database row
FailedObject:17687[OID]org.apache.hadoop.hive.metastore.model.MSerDeInfo
NestedThrowables:
org.datanucleus.exceptions.NucleusObjectNotFoundException: No such database row
	at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:159)
	at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:147)
	at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:182)
	at mrjob.oozieaction.InvalidateMetadata.doExecute(InvalidateMetadata.java:24)
	at mrjob.oozieaction.HiveOperationBase.execute(HiveOperationBase.java:45)
	... 14 more

 


 

It was working for a week without any problem.

What went wrong?

We do add partition each hour so it happened one ducring 7days*24 hours, now 10 hours passed and there are no problems.

Cloudera Employee
Posts: 307
Registered: ‎10-16-2013

Re: INVALIDATE METADATA suddenly caused: javax.jdo.JDOObjectNotFoundException: No such database row

Hi Sergey,

 

the Hive metastore uses Datanucleus for object-relational mapping. Datanucleus has a bunch of known issues, in particular, related to concurrency (e.g., HIVE-3826, HIVE-5457, HIVE-5181). From your description, I got the impression that there may be concurrent things happening while you invalidate metadata. I know it's not ideal, but I'd encourage you to avoid that, and also have a look at those JIRAs to see if you think they might apply.

I'm happy to investigate further, but for now, I'm not exactly sure what the problem is. If it is indeed a concurrencey problem, I believe simply retrying invalidate may do the trick (again not ideal, sorry).

Highlighted
Expert Contributor
Posts: 162
Registered: ‎07-29-2013

Re: INVALIDATE METADATA suddenly caused: javax.jdo.JDOObjectNotFoundException: No such database row

It's unpredictable. It works for 24 times a day for a whole week and once it fails.
Resubmitting helps.
We'll wait for new version. Thanks.
Announcements