Reply
Expert Contributor
Posts: 162
Registered: ‎07-29-2013

Strage behaviour of Impala while working with metadata via JDBC

Server version: impalad version 1.1.1 RELEASE (build 83d5868f005966883a918a819a449f636a5b3d5f)

I've wrote stupid java action for using it in oozie workflow. The aim is to prepare hour aggregates using day fact data.

 

Here are queries issued via JDBC, I've put them into sysout.

See that problem happens on fulldate='2013-10-09-07' partition. It means that 00..06 partitions were created successfully!
 What does it mean? One more heisenbug or I do someting wrong? Thanks!

Here is a part of code:

 

 invalidateMetadata()
            executeQuery(prepareDropPartitionQuery(targetTable, hourPartition))
            sleep(1000) //it doesn't help :)))
            executeQuery(prepareAddPartitionQuery(targetTable, hourPartition))

            long start = System.currentTimeMillis()
            executeQuery(prepareCalculateAggregatePartitionQuery(sourceTable:sourceTable,
                                                                 targetTable:targetTable,
                                                                 dayPartition:dayPartition,
                                                                 hourPartition:hourPartition,
                                                                 fromTsInclusive:fromTsInclusive,
                                                                 toTsInclusive:toTsInclusive
                                                                ))

 

def executeQuery(String query){
        def sql = new HiveClient(jdbcUrl, user).openConnection()
        println "executing sql:\n${query}\n"
        sql.execute(query)
        sql.close()
    }

 

 

 

executing sql:
INVALIDATE METADATA

executing sql:
ALTER TABLE FLOW_HOURLY DROP IF EXISTS PARTITION (fulldate='2013-10-09-07')

executing sql:
ALTER TABLE FLOW_HOURLY ADD PARTITION (fulldate='2013-10-09-07')


<<< Invocation of Main class completed <<<

Failing Oozie Launcher, Main class [ru.lol.heatmap.oozie.action.ZoneWeightHourPartitionPreparatorAction], main() threw exception, AlreadyExistsException: Partition already exists:Partition(values:[2013-10-09-07], dbName:default, tableName:FLOW_HOURLY, createTime:0, lastAccessTime:0, sd:StorageDescriptor(cols:[FieldSchema(name:weight, type:bigint, comment:null), FieldSchema(name:tile_id, type:int, comment:null), FieldSchema(name:zone_col, type:int, comment:null), FieldSchema(name:zone_row, type:int, comment:null), FieldSchema(name:top_left_lon, type:double, comment:null), FieldSchema(name:top_left_lat, type:double, comment:null), FieldSchema(name:bottom_right_lon, type:double, comment:null), FieldSchema(name:bottom_right_lat, type:double, comment:null)], location:null, inputFormat:org.apache.hadoop.mapred.TextInputFormat, outputFormat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat, compressed:false, numBuckets:0, serdeInfo:SerDeInfo(name:null, serializationLib:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, parameters:{field.delim=	, serialization.format=	}), bucketCols:[], sortCols:[], parameters:{}, skewedInfo:SkewedInfo(skewedColNames:[], skewedColValues:[], skewedColValueLocationMaps:{}), storedAsSubDirectories:false), parameters:null)
java.sql.SQLException: AlreadyExistsException: Partition already exists:Partition(values:[2013-10-09-07], dbName:default, tableName:MOSCOW_OUTDOOR_CITIZENS_FLOW_HOURLY, createTime:0, lastAccessTime:0, sd:StorageDescriptor(cols:[FieldSchema(name:weight, type:bigint, comment:null), FieldSchema(name:tile_id, type:int, comment:null), FieldSchema(name:zone_col, type:int, comment:null), FieldSchema(name:zone_row, type:int, comment:null), FieldSchema(name:top_left_lon, type:double, comment:null), FieldSchema(name:top_left_lat, type:double, comment:null), FieldSchema(name:bottom_right_lon, type:double, comment:null), FieldSchema(name:bottom_right_lat, type:double, comment:null)], location:null, inputFormat:org.apache.hadoop.mapred.TextInputFormat, outputFormat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat, compressed:false, numBuckets:0, serdeInfo:SerDeInfo(name:null, serializationLib:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, parameters:{field.delim=	, serialization.format=	}), bucketCols:[], sortCols:[], parameters:{}, skewedInfo:SkewedInfo(skewedColNames:[], skewedColValues:[], skewedColValueLocationMaps:{}), storedAsSubDirectories:false), parameters:null)
	at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:162)
	at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:150)
	at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:196)
	at groovy.sql.Sql.execute(Sql.java:2248)
	at groovy.sql.Sql$execute.call(Unknown Source)
	at ru.lol.heatmap.sql.ZoneWeightHourPartitionPreparator.executeQuery(ZoneWeightHourPartitionPreparator.groovy:129)
	at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:361)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:909)
	at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:66)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:141)
	at ru.lol.heatmap.sql.ZoneWeightHourPartitionPreparator$_prepareWeightAggregate_closure1.doCall(ZoneWeightHourPartitionPreparator.groovy:35)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:272)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:909)
	at groovy.lang.Closure.call(Closure.java:411)
	at groovy.lang.Closure.call(Closure.java:427)
	at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:1325)
	at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:1297)
	at org.codehaus.groovy.runtime.dgm$148.invoke(Unknown Source)
	at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:271)
	at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:53)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
	at ru.lol.heatmap.sql.ZoneWeightHourPartitionPreparator.prepareWeightAggregate(ZoneWeightHourPartitionPreparator.groovy:22)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrap.invoke(PogoMetaMethodSite.java:252)
	at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.call(PogoMetaMethodSite.java:64)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:124)
	at ru.lol.heatmap.oozie.action.ZoneWeightHourPartitionPreparatorAction.main(ZoneWeightHourPartitionPreparatorAction.groovy:20)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:495)
	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:417)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
	at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
	at org.apache.hadoop.mapred.Child.main(Child.java:262)

Oozie Launcher failed, finishing Hadoop job gracefully
Highlighted
Expert Contributor
Posts: 162
Registered: ‎07-29-2013

Re: Strage behaviour of Impala while working with metadata via JDBC

Announcements