Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Strage behaviour of Impala while working with metadata via JDBC

Strage behaviour of Impala while working with metadata via JDBC

Expert Contributor

Server version: impalad version 1.1.1 RELEASE (build 83d5868f005966883a918a819a449f636a5b3d5f)

I've wrote stupid java action for using it in oozie workflow. The aim is to prepare hour aggregates using day fact data.

 

Here are queries issued via JDBC, I've put them into sysout.

See that problem happens on fulldate='2013-10-09-07' partition. It means that 00..06 partitions were created successfully!
 What does it mean? One more heisenbug or I do someting wrong? Thanks!

Here is a part of code:

 

 invalidateMetadata()
            executeQuery(prepareDropPartitionQuery(targetTable, hourPartition))
            sleep(1000) //it doesn't help :)))
            executeQuery(prepareAddPartitionQuery(targetTable, hourPartition))

            long start = System.currentTimeMillis()
            executeQuery(prepareCalculateAggregatePartitionQuery(sourceTable:sourceTable,
                                                                 targetTable:targetTable,
                                                                 dayPartition:dayPartition,
                                                                 hourPartition:hourPartition,
                                                                 fromTsInclusive:fromTsInclusive,
                                                                 toTsInclusive:toTsInclusive
                                                                ))

 

def executeQuery(String query){
        def sql = new HiveClient(jdbcUrl, user).openConnection()
        println "executing sql:\n${query}\n"
        sql.execute(query)
        sql.close()
    }

 

 

 

executing sql:
INVALIDATE METADATA

executing sql:
ALTER TABLE FLOW_HOURLY DROP IF EXISTS PARTITION (fulldate='2013-10-09-07')

executing sql:
ALTER TABLE FLOW_HOURLY ADD PARTITION (fulldate='2013-10-09-07')


<<< Invocation of Main class completed <<<

Failing Oozie Launcher, Main class [ru.lol.heatmap.oozie.action.ZoneWeightHourPartitionPreparatorAction], main() threw exception, AlreadyExistsException: Partition already exists:Partition(values:[2013-10-09-07], dbName:default, tableName:FLOW_HOURLY, createTime:0, lastAccessTime:0, sd:StorageDescriptor(cols:[FieldSchema(name:weight, type:bigint, comment:null), FieldSchema(name:tile_id, type:int, comment:null), FieldSchema(name:zone_col, type:int, comment:null), FieldSchema(name:zone_row, type:int, comment:null), FieldSchema(name:top_left_lon, type:double, comment:null), FieldSchema(name:top_left_lat, type:double, comment:null), FieldSchema(name:bottom_right_lon, type:double, comment:null), FieldSchema(name:bottom_right_lat, type:double, comment:null)], location:null, inputFormat:org.apache.hadoop.mapred.TextInputFormat, outputFormat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat, compressed:false, numBuckets:0, serdeInfo:SerDeInfo(name:null, serializationLib:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, parameters:{field.delim=	, serialization.format=	}), bucketCols:[], sortCols:[], parameters:{}, skewedInfo:SkewedInfo(skewedColNames:[], skewedColValues:[], skewedColValueLocationMaps:{}), storedAsSubDirectories:false), parameters:null)
java.sql.SQLException: AlreadyExistsException: Partition already exists:Partition(values:[2013-10-09-07], dbName:default, tableName:MOSCOW_OUTDOOR_CITIZENS_FLOW_HOURLY, createTime:0, lastAccessTime:0, sd:StorageDescriptor(cols:[FieldSchema(name:weight, type:bigint, comment:null), FieldSchema(name:tile_id, type:int, comment:null), FieldSchema(name:zone_col, type:int, comment:null), FieldSchema(name:zone_row, type:int, comment:null), FieldSchema(name:top_left_lon, type:double, comment:null), FieldSchema(name:top_left_lat, type:double, comment:null), FieldSchema(name:bottom_right_lon, type:double, comment:null), FieldSchema(name:bottom_right_lat, type:double, comment:null)], location:null, inputFormat:org.apache.hadoop.mapred.TextInputFormat, outputFormat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat, compressed:false, numBuckets:0, serdeInfo:SerDeInfo(name:null, serializationLib:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, parameters:{field.delim=	, serialization.format=	}), bucketCols:[], sortCols:[], parameters:{}, skewedInfo:SkewedInfo(skewedColNames:[], skewedColValues:[], skewedColValueLocationMaps:{}), storedAsSubDirectories:false), parameters:null)
	at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:162)
	at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:150)
	at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:196)
	at groovy.sql.Sql.execute(Sql.java:2248)
	at groovy.sql.Sql$execute.call(Unknown Source)
	at ru.lol.heatmap.sql.ZoneWeightHourPartitionPreparator.executeQuery(ZoneWeightHourPartitionPreparator.groovy:129)
	at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:361)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:909)
	at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:66)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:141)
	at ru.lol.heatmap.sql.ZoneWeightHourPartitionPreparator$_prepareWeightAggregate_closure1.doCall(ZoneWeightHourPartitionPreparator.groovy:35)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:272)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:909)
	at groovy.lang.Closure.call(Closure.java:411)
	at groovy.lang.Closure.call(Closure.java:427)
	at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:1325)
	at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:1297)
	at org.codehaus.groovy.runtime.dgm$148.invoke(Unknown Source)
	at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:271)
	at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:53)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
	at ru.lol.heatmap.sql.ZoneWeightHourPartitionPreparator.prepareWeightAggregate(ZoneWeightHourPartitionPreparator.groovy:22)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrap.invoke(PogoMetaMethodSite.java:252)
	at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.call(PogoMetaMethodSite.java:64)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:124)
	at ru.lol.heatmap.oozie.action.ZoneWeightHourPartitionPreparatorAction.main(ZoneWeightHourPartitionPreparatorAction.groovy:20)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:495)
	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:417)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
	at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
	at org.apache.hadoop.mapred.Child.main(Child.java:262)

Oozie Launcher failed, finishing Hadoop job gracefully
1 REPLY 1

Re: Strage behaviour of Impala while working with metadata via JDBC

Expert Contributor