Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Impala Catalog Crash

Impala Catalog Crash

New Contributor

Hi,

 

ICS crashes with error

 

I1107 19:37:40.663094 27456 HdfsTable.java:1147] table #rows=21513537512
I1107 19:37:40.663384 27456 HiveMetaStoreClient.java:495] Closed a connection to metastore, current connections: 5
I1107 19:37:48.803349 26643 jni-util.cc:177] java.lang.OutOfMemoryError: Requested array size exceeds VM limit
	at java.util.Arrays.copyOf(Arrays.java:2271)
	at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113)
	at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
	at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140)
	at org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:145)
	at org.apache.thrift.protocol.TBinaryProtocol.writeString(TBinaryProtocol.java:187)
	at com.cloudera.impala.thrift.THdfsPartition$THdfsPartitionStandardScheme.write(THdfsPartition.java:1831)
	at com.cloudera.impala.thrift.THdfsPartition$THdfsPartitionStandardScheme.write(THdfsPartition.java:1543)
	at com.cloudera.impala.thrift.THdfsPartition.write(THdfsPartition.java:1389)
	at com.cloudera.impala.thrift.THdfsTable$THdfsTableStandardScheme.write(THdfsTable.java:1123)
	at com.cloudera.impala.thrift.THdfsTable$THdfsTableStandardScheme.write(THdfsTable.java:969)
	at com.cloudera.impala.thrift.THdfsTable.write(THdfsTable.java:848)
	at com.cloudera.impala.thrift.TTable$TTableStandardScheme.write(TTable.java:1628)
	at com.cloudera.impala.thrift.TTable$TTableStandardScheme.write(TTable.java:1395)
	at com.cloudera.impala.thrift.TTable.write(TTable.java:1209)
	at com.cloudera.impala.thrift.TCatalogObject$TCatalogObjectStandardScheme.write(TCatalogObject.java:1241)
	at com.cloudera.impala.thrift.TCatalogObject$TCatalogObjectStandardScheme.write(TCatalogObject.java:1098)
	at com.cloudera.impala.thrift.TCatalogObject.write(TCatalogObject.java:938)
	at com.cloudera.impala.thrift.TGetAllCatalogObjectsResponse$TGetAllCatalogObjectsResponseStandardScheme.write(TGetAllCatalogObjectsResponse.java:487)
	at com.cloudera.impala.thrift.TGetAllCatalogObjectsResponse$TGetAllCatalogObjectsResponseStandardScheme.write(TGetAllCatalogObjectsResponse.java:421)
	at com.cloudera.impala.thrift.TGetAllCatalogObjectsResponse.write(TGetAllCatalogObjectsResponse.java:365)
	at org.apache.thrift.TSerializer.serialize(TSerializer.java:79)
	at com.cloudera.impala.service.JniCatalog.getCatalogObjects(JniCatalog.java:110)
I1107 19:37:48.804090 26643 status.cc:114] OutOfMemoryError: Requested array size exceeds VM limit
    @           0x79c553  (unknown)
    @           0xa684a5  (unknown)
    @           0x78c518  (unknown)
    @           0x7748a4  (unknown)
    @           0xaa518a  (unknown)
    @           0xaa7310  (unknown)
    @           0xca93b3  (unknown)
    @       0x3ebe4079d1  (unknown)
    @       0x3ebe0e88fd  (unknown)

 

It can be reproduced by clean restarting the service

and describe table_name command on two of the biggest tables with around 17000 partitions each.

 

This is not related to Java heap size, as it is the same if heap is 8 / 16 / 32 / 64 GB.

 

The suspect is that byte array use to copy thrift objects is hitting the VM limit of 2^31 -1 but I am not sure.

 

Question is, can this be fixed though any property

Is it a bug

or do we have to redesign considering these limits in the system

 

2 REPLIES 2

Re: Impala Catalog Crash

New Contributor

 

This has been taken up.

Re: Impala Catalog Crash

Explorer

How was it resolved?

Don't have an account?
Coming from Hortonworks? Activate your account here