Hi,

ICS crashes with error

I1107 19:37:40.663094 27456 HdfsTable.java:1147] table #rows=21513537512
I1107 19:37:40.663384 27456 HiveMetaStoreClient.java:495] Closed a connection to metastore, current connections: 5
I1107 19:37:48.803349 26643 jni-util.cc:177] java.lang.OutOfMemoryError: Requested array size exceeds VM limit
	at java.util.Arrays.copyOf(Arrays.java:2271)
	at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113)
	at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
	at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140)
	at org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:145)
	at org.apache.thrift.protocol.TBinaryProtocol.writeString(TBinaryProtocol.java:187)
	at com.cloudera.impala.thrift.THdfsPartition$THdfsPartitionStandardScheme.write(THdfsPartition.java:1831)
	at com.cloudera.impala.thrift.THdfsPartition$THdfsPartitionStandardScheme.write(THdfsPartition.java:1543)
	at com.cloudera.impala.thrift.THdfsPartition.write(THdfsPartition.java:1389)
	at com.cloudera.impala.thrift.THdfsTable$THdfsTableStandardScheme.write(THdfsTable.java:1123)
	at com.cloudera.impala.thrift.THdfsTable$THdfsTableStandardScheme.write(THdfsTable.java:969)
	at com.cloudera.impala.thrift.THdfsTable.write(THdfsTable.java:848)
	at com.cloudera.impala.thrift.TTable$TTableStandardScheme.write(TTable.java:1628)
	at com.cloudera.impala.thrift.TTable$TTableStandardScheme.write(TTable.java:1395)
	at com.cloudera.impala.thrift.TTable.write(TTable.java:1209)
	at com.cloudera.impala.thrift.TCatalogObject$TCatalogObjectStandardScheme.write(TCatalogObject.java:1241)
	at com.cloudera.impala.thrift.TCatalogObject$TCatalogObjectStandardScheme.write(TCatalogObject.java:1098)
	at com.cloudera.impala.thrift.TCatalogObject.write(TCatalogObject.java:938)
	at com.cloudera.impala.thrift.TGetAllCatalogObjectsResponse$TGetAllCatalogObjectsResponseStandardScheme.write(TGetAllCatalogObjectsResponse.java:487)
	at com.cloudera.impala.thrift.TGetAllCatalogObjectsResponse$TGetAllCatalogObjectsResponseStandardScheme.write(TGetAllCatalogObjectsResponse.java:421)
	at com.cloudera.impala.thrift.TGetAllCatalogObjectsResponse.write(TGetAllCatalogObjectsResponse.java:365)
	at org.apache.thrift.TSerializer.serialize(TSerializer.java:79)
	at com.cloudera.impala.service.JniCatalog.getCatalogObjects(JniCatalog.java:110)
I1107 19:37:48.804090 26643 status.cc:114] OutOfMemoryError: Requested array size exceeds VM limit
    @           0x79c553  (unknown)
    @           0xa684a5  (unknown)
    @           0x78c518  (unknown)
    @           0x7748a4  (unknown)
    @           0xaa518a  (unknown)
    @           0xaa7310  (unknown)
    @           0xca93b3  (unknown)
    @       0x3ebe4079d1  (unknown)
    @       0x3ebe0e88fd  (unknown)

It can be reproduced by clean restarting the service

and describe table_name command on two of the biggest tables with around 17000 partitions each.

This is not related to Java heap size, as it is the same if heap is 8 / 16 / 32 / 64 GB.

The suspect is that byte array use to copy thrift objects is hitting the VM limit of 2^31 -1 but I am not sure.

Question is, can this be fixed though any property

Is it a bug

or do we have to redesign considering these limits in the system