Ive always been under the impression that I can modify hive.tez.container.size for different queries within a hive script. For example: I set the container size to 2GB for small queries but increase it to 4GB for larger queries and then reset it back to 2GB for smaller queries. Is this how it works? It appears that I can definitely change this value to increase the container size at different time but does reseting the value to a lower number actually cause smaller containers from that point in the script? How can I tell what the container size for task was? ... View more

Could you remind me whats going on here in this example explain plan. The table contains 611 rows, which I see are being read. Then it appears as the key is not null predicate is applied and the number of rows drops to 306. There are no null fields in this dataset. How is this pruning data? I would have expected that to be the same as the input size. Map Operator Tree: TableScan alias: a filterExpr: key is not null (type: boolean) Statistics: Num rows: 611 Data size: 1833 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: key is not null (type: boolean) Statistics: Num rows: 306 Data size: 918 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: key (type: string) sort order: + Map-reduce partition columns: key (type: string) Statistics: Num rows: 306 Data size: 918 Basic stats: COMPLETE Column stats: NONE ... View more