It looks like there was plenty of memory available in the system, that query just hit its individual memory limit. There were a lot of improvements to avoid out-of-memory between 5.15 and 6.1, particularly for queries with a lot of scans that use a significant amount of memory. It looks like one of the scans was using a large chunk of the query memory: HDFS_SCAN_NODE (id=0): Total=393.15 MB Peak=393.15 MB There's one specific regression that I'm aware of that affected Avro scans: https://issues.apache.org/jira/browse/IMPALA-7078. The fix is in 5.15.1 and 5.15.2. I don't know the file format but thought I'd flag that. The IMPALA-7078 fix actually had a few tweaks that would benefit all file formats too. So I'd suggest: Give the queries a bit more memory - in practice we've seen 2GB be a lot better with a wider variety of queries in CDH5.x. 1GB is a bit squeezy for a query with 49 operators. Pick up the 5.15.2 or 5.16.2 maintenance releases to get the fix for IMPALA-7078 - that may be enough to solve the problem. Look at CDH6.1, it does address a bunch of issues in this area more systematically - it moves the scan operations to use a much more robust memory throttling/reservation system (I spent a bunch of time last year working on problems in this general area). 1 GB might just not be enough to run a query with that many operators on the version of Impala that you're running.
... View more
