If you want to experiment you could try setting the query options "PREFETCH_MODE=0" and "DISABLE_STREAMING_PREAGGREGATIONS=1". You could also try adjusting NUM_SCANNER_THREADS.   With the last option, we changed our recommendation recently to set it to the # of logical cores on the machine (it was 3x logical cores originally - we changed the default in a later release). You can set the default via the "default query options" impalad startup option. ... View more

Can you share the explain plan and the query? I'm interested to know what data types and aggregate functions you're using. We're not aware of any performance regressions between those versions - we actually made a number of improvements, e.g. changing the hash table implementation and adding software prefetching. It's possible that your workload hit one of the edge cases where these changes had a detrimental effect. ... View more

Hi Sanjumani,   My guess is that it wasn't able to get enough memory due to other concurrent queries. The query consumed only 160.58MB of memory and I think probably wasn't able to get more.   If you have access to the Impala debug web UI, you can look at http://hostname:25000/queries to see what other queries are running on that coordinator, and http://hostname:25000/memz?detailed=true to see what is consuming memory on each host.   It's also good to confirm Impala's memory limit setting: you can see "mem_limit" on http://hostname:25000/varz   - Tim ... View more

Hi Akhil, It is technically possible to read files from an Impala C++ UDF, since we don't sandbox UDFs. However, I would strongly recommend against this because it can lead to resource usage problems and any mistakes can compromise the stability of Impala. My recommendation is to rethink your workflow to avoid the need for this kind of approach if possible. - Tim ... View more

That's a good point. I updated the JIRA description to provide that additional motivation.   As an open-source project, we're somewhat dependent on people finding time to pick up new features like this that are nice-to-have but not critical for many users. ... View more

This blog post provides a nice introduction to Impala's admission control:   https://blog.cloudera.com/blog/2016/12/resource-management-for-apache-impala-incubating/   There are a few ways to inspect a query's memory usage. The query profile and summary will have stats about peak memory usage per host and for each operator in the query. The stats are generally per-host, instead of cluster-wide aggregates. If you're using impala-shell, you can also "set live_summary=1" to get a live update of the query as it makes progress.   If you want to see the live state of all queries running on an Impala daemon as an admin, you can look at the "memz" tab of the Impala web UI (by default on port 25000). That will show the full tree of tracked memory from the process down to the operator level.   Cloudera Manager also has various charts of aggregate memory usage. ... View more

Hi hakki, We haven't implemented that yet. Currently I think the only way to get that info is to run "Show files". - Tim ... View more