Urgent !!! - Presto Query Error: Query exceeded max memory size of 50GB

[#1]

We are getting below error, can any one please help me how i can debug the problem and identify root course of error

Presto Query Error: Query exceeded max memory size of 50GB,

Also i know the simple approach is to increase the ram allocation from 50GB but there are some concerns

1. How we can come to know what is optimum size of memory/ram allocation.

2. Suppose even if we increase it to 100GB then there is no guarantee that the user will not receive the error again of out of memory.

3. Is there any way to restrict user from launching such a huge query or any other preventive approach. OR (limit to end user that we cannot process any query which require more than 100GB or 150GB.)

Thanks,

Bhupesh Khanna

[#2]

You can take a look at this post:

https://community.hortonworks.com/articles/10485/introduction-to-presto.html

Then let me answer you question:

Q1)How we can come to know what is optimum size of memory/ram allocation.

A) You can get the value from presto UI.

http://<host-name>:8080/

All the queries are listed here. If you click one of them, you can see details , such as peak memory usage.

Q3) Is there any way to restrict user from launching such a huge query or any other preventive approach

A) Yes. ref url

config.properties example:

query.max-memory=30GB
query.max-memory-per-node=5GB

[#3]

You can spill to disk in case maximum memory per node limit is crossed.

https://docs.qubole.com/en/latest/admin-guide/engine-admin/presto-admin/spill-disk-config.html

Hope it helps!