Created 04-15-2025 12:18 AM
Facing the below issue on Impala -
Failed to get minimum memory reservation of 429.94 MB on daemon hydwtbdn1.icicisecurities.corp:22000 for query b44fc02d59517c5b:40ea915f00000000 due to following error: Failed to increase reservation by 429.94 MB because it would exceed the applicable reservation limit for the "Process" ReservationTracker: reservation_limit=255.00 GB reservation=255.00 GB used_reservation=0 child_reservations=255.00 GB The top 5 queries that allocated memory under this tracker are: Query(2e4e69d2f55f0014:fe08e92f00000000): Reservation=153.90 GB ReservationLimit=240.00 GB OtherMemory=1.87 GB Total=155.77 GB Peak=155.85 GB Query(7b478b1891d4039d:8bb6141800000000): Reservation=101.08 GB ReservationLimit=240.00 GB OtherMemory=237.92 MB Total=101.31 GB Peak=101.32 GB Query(e5471eefc1b23d4c:36badde100000000): Reservation=0 ReservationLimit=240.00 GB OtherMemory=6.33 MB Total=6.33 MB Peak=925.69 MB Query(6046dfb9cc825b97:45bd0eef00000000): Reservation=0 ReservationLimit=240.00 GB OtherMemory=2.77 MB Total=2.77 MB Peak=1.66 GB Query(d04b18aee1a7fc1f:e41983ca00000000): Reservation=0 ReservationLimit=240.00 GB OtherMemory=664.13 KB Total=664.13 KB Peak=850.57 MB Memory is likely oversubscribed. Reducing query concurrency or configuring admission control may help avoid this error.
Could someone help in this please.
Created 04-15-2025 09:47 AM
The key part of error is this:
Failed to increase reservation by 429.94 MB because it would exceed the applicable reservation limit for the "Process" ReservationTracker: reservation_limit=255.00 GB reservation=255.00 GB
The Impala Daemon hydwtbdn1.icicisecurities.corp does not have enough process memory to reserve for the new query.
Basically too many queries are reserving too much memory concurrently at the same time. So this new query coming in could not run as not enough memory was available. Either reduce concurrency of queries to be able to run the query. Or set memory limits for queries so that each individual query reserves less memory.
https://docs.cloudera.com/runtime/7.3.1/impala-manage/topics/impala-admission.html
Or you can give Impala more process mem_limit in Configurations.