Support Questions

Find answers, ask questions, and share your expertise

Facing issue on Impala while running a job

avatar
Frequent Visitor

Facing the below issue on Impala -

 

Failed to get minimum memory reservation of 429.94 MB on daemon hydwtbdn1.icicisecurities.corp:22000 for query b44fc02d59517c5b:40ea915f00000000 due to following error: Failed to increase reservation by 429.94 MB because it would exceed the applicable reservation limit for the "Process" ReservationTracker: reservation_limit=255.00 GB reservation=255.00 GB used_reservation=0 child_reservations=255.00 GB The top 5 queries that allocated memory under this tracker are: Query(2e4e69d2f55f0014:fe08e92f00000000): Reservation=153.90 GB ReservationLimit=240.00 GB OtherMemory=1.87 GB Total=155.77 GB Peak=155.85 GB Query(7b478b1891d4039d:8bb6141800000000): Reservation=101.08 GB ReservationLimit=240.00 GB OtherMemory=237.92 MB Total=101.31 GB Peak=101.32 GB Query(e5471eefc1b23d4c:36badde100000000): Reservation=0 ReservationLimit=240.00 GB OtherMemory=6.33 MB Total=6.33 MB Peak=925.69 MB Query(6046dfb9cc825b97:45bd0eef00000000): Reservation=0 ReservationLimit=240.00 GB OtherMemory=2.77 MB Total=2.77 MB Peak=1.66 GB Query(d04b18aee1a7fc1f:e41983ca00000000): Reservation=0 ReservationLimit=240.00 GB OtherMemory=664.13 KB Total=664.13 KB Peak=850.57 MB Memory is likely oversubscribed. Reducing query concurrency or configuring admission control may help avoid this error.

 

Could someone help in this please.

1 REPLY 1

avatar
Expert Contributor

The key part of error is this:

Failed to increase reservation by 429.94 MB because it would exceed the applicable reservation limit for the "Process" ReservationTracker: reservation_limit=255.00 GB reservation=255.00 GB

The Impala Daemon hydwtbdn1.icicisecurities.corp does not have enough process memory to reserve for the new query.

 

Basically too many queries are reserving too much memory concurrently at the same time. So this new query coming in could not run as not enough memory was available. Either reduce concurrency of queries to be able to run the query. Or set memory limits for queries so that each individual query reserves less memory.

https://docs.cloudera.com/runtime/7.3.1/impala-manage/topics/impala-admission.html

 

Or you can give Impala more process mem_limit in Configurations.