Reply
Highlighted
New Contributor
Posts: 3
Registered: ‎08-30-2018
Accepted Solution

Clarification on "Failed to get minimum memory reservation" message

Hello,

 

I have the error message below, in which a query fails to run in a single node setup due to lack of memory left (mem-limit is 10GB). This node is also a coordinator.

 

I have pretty much only this log at this point and I am trying to understand it in details:


1) queue2 is set to have a capacity of only 200 queued queries, but I see way more queries than that (more than 800, although some lines were ommited due to limited message capacity) under "RequestPool=root.queue2". I'm aware that they are not using any resources, since all have Reservation=0 (except one). What would be the state of all those queries with "Reservation=0", since it should not be possible they are queued due to the limited capacity? Is there a explanation (perhaps client application not closing them properly, timed out queries, etc) about why I'm seeing a lot of queries with Reservation=0 in queue root.queue2?

 

 

2) For the single query below that has Reservation different than zero (68.00MB) in root.queue2, would that query be running? Do queued queries use any memory?

 

"Query(634dddb83155d4ad:4a81c27400000000): Reservation=68.00 MB ReservationLimit=8.00 GB OtherMemory=5.54 MB Total=73.54 MB Peak=73.54 MB"

 

 

3) I can see almost all of the memory usage is due to "Untracked memory". What sources of memory usage are actually classified as "Untracked" and how could I improve the amount of memory left to actually admitted/running queries?

 

SQLException message ExecQueryFInstances rpc query_id=11408c9da832da7a:abd592d500000000 failed: Failed to get minimum memory reservation of 170.00 MB on daemon ***** for query 11408c9da832da7a:abd592d500000000 because it would exceed an applicable memory limit. Memory is likely oversubscribed. Reducing query concurrency or configuring admission control may help avoid this error. Memory usage:
Process: Limit=10.00 GB Total=9.91 GB Peak=9.94 GB
  Buffer Pool: Free Buffers: Total=8.00 MB
  Buffer Pool: Clean Pages: Total=0
  Buffer Pool: Unused Reservation: Total=-68.00 MB
  RequestPool=root.queue1: Total=0 Peak=183.82 MB
  RequestPool=fe-eval-exprs: Total=0 Peak=8.00 KB
  RequestPool=root.queue2: Total=68.10 MB Peak=665.37 MB
    Query(5f43cd03daf96bb9:baa5a2d800000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=3.70 MB
    Query(b47085b7cadb14f:96004d3900000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(3144df4ca39c5c43:568f7bc300000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=3.70 MB
    Query(7b498e1fdebd888e:e5b1f12e00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(204cf000e69e59ed:fafd10c300000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.78 MB
    Query(7043e932b1d1466b:d98e5ede00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(8847019e3b25963b:acc6f7b400000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB
    Query(bd45c0213bce3393:f5f2fa0300000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB
    Query(ee48a57249dd4978:a102b0ad00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB
    Query(4c459525ed4df315:93b08e7b00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(bd48f4a0d22d59be:2619bb1f00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(284e59d6ca0cb9da:80f725f100000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.63 MB
    Query(86493f777c1da6bc:5eb8ea3f00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.63 MB
    Query(8b4d0176df0ba677:c1b4763e00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB
    Query(724887ee7cd3217d:a0e08de100000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=183.58 MB
    Query(ef41cf447eca639b:5e41b11000000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB
    Query(c54906ebf4abe318:ea2944e500000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.63 MB
    Query(7446ff9b5ac7d275:8a37741700000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB
    Query(e04b4bd758eca3c3:a61a09a00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.78 MB
    Query(ca492ca6ccc56f85:362a3e0d00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.90 MB
    Query(e544e5087b634651:dfcee100000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.60 MB
    Query(6446d35730b2be35:52a0234a00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=3.82 MB
    Query(9a46a2d493133a0b:6c5b60600000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.60 MB
    Query(75441592562116b8:cf837fb100000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=183.58 MB
    Query(8f4200530b5a4f1f:26800a5000000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(2c4a13912d0e935d:342d190e00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(324d69276fd8f103:ff33dcdb00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=183.58 MB
    Query(1845cb3beb207cde:79b4fc2300000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=3.70 MB
    ...
    Query(3e49d97834641202:3b748bc300000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.78 MB
    Query(98456d4ddc620b50:e646fa2800000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(844bad591e665c18:179e656900000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=3.70 MB
    Query(654b1c7289502ef8:ab761fef00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=3.70 MB
    Query(244f94b2d9e58a38:df7016d000000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(454e94c293565b3f:7fd5b73300000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(6e4c66af63a46407:8dc0c76400000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(f94942f5ab401fd8:37eb33a400000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB
    Query(d74eee6c82c165c9:e148100e00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=183.58 MB
    Query(67474e539897481b:134d286000000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=184.02 MB
    Query(8e436ffc341a56f9:756968d600000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(a6451bbbe49a00de:a6bf551400000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=183.58 MB
    Query(834fc77e54321b0a:75d2ff3000000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.63 MB
    Query(1347d5c5022866d3:83e8b21a00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB
    Query(104baefc4570b66d:21a871400000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(fa4a22b1757e6fec:632255dc00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB
    Query(414273acc25798cf:68cb9bf100000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(c2420a7854fe9a75:15af81db00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=183.58 MB
    Query(2d45cb79dcadc36b:f6891e1100000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=3.70 MB
    Query(81488f5bc25d1983:9a6fd6c200000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(2d476aaec2c232b2:28fe827900000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(3247f3eb0eb299b6:187d7d7f00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.78 MB
    Query(e346a07986e17f0e:bc190be100000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(ce4328ffffe7ab0b:c38ca81400000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(cb432a1f54e61433:390a43bb00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=184.02 MB
    Query(144d88900c76eb2a:6666b97200000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=183.58 MB
    Query(454567b5e76497ff:cb25dd200000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(cd43bd3a691fcfc4:6bac265700000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(75433800affbf182:8824864800000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(9849d62a6364fd4f:aff5bf3200000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=183.58 MB
    Query(ac45feb8de470759:4763018000000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(bc4a32011a1e8aec:a9c8fd6700000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(e14f73f53e1ad843:34599c3300000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.78 MB
    Query(fc495b5f57b4b432:4ad1a2c300000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(7b41cbbc95b09930:ba2735a700000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(b54d20d15ace82b4:84cc180e00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.63 MB
    Query(1d4f218d18416095:2e6f35000000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.63 MB
    Query(c84ca1bfef6e05c0:11fc7cc400000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=183.58 MB
    Query(f74940f01957fba6:baa9c7b300000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(34489b76ce317167:eb9b435e00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB
    Query(4541d375146603d6:fca8a3500000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=183.58 MB
    Query(e54c5b80063f58f5:cdd2760c00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.78 MB
    Query(e044d953c28aa597:c44fa83b00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(534b85a5c3f28ab2:46e10e7600000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=184.02 MB
    Query(b649c149d01e23c5:6c0d0cc500000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.78 MB
    Query(9b47608c3c41d568:2f50dd5800000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(db4e50b16905e947:7d042ef100000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(8b49df3782702aba:80ec276100000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=3.70 MB
    Query(d141e530a559f3d4:397cb92800000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(a04089a4c76a423c:bf44276d00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=183.58 MB
    Query(1c41a8d871dcf18b:a4bcf1d200000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(1c4a58e5882d91a9:c2461b4100000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(a4a835fc563d6da:6e93c5e000000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=184.02 MB
    Query(ef4b23761ac16731:b4c80d6300000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(ec4667b66018cf87:8eec3f3600000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(2b4bb99d536849cf:2f5874f500000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(c34f7111d29e5173:ced9042000000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=184.02 MB
    Query(7240e0ec3d64dad3:5d61477f00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(844a6b21b61d2960:308ed74900000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.63 MB
    Query(764ed77590624e32:83a947fb00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB
    Query(cf4de04948189969:3bc06e100000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB
    Query(e742b81e589c8641:788c177c00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.78 MB
    Query(f44790566ba9d10e:23b32ae00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(8c4ff4cf1036afd3:a17caca100000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=3.70 MB
    Query(1d43f7b134673c04:dbedf55900000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(2e40b0b85c13afff:785224b000000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.63 MB
    Query(ca44b96bf219228e:d97b3ed400000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=183.58 MB
    Query(94a9e42dd750cd8:1ee8e05600000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB
    Query(9e4854f3a6e56136:7283038800000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(be4e10120dde01a8:c36fd46e00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(f640a4953a85311f:b872d35900000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB
    Query(4841ade806d8003d:5ba6265700000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB
    Query(884e104c6c220ff9:77adf2a100000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(f341827507f1e563:d6e196bf00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=183.58 MB
    Query(dd4e058ac0a1ba8c:26309f1500000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=3.70 MB
    Query(3f492eb2666d8997:54b0d3ec00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(b84abd2f7b9de3b8:14626cc200000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=184.02 MB
    Query(ea451792ac9bee90:6da1ca5700000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=3.70 MB
    Query(ad4291ca45c30b72:72c849da00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB
    Query(854f135a9ff1232d:a860d64c00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(5c44fc88f7b2e114:b131332800000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(694103942e16a154:a46ce0200000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB
    Query(1346ba66f8f7e87f:ad4c9bf300000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(1844f51724fb7f86:f79bef5300000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.63 MB
    Query(7340aed6f03244c4:d30c084700000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB
    Query(f342689556b11af2:cd491d3500000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(8f4218b2cd8989d4:d289ac2e00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=183.58 MB
    Query(e94c528f3e24c46b:fb0dec5c00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(e44ee4c15fff0e4:e0bf0c0a00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.78 MB
    Query(6948d40a7e5cee1f:a8045e400000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB
    Query(d74d9bab649c2943:f58280e00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=183.58 MB
    Query(11460274d6a89fc6:b127d44100000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.63 MB
    Query(f647e01bc54fc479:d0a16cff00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=183.58 MB
    Query(e4e1ed9999af822:cb73767900000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=3.70 MB
    Query(f340d513d6f03139:31d6fcde00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(424a1ea080b57bc6:a65f71aa00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.63 MB
    Query(f442cce55a73b0b6:52e1703600000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=3.70 MB
    Query(bf4fe233bc23e43d:f68e7bfb00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=184.02 MB
    Query(d04f91d319a8948d:7645778700000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB
    Query(6c4233f5cade6a9d:a642e31600000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=183.58 MB
    Query(16455574b50ec681:cd7d152d00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.63 MB
    Query(9440174609ea6b40:668959e000000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(a34f01835217010f:b68ab44c00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(394586f404adaa17:29aebc4300000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=183.58 MB
    Query(ec4c37a2edec98c8:1310d64100000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(7644e786a923464f:557eed4900000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(4c4c5612214b5204:95b1291000000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.78 MB
    Query(6c41a54b130da058:3d6571e100000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(e54f1ebb0b0c29de:388be7a800000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(f246941730e391be:2ce6a06200000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(bc46de645ccfac17:fadefd1d00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(6c4a5f3d6e626a42:3a5a652900000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB
    Query(a5499684fa54ce57:a76c15d00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB
    Query(3748e11b8da2f89f:6e92176000000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(b846621ac3ed8e8d:c909cdf500000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.63 MB
    Query(cc4dd05bf1168ac4:3c269c6000000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=184.02 MB
    Query(344a178076b9e2fd:281013ee00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB
    Query(3e41d90b06cc8abf:55b960fa00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(884aaa642f96b5f6:afb986ea00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.63 MB
    Query(bd4e0c6c49578a24:619c3fe800000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(b244ef8dffba32e3:fd52bce500000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(bb4b7960d1d5642f:c7c412ef00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB
    Query(9d4c71f22e0cffa5:a13ef4db00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(d7400ba442a83dc8:f531c76b00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=184.02 MB
    Query(504640f25e2ba657:373a607200000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=3.70 MB
    Query(5461167949fe9e0:3c03f09500000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(ff49902873776e13:ae53d5c200000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB
    Query(9d4a9bf18f99a438:51f58d2b00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=183.58 MB
    Query(fd4f70dfdae1d4ae:f45b869900000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(ee45c15a953eceba:17f3bf8800000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.63 MB
    Query(de4831a414f5bf7f:c12fff3c00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB
    Query(364624a998401594:aac9fdac00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB
    Query(40427760aa27509a:d70532a500000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(6b45462cffb4e613:b3bce2c700000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(fc4cfc0a1b7452e7:8686849600000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB
    Query(14441749087ea961:7b05e57900000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(db4795ce1ece7828:c0aa533c00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(7f4d8fe3a4a799fd:aa7f584900000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.63 MB
    Query(e849fe3d0f4f4691:e06c79b000000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=3.70 MB
    Query(d3419cba926a5b08:1a93eab00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.78 MB
    Query(ac4bcb1729e5df89:912d4a2100000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(554f04067fc90390:ba205e7300000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(5408bd49f6bac41:ac1c4bf200000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(a446a61c499f98e8:c061887e00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(5041327d97964ba5:3559e8d400000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=3.70 MB
    Query(634807cdfcd7b150:f9657f9c00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(d54b7a01ffb78980:9fd96e3d00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.63 MB
    Query(964bcd0f07aea38c:bd85990300000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=184.02 MB
    Query(fc43006361b6c0b3:5a4ae6f100000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.63 MB
    Query(cc4660d624fa6066:550f1e5b00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(e4ed4ee9a7c393c:310fbe7c00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.63 MB
    Query(814072a748c93b25:b0144a200000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(f0473d41449d27c7:d4b431a800000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(3f4333ef53a760d2:61492de600000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.78 MB
    Query(e841b8c2df129a50:a5a6223400000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.63 MB
    Query(ae4484191e16018e:77f3f81100000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.78 MB
    Query(a14ed52a210f833d:a91dfcc800000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.63 MB
    Query(954592b194bfba10:44c9163700000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB
    Query(9d4ce2d0dfee0fa6:3bbeffb100000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=183.58 MB
    Query(dc45d0c88d847a6e:5d4f836e00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(3648b7e8c7f604ca:6456b57a00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.63 MB
    Query(3549e5e1487c3157:56fa68c900000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB
    Query(634cbd807aeedf8d:34196b9400000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(a4ac279688ae0d8:5dfd991200000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB
    Query(7b42aeba14fe0594:4769e42a00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(f84e4d07c9a9e105:9047b87200000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(7846522233d40baa:f2b25bca00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.78 MB
    Query(55469be2d4de9e26:722541e800000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=183.58 MB
    Query(464b62a9faac0f3e:9446773100000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=183.58 MB
    Query(9c4c22b3445eff38:7a56fed00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=3.70 MB
    Query(ee451869fad3b405:b14d508a00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(a74f78556e504bf7:a521067700000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(584a35e51dd45338:1d1651cb00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(524f113ac4153334:e7cb37300000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(7b4c51808e21a05a:e0e1e9f600000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(524ee7797f8c366c:1a57d7a00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(4c4425ef169a0eab:7e98714300000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=184.02 MB
    Query(55495f874653633d:cb37eec700000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(a04921a8a43861f0:8eb7edaf00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.63 MB
    Query(d24eb199e31a2d3c:2d46f9ec00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=183.58 MB
    Query(544a235340d32847:81ffd55f00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=3.70 MB
    Query(6247be727dc675db:99bf53a00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(804d736e0673b6bf:884671b000000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.63 MB
    Query(de44e090fd551a93:82d0c6e200000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(de4183726ee9f599:b597acec00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(df4fab35a1c74814:edb2f47b00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(124ce555361d1b42:7865990400000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.78 MB
    Query(874c3bfa9bd67bd5:63a25d8500000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(ec466c9c49fb9fd4:8cb325c100000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB
    Query(b3497a2a76e12018:e3da6ff00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(164e96bd888c1a90:491fbbec00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(2d4971784b4e89ce:d1bf6fa800000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.63 MB
    Query(75454b8a436074aa:17143b3400000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=183.58 MB
    Query(6f409e31d90482dc:ad138fd100000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(e1467b4e90e5004b:301db59b00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(ad4b33779be029bf:457a9be800000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(194129fe19af1a87:3ab5845d00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(274f9a32c7c658a8:5d31d68200000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=183.58 MB
    Query(544eb6720c1dac92:3198af7f00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(684ddc67c6fad15a:c3ba3bea00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.78 MB
    Query(f54bb563ad4e05dd:b1e0b3b900000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(764f75de17401afc:a2e3bd2a00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
    Query(a748aa9a18756667:a92b3cc600000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.78 MB
    Query(44ec94a03ffdff:16c74f9e00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=8.00 KB Total=8.00 KB Peak=73.54 MB
    Query(fb4b7c060dfbdf68:9d92da5100000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=8.00 KB Total=8.00 KB Peak=73.48 MB
    Query(3f401bd76a149018:b428f48800000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=8.00 KB Total=8.00 KB Peak=73.48 MB
    Query(9f41406bd4a3be12:3a7829f600000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=16.00 KB Total=16.00 KB Peak=3.70 MB
    Query(e54a3b7807962ac2:63f87ca500000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=0
    Query(634dddb83155d4ad:4a81c27400000000): Reservation=68.00 MB ReservationLimit=8.00 GB OtherMemory=5.54 MB Total=73.54 MB Peak=73.54 MB
    Query(11408c9da832da7a:abd592d500000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=0
  Untracked Memory: Total=9.90 GB

 

Cloudera Employee
Posts: 368
Registered: ‎07-29-2015

Re: Clarification on "Failed to get minimum memory reservation" message

Untracked memory is really anything that isn't explicitly tracked by query execution. We track all of the large amounts of memory used by query execution - buffers for reading from disk, the actual row data, hash tables in joins, etc, etc. The untracked memory should be small relative to that - it's mostly overhead for control structures like the runtime profile and things like that. That's usually small but it could add up easily when there are lots of queries being left open.

 

It looks like something is unhealthy there. For one, there are a lot of queries that are still hanging around. I'd guess that there's probably a client that is misbehaving and not closing queries once it is finished with them. Those queries look like they were probably cancelled (or had all the results fetched) but were not closed by the client.

 

One workaround for problems like that is to set an idle session timeout to periodically clear out user sessions that are not active: https://www.cloudera.com/documentation/enterprise/latest/topics/impala_timeouts.html

Announcements