Created on 09-06-2018 11:55 AM - edited 09-16-2022 06:40 AM
Hello,
I have the error message below, in which a query fails to run in a single node setup due to lack of memory left (mem-limit is 10GB). This node is also a coordinator.
I have pretty much only this log at this point and I am trying to understand it in details:
1) queue2 is set to have a capacity of only 200 queued queries, but I see way more queries than that (more than 800, although some lines were ommited due to limited message capacity) under "RequestPool=root.queue2". I'm aware that they are not using any resources, since all have Reservation=0 (except one). What would be the state of all those queries with "Reservation=0", since it should not be possible they are queued due to the limited capacity? Is there a explanation (perhaps client application not closing them properly, timed out queries, etc) about why I'm seeing a lot of queries with Reservation=0 in queue root.queue2?
2) For the single query below that has Reservation different than zero (68.00MB) in root.queue2, would that query be running? Do queued queries use any memory?
"Query(634dddb83155d4ad:4a81c27400000000): Reservation=68.00 MB ReservationLimit=8.00 GB OtherMemory=5.54 MB Total=73.54 MB Peak=73.54 MB"
3) I can see almost all of the memory usage is due to "Untracked memory". What sources of memory usage are actually classified as "Untracked" and how could I improve the amount of memory left to actually admitted/running queries?
SQLException message ExecQueryFInstances rpc query_id=11408c9da832da7a:abd592d500000000 failed: Failed to get minimum memory reservation of 170.00 MB on daemon ***** for query 11408c9da832da7a:abd592d500000000 because it would exceed an applicable memory limit. Memory is likely oversubscribed. Reducing query concurrency or configuring admission control may help avoid this error. Memory usage: Process: Limit=10.00 GB Total=9.91 GB Peak=9.94 GB Buffer Pool: Free Buffers: Total=8.00 MB Buffer Pool: Clean Pages: Total=0 Buffer Pool: Unused Reservation: Total=-68.00 MB RequestPool=root.queue1: Total=0 Peak=183.82 MB RequestPool=fe-eval-exprs: Total=0 Peak=8.00 KB RequestPool=root.queue2: Total=68.10 MB Peak=665.37 MB Query(5f43cd03daf96bb9:baa5a2d800000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=3.70 MB Query(b47085b7cadb14f:96004d3900000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(3144df4ca39c5c43:568f7bc300000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=3.70 MB Query(7b498e1fdebd888e:e5b1f12e00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(204cf000e69e59ed:fafd10c300000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.78 MB Query(7043e932b1d1466b:d98e5ede00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(8847019e3b25963b:acc6f7b400000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB Query(bd45c0213bce3393:f5f2fa0300000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB Query(ee48a57249dd4978:a102b0ad00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB Query(4c459525ed4df315:93b08e7b00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(bd48f4a0d22d59be:2619bb1f00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(284e59d6ca0cb9da:80f725f100000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.63 MB Query(86493f777c1da6bc:5eb8ea3f00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.63 MB Query(8b4d0176df0ba677:c1b4763e00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB Query(724887ee7cd3217d:a0e08de100000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=183.58 MB Query(ef41cf447eca639b:5e41b11000000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB Query(c54906ebf4abe318:ea2944e500000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.63 MB Query(7446ff9b5ac7d275:8a37741700000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB Query(e04b4bd758eca3c3:a61a09a00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.78 MB Query(ca492ca6ccc56f85:362a3e0d00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.90 MB Query(e544e5087b634651:dfcee100000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.60 MB Query(6446d35730b2be35:52a0234a00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=3.82 MB Query(9a46a2d493133a0b:6c5b60600000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.60 MB Query(75441592562116b8:cf837fb100000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=183.58 MB Query(8f4200530b5a4f1f:26800a5000000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(2c4a13912d0e935d:342d190e00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(324d69276fd8f103:ff33dcdb00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=183.58 MB Query(1845cb3beb207cde:79b4fc2300000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=3.70 MB ... Query(3e49d97834641202:3b748bc300000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.78 MB Query(98456d4ddc620b50:e646fa2800000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(844bad591e665c18:179e656900000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=3.70 MB Query(654b1c7289502ef8:ab761fef00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=3.70 MB Query(244f94b2d9e58a38:df7016d000000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(454e94c293565b3f:7fd5b73300000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(6e4c66af63a46407:8dc0c76400000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(f94942f5ab401fd8:37eb33a400000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB Query(d74eee6c82c165c9:e148100e00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=183.58 MB Query(67474e539897481b:134d286000000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=184.02 MB Query(8e436ffc341a56f9:756968d600000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(a6451bbbe49a00de:a6bf551400000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=183.58 MB Query(834fc77e54321b0a:75d2ff3000000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.63 MB Query(1347d5c5022866d3:83e8b21a00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB Query(104baefc4570b66d:21a871400000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(fa4a22b1757e6fec:632255dc00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB Query(414273acc25798cf:68cb9bf100000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(c2420a7854fe9a75:15af81db00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=183.58 MB Query(2d45cb79dcadc36b:f6891e1100000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=3.70 MB Query(81488f5bc25d1983:9a6fd6c200000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(2d476aaec2c232b2:28fe827900000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(3247f3eb0eb299b6:187d7d7f00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.78 MB Query(e346a07986e17f0e:bc190be100000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(ce4328ffffe7ab0b:c38ca81400000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(cb432a1f54e61433:390a43bb00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=184.02 MB Query(144d88900c76eb2a:6666b97200000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=183.58 MB Query(454567b5e76497ff:cb25dd200000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(cd43bd3a691fcfc4:6bac265700000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(75433800affbf182:8824864800000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(9849d62a6364fd4f:aff5bf3200000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=183.58 MB Query(ac45feb8de470759:4763018000000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(bc4a32011a1e8aec:a9c8fd6700000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(e14f73f53e1ad843:34599c3300000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.78 MB Query(fc495b5f57b4b432:4ad1a2c300000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(7b41cbbc95b09930:ba2735a700000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(b54d20d15ace82b4:84cc180e00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.63 MB Query(1d4f218d18416095:2e6f35000000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.63 MB Query(c84ca1bfef6e05c0:11fc7cc400000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=183.58 MB Query(f74940f01957fba6:baa9c7b300000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(34489b76ce317167:eb9b435e00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB Query(4541d375146603d6:fca8a3500000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=183.58 MB Query(e54c5b80063f58f5:cdd2760c00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.78 MB Query(e044d953c28aa597:c44fa83b00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(534b85a5c3f28ab2:46e10e7600000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=184.02 MB Query(b649c149d01e23c5:6c0d0cc500000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.78 MB Query(9b47608c3c41d568:2f50dd5800000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(db4e50b16905e947:7d042ef100000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(8b49df3782702aba:80ec276100000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=3.70 MB Query(d141e530a559f3d4:397cb92800000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(a04089a4c76a423c:bf44276d00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=183.58 MB Query(1c41a8d871dcf18b:a4bcf1d200000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(1c4a58e5882d91a9:c2461b4100000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(a4a835fc563d6da:6e93c5e000000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=184.02 MB Query(ef4b23761ac16731:b4c80d6300000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(ec4667b66018cf87:8eec3f3600000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(2b4bb99d536849cf:2f5874f500000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(c34f7111d29e5173:ced9042000000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=184.02 MB Query(7240e0ec3d64dad3:5d61477f00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(844a6b21b61d2960:308ed74900000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.63 MB Query(764ed77590624e32:83a947fb00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB Query(cf4de04948189969:3bc06e100000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB Query(e742b81e589c8641:788c177c00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.78 MB Query(f44790566ba9d10e:23b32ae00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(8c4ff4cf1036afd3:a17caca100000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=3.70 MB Query(1d43f7b134673c04:dbedf55900000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(2e40b0b85c13afff:785224b000000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.63 MB Query(ca44b96bf219228e:d97b3ed400000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=183.58 MB Query(94a9e42dd750cd8:1ee8e05600000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB Query(9e4854f3a6e56136:7283038800000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(be4e10120dde01a8:c36fd46e00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(f640a4953a85311f:b872d35900000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB Query(4841ade806d8003d:5ba6265700000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB Query(884e104c6c220ff9:77adf2a100000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(f341827507f1e563:d6e196bf00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=183.58 MB Query(dd4e058ac0a1ba8c:26309f1500000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=3.70 MB Query(3f492eb2666d8997:54b0d3ec00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(b84abd2f7b9de3b8:14626cc200000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=184.02 MB Query(ea451792ac9bee90:6da1ca5700000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=3.70 MB Query(ad4291ca45c30b72:72c849da00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB Query(854f135a9ff1232d:a860d64c00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(5c44fc88f7b2e114:b131332800000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(694103942e16a154:a46ce0200000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB Query(1346ba66f8f7e87f:ad4c9bf300000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(1844f51724fb7f86:f79bef5300000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.63 MB Query(7340aed6f03244c4:d30c084700000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB Query(f342689556b11af2:cd491d3500000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(8f4218b2cd8989d4:d289ac2e00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=183.58 MB Query(e94c528f3e24c46b:fb0dec5c00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(e44ee4c15fff0e4:e0bf0c0a00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.78 MB Query(6948d40a7e5cee1f:a8045e400000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB Query(d74d9bab649c2943:f58280e00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=183.58 MB Query(11460274d6a89fc6:b127d44100000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.63 MB Query(f647e01bc54fc479:d0a16cff00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=183.58 MB Query(e4e1ed9999af822:cb73767900000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=3.70 MB Query(f340d513d6f03139:31d6fcde00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(424a1ea080b57bc6:a65f71aa00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.63 MB Query(f442cce55a73b0b6:52e1703600000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=3.70 MB Query(bf4fe233bc23e43d:f68e7bfb00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=184.02 MB Query(d04f91d319a8948d:7645778700000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB Query(6c4233f5cade6a9d:a642e31600000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=183.58 MB Query(16455574b50ec681:cd7d152d00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.63 MB Query(9440174609ea6b40:668959e000000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(a34f01835217010f:b68ab44c00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(394586f404adaa17:29aebc4300000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=183.58 MB Query(ec4c37a2edec98c8:1310d64100000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(7644e786a923464f:557eed4900000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(4c4c5612214b5204:95b1291000000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.78 MB Query(6c41a54b130da058:3d6571e100000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(e54f1ebb0b0c29de:388be7a800000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(f246941730e391be:2ce6a06200000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(bc46de645ccfac17:fadefd1d00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(6c4a5f3d6e626a42:3a5a652900000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB Query(a5499684fa54ce57:a76c15d00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB Query(3748e11b8da2f89f:6e92176000000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(b846621ac3ed8e8d:c909cdf500000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.63 MB Query(cc4dd05bf1168ac4:3c269c6000000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=184.02 MB Query(344a178076b9e2fd:281013ee00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB Query(3e41d90b06cc8abf:55b960fa00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(884aaa642f96b5f6:afb986ea00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.63 MB Query(bd4e0c6c49578a24:619c3fe800000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(b244ef8dffba32e3:fd52bce500000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(bb4b7960d1d5642f:c7c412ef00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB Query(9d4c71f22e0cffa5:a13ef4db00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(d7400ba442a83dc8:f531c76b00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=184.02 MB Query(504640f25e2ba657:373a607200000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=3.70 MB Query(5461167949fe9e0:3c03f09500000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(ff49902873776e13:ae53d5c200000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB Query(9d4a9bf18f99a438:51f58d2b00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=183.58 MB Query(fd4f70dfdae1d4ae:f45b869900000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(ee45c15a953eceba:17f3bf8800000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.63 MB Query(de4831a414f5bf7f:c12fff3c00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB Query(364624a998401594:aac9fdac00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB Query(40427760aa27509a:d70532a500000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(6b45462cffb4e613:b3bce2c700000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(fc4cfc0a1b7452e7:8686849600000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB Query(14441749087ea961:7b05e57900000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(db4795ce1ece7828:c0aa533c00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(7f4d8fe3a4a799fd:aa7f584900000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.63 MB Query(e849fe3d0f4f4691:e06c79b000000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=3.70 MB Query(d3419cba926a5b08:1a93eab00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.78 MB Query(ac4bcb1729e5df89:912d4a2100000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(554f04067fc90390:ba205e7300000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(5408bd49f6bac41:ac1c4bf200000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(a446a61c499f98e8:c061887e00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(5041327d97964ba5:3559e8d400000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=3.70 MB Query(634807cdfcd7b150:f9657f9c00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(d54b7a01ffb78980:9fd96e3d00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.63 MB Query(964bcd0f07aea38c:bd85990300000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=184.02 MB Query(fc43006361b6c0b3:5a4ae6f100000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.63 MB Query(cc4660d624fa6066:550f1e5b00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(e4ed4ee9a7c393c:310fbe7c00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.63 MB Query(814072a748c93b25:b0144a200000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(f0473d41449d27c7:d4b431a800000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(3f4333ef53a760d2:61492de600000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.78 MB Query(e841b8c2df129a50:a5a6223400000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.63 MB Query(ae4484191e16018e:77f3f81100000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.78 MB Query(a14ed52a210f833d:a91dfcc800000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.63 MB Query(954592b194bfba10:44c9163700000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB Query(9d4ce2d0dfee0fa6:3bbeffb100000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=183.58 MB Query(dc45d0c88d847a6e:5d4f836e00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(3648b7e8c7f604ca:6456b57a00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.63 MB Query(3549e5e1487c3157:56fa68c900000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB Query(634cbd807aeedf8d:34196b9400000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(a4ac279688ae0d8:5dfd991200000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB Query(7b42aeba14fe0594:4769e42a00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(f84e4d07c9a9e105:9047b87200000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(7846522233d40baa:f2b25bca00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.78 MB Query(55469be2d4de9e26:722541e800000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=183.58 MB Query(464b62a9faac0f3e:9446773100000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=183.58 MB Query(9c4c22b3445eff38:7a56fed00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=3.70 MB Query(ee451869fad3b405:b14d508a00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(a74f78556e504bf7:a521067700000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(584a35e51dd45338:1d1651cb00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(524f113ac4153334:e7cb37300000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(7b4c51808e21a05a:e0e1e9f600000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(524ee7797f8c366c:1a57d7a00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(4c4425ef169a0eab:7e98714300000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=184.02 MB Query(55495f874653633d:cb37eec700000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(a04921a8a43861f0:8eb7edaf00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.63 MB Query(d24eb199e31a2d3c:2d46f9ec00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=183.58 MB Query(544a235340d32847:81ffd55f00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=3.70 MB Query(6247be727dc675db:99bf53a00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(804d736e0673b6bf:884671b000000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.63 MB Query(de44e090fd551a93:82d0c6e200000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(de4183726ee9f599:b597acec00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(df4fab35a1c74814:edb2f47b00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(124ce555361d1b42:7865990400000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.78 MB Query(874c3bfa9bd67bd5:63a25d8500000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(ec466c9c49fb9fd4:8cb325c100000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB Query(b3497a2a76e12018:e3da6ff00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(164e96bd888c1a90:491fbbec00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(2d4971784b4e89ce:d1bf6fa800000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.63 MB Query(75454b8a436074aa:17143b3400000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=183.58 MB Query(6f409e31d90482dc:ad138fd100000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(e1467b4e90e5004b:301db59b00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(ad4b33779be029bf:457a9be800000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(194129fe19af1a87:3ab5845d00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(274f9a32c7c658a8:5d31d68200000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=183.58 MB Query(544eb6720c1dac92:3198af7f00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(684ddc67c6fad15a:c3ba3bea00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.78 MB Query(f54bb563ad4e05dd:b1e0b3b900000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(764f75de17401afc:a2e3bd2a00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB Query(a748aa9a18756667:a92b3cc600000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.78 MB Query(44ec94a03ffdff:16c74f9e00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=8.00 KB Total=8.00 KB Peak=73.54 MB Query(fb4b7c060dfbdf68:9d92da5100000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=8.00 KB Total=8.00 KB Peak=73.48 MB Query(3f401bd76a149018:b428f48800000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=8.00 KB Total=8.00 KB Peak=73.48 MB Query(9f41406bd4a3be12:3a7829f600000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=16.00 KB Total=16.00 KB Peak=3.70 MB Query(e54a3b7807962ac2:63f87ca500000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=0 Query(634dddb83155d4ad:4a81c27400000000): Reservation=68.00 MB ReservationLimit=8.00 GB OtherMemory=5.54 MB Total=73.54 MB Peak=73.54 MB Query(11408c9da832da7a:abd592d500000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=0 Untracked Memory: Total=9.90 GB
Created 09-06-2018 12:47 PM
Untracked memory is really anything that isn't explicitly tracked by query execution. We track all of the large amounts of memory used by query execution - buffers for reading from disk, the actual row data, hash tables in joins, etc, etc. The untracked memory should be small relative to that - it's mostly overhead for control structures like the runtime profile and things like that. That's usually small but it could add up easily when there are lots of queries being left open.
It looks like something is unhealthy there. For one, there are a lot of queries that are still hanging around. I'd guess that there's probably a client that is misbehaving and not closing queries once it is finished with them. Those queries look like they were probably cancelled (or had all the results fetched) but were not closed by the client.
One workaround for problems like that is to set an idle session timeout to periodically clear out user sessions that are not active: https://www.cloudera.com/documentation/enterprise/latest/topics/impala_timeouts.html
Created 09-06-2018 12:47 PM
Untracked memory is really anything that isn't explicitly tracked by query execution. We track all of the large amounts of memory used by query execution - buffers for reading from disk, the actual row data, hash tables in joins, etc, etc. The untracked memory should be small relative to that - it's mostly overhead for control structures like the runtime profile and things like that. That's usually small but it could add up easily when there are lots of queries being left open.
It looks like something is unhealthy there. For one, there are a lot of queries that are still hanging around. I'd guess that there's probably a client that is misbehaving and not closing queries once it is finished with them. Those queries look like they were probably cancelled (or had all the results fetched) but were not closed by the client.
One workaround for problems like that is to set an idle session timeout to periodically clear out user sessions that are not active: https://www.cloudera.com/documentation/enterprise/latest/topics/impala_timeouts.html