Member since
08-30-2018
12
Posts
3
Kudos Received
0
Solutions
06-18-2019
03:30 PM
Thanks a lot, Tim!
... View more
06-18-2019
02:05 PM
Hi, I have a dashboard application (using Zoomdata) in which we submit queries to a Impala resource pool with limited number of concurrent requests. While we load the dashboard, some queries might get queued if conc. requests limit is reached. When I change the dashboard time range, I noticed the running queries are properly cancelled, but the queued ones stay in the queue and are going to execute before the new queries submitted. Do you know if there is any Impala JDBC driver property or Impala server setting to be able to cancel queued queries and prevent them from running?
... View more
Labels:
- Labels:
-
Apache Impala
03-28-2019
10:19 AM
Hi, The Impala version I'm using is 2.11, so I have those changes there. One thing I noticed is the duration, when query is submitted from impala-shell, seems to match the duration of the query reported after all rows have been fetched, but that does not seem to be the case when we actually time the duration in the client that submitted the query. As you said, probably Impala is still dealing with further processing/closing/cleanup of the query at the time the client was already able to fetch all the results of the query and print out the elapsed time. Thanks for your answer, Paulo.
... View more
03-22-2019
10:55 AM
Hi, I use Impala UI, under /queries tab, in order to get runtime metrics for queries. I'm a little bit confused about the 'duration' and 'waiting time' metrics. What is being computed and tracked in each of them? I have also timed how long it takes to run the query (since statement is submitted until all rows are fetched) from the client side (I use Ibis framework to submit queries to Impala) and I can see a much smaller runtime than the one reported in the UI. Why is that? Is perhaps the metrics reported an aggregated computing time across all impalad nodes? Thanks, Paulo.
... View more
Labels:
- Labels:
-
Apache Impala
03-12-2019
02:04 PM
Hi, If I set the following property in Impala config for admission control and pool "default", in llama-site.xml: <property> <name>llama.am.throttling.maximum.placed.reservations.root.default</name> <value>4</value> </property> Suppose I have 3 Impala nodes and all of them have the same configuration above. Does that mean I can run up to 12 concurrent queries (4 in each impalad) in default pool or just 4 overall? Thanks, Paulo.
... View more
Labels:
- Labels:
-
Apache Impala
01-23-2019
12:53 PM
1 Kudo
Thanks a lot! I just looked into IMPALA-7330 and I will consider upgrading Impala to 3.1.
... View more
01-23-2019
12:48 PM
1 Kudo
Hi, Thanks for your answer. I tried to enable debug log level while calling load data in Ibis framework and I could see a call for cmd=listStatus being done for every partition in the table, then finally a call to cmd=open, which is final step to handle load data statement. I'm doing load data passing on the table and partition. We can find mentions for listStatus command in Impala code here: https://github.com/apache/impala/search?q=listStatus&unscoped_q=listStatus. It is complicated to understand the entire thing, but I suppose those calls are being used as a "light refresh" to identify modified files in the partition. If listStatus performance depends on the number of files in the partition (but probably not its size) and that is being called for every partition in the table, I can see why the load data runtime is increasing as the chart suggests. I think you also have noticed that refresh grows A LOT as the partition size increases (for instance, in the same partition containing 10k files in my experiment it would take more than 40 seconds), so I'm somewhat happy with this workflow using load data, despite the seemingly unavoidable degradation in performance as table grows. It also can be noticed that the refresh runtime does not suffer a small jump as we add a new partition. Rather, it grows exclusively as the partition you are refreshing grows. Other workflows using Hive/direct upload into HDFS would require REFRESH, which would turn impactical long term as the partition grows. Impala version is 2.11.
... View more
01-18-2019
12:49 PM
1 Kudo
Hi,
I noticed that the time it takes for load data to run in order to load a single file into a partitioned, internal, parquet table is proportional to the partition and table size.
Then I did an experiment in which I run load time for the same parquet file (renaming it uniquely), which has around 1MB in size, until one partition was filled up with 10k files. Then, I continue filling up the next partition, until 5 partitions were full with 10k files each (full partition has around 12GB in size).
The results are the following:
It is noticeable that load data time increases slightly inside a partition and the next load data time continues from where the previous partition left.
Obs: we are using Ibis framework to do the load data.
Could you please explain a little bit about the Impala load data internals in order to justify increase in time as the table grows?
Thanks,
Paulo.
... View more
Labels:
- Labels:
-
Apache Impala
09-06-2018
11:55 AM
Hello, I have the error message below, in which a query fails to run in a single node setup due to lack of memory left (mem-limit is 10GB). This node is also a coordinator. I have pretty much only this log at this point and I am trying to understand it in details: 1) queue2 is set to have a capacity of only 200 queued queries, but I see way more queries than that (more than 800, although some lines were ommited due to limited message capacity) under "RequestPool=root.queue2". I'm aware that they are not using any resources, since all have Reservation=0 (except one). What would be the state of all those queries with "Reservation=0", since it should not be possible they are queued due to the limited capacity? Is there a explanation (perhaps client application not closing them properly, timed out queries, etc) about why I'm seeing a lot of queries with Reservation=0 in queue root.queue2? 2) For the single query below that has Reservation different than zero (68.00MB) in root.queue2, would that query be running? Do queued queries use any memory? "Query(634dddb83155d4ad:4a81c27400000000): Reservation=68.00 MB ReservationLimit=8.00 GB OtherMemory=5.54 MB Total=73.54 MB Peak=73.54 MB" 3) I can see almost all of the memory usage is due to "Untracked memory". What sources of memory usage are actually classified as "Untracked" and how could I improve the amount of memory left to actually admitted/running queries? SQLException message ExecQueryFInstances rpc query_id=11408c9da832da7a:abd592d500000000 failed: Failed to get minimum memory reservation of 170.00 MB on daemon ***** for query 11408c9da832da7a:abd592d500000000 because it would exceed an applicable memory limit. Memory is likely oversubscribed. Reducing query concurrency or configuring admission control may help avoid this error. Memory usage:
Process: Limit=10.00 GB Total=9.91 GB Peak=9.94 GB
Buffer Pool: Free Buffers: Total=8.00 MB
Buffer Pool: Clean Pages: Total=0
Buffer Pool: Unused Reservation: Total=-68.00 MB
RequestPool=root.queue1: Total=0 Peak=183.82 MB
RequestPool=fe-eval-exprs: Total=0 Peak=8.00 KB
RequestPool=root.queue2: Total=68.10 MB Peak=665.37 MB
Query(5f43cd03daf96bb9:baa5a2d800000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=3.70 MB
Query(b47085b7cadb14f:96004d3900000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(3144df4ca39c5c43:568f7bc300000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=3.70 MB
Query(7b498e1fdebd888e:e5b1f12e00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(204cf000e69e59ed:fafd10c300000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.78 MB
Query(7043e932b1d1466b:d98e5ede00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(8847019e3b25963b:acc6f7b400000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB
Query(bd45c0213bce3393:f5f2fa0300000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB
Query(ee48a57249dd4978:a102b0ad00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB
Query(4c459525ed4df315:93b08e7b00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(bd48f4a0d22d59be:2619bb1f00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(284e59d6ca0cb9da:80f725f100000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.63 MB
Query(86493f777c1da6bc:5eb8ea3f00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.63 MB
Query(8b4d0176df0ba677:c1b4763e00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB
Query(724887ee7cd3217d:a0e08de100000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=183.58 MB
Query(ef41cf447eca639b:5e41b11000000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB
Query(c54906ebf4abe318:ea2944e500000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.63 MB
Query(7446ff9b5ac7d275:8a37741700000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB
Query(e04b4bd758eca3c3:a61a09a00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.78 MB
Query(ca492ca6ccc56f85:362a3e0d00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.90 MB
Query(e544e5087b634651:dfcee100000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.60 MB
Query(6446d35730b2be35:52a0234a00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=3.82 MB
Query(9a46a2d493133a0b:6c5b60600000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.60 MB
Query(75441592562116b8:cf837fb100000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=183.58 MB
Query(8f4200530b5a4f1f:26800a5000000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(2c4a13912d0e935d:342d190e00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(324d69276fd8f103:ff33dcdb00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=183.58 MB
Query(1845cb3beb207cde:79b4fc2300000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=3.70 MB
...
Query(3e49d97834641202:3b748bc300000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.78 MB
Query(98456d4ddc620b50:e646fa2800000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(844bad591e665c18:179e656900000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=3.70 MB
Query(654b1c7289502ef8:ab761fef00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=3.70 MB
Query(244f94b2d9e58a38:df7016d000000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(454e94c293565b3f:7fd5b73300000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(6e4c66af63a46407:8dc0c76400000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(f94942f5ab401fd8:37eb33a400000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB
Query(d74eee6c82c165c9:e148100e00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=183.58 MB
Query(67474e539897481b:134d286000000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=184.02 MB
Query(8e436ffc341a56f9:756968d600000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(a6451bbbe49a00de:a6bf551400000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=183.58 MB
Query(834fc77e54321b0a:75d2ff3000000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.63 MB
Query(1347d5c5022866d3:83e8b21a00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB
Query(104baefc4570b66d:21a871400000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(fa4a22b1757e6fec:632255dc00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB
Query(414273acc25798cf:68cb9bf100000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(c2420a7854fe9a75:15af81db00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=183.58 MB
Query(2d45cb79dcadc36b:f6891e1100000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=3.70 MB
Query(81488f5bc25d1983:9a6fd6c200000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(2d476aaec2c232b2:28fe827900000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(3247f3eb0eb299b6:187d7d7f00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.78 MB
Query(e346a07986e17f0e:bc190be100000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(ce4328ffffe7ab0b:c38ca81400000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(cb432a1f54e61433:390a43bb00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=184.02 MB
Query(144d88900c76eb2a:6666b97200000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=183.58 MB
Query(454567b5e76497ff:cb25dd200000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(cd43bd3a691fcfc4:6bac265700000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(75433800affbf182:8824864800000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(9849d62a6364fd4f:aff5bf3200000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=183.58 MB
Query(ac45feb8de470759:4763018000000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(bc4a32011a1e8aec:a9c8fd6700000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(e14f73f53e1ad843:34599c3300000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.78 MB
Query(fc495b5f57b4b432:4ad1a2c300000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(7b41cbbc95b09930:ba2735a700000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(b54d20d15ace82b4:84cc180e00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.63 MB
Query(1d4f218d18416095:2e6f35000000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.63 MB
Query(c84ca1bfef6e05c0:11fc7cc400000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=183.58 MB
Query(f74940f01957fba6:baa9c7b300000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(34489b76ce317167:eb9b435e00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB
Query(4541d375146603d6:fca8a3500000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=183.58 MB
Query(e54c5b80063f58f5:cdd2760c00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.78 MB
Query(e044d953c28aa597:c44fa83b00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(534b85a5c3f28ab2:46e10e7600000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=184.02 MB
Query(b649c149d01e23c5:6c0d0cc500000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.78 MB
Query(9b47608c3c41d568:2f50dd5800000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(db4e50b16905e947:7d042ef100000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(8b49df3782702aba:80ec276100000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=3.70 MB
Query(d141e530a559f3d4:397cb92800000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(a04089a4c76a423c:bf44276d00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=183.58 MB
Query(1c41a8d871dcf18b:a4bcf1d200000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(1c4a58e5882d91a9:c2461b4100000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(a4a835fc563d6da:6e93c5e000000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=184.02 MB
Query(ef4b23761ac16731:b4c80d6300000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(ec4667b66018cf87:8eec3f3600000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(2b4bb99d536849cf:2f5874f500000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(c34f7111d29e5173:ced9042000000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=184.02 MB
Query(7240e0ec3d64dad3:5d61477f00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(844a6b21b61d2960:308ed74900000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.63 MB
Query(764ed77590624e32:83a947fb00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB
Query(cf4de04948189969:3bc06e100000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB
Query(e742b81e589c8641:788c177c00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.78 MB
Query(f44790566ba9d10e:23b32ae00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(8c4ff4cf1036afd3:a17caca100000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=3.70 MB
Query(1d43f7b134673c04:dbedf55900000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(2e40b0b85c13afff:785224b000000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.63 MB
Query(ca44b96bf219228e:d97b3ed400000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=183.58 MB
Query(94a9e42dd750cd8:1ee8e05600000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB
Query(9e4854f3a6e56136:7283038800000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(be4e10120dde01a8:c36fd46e00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(f640a4953a85311f:b872d35900000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB
Query(4841ade806d8003d:5ba6265700000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB
Query(884e104c6c220ff9:77adf2a100000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(f341827507f1e563:d6e196bf00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=183.58 MB
Query(dd4e058ac0a1ba8c:26309f1500000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=3.70 MB
Query(3f492eb2666d8997:54b0d3ec00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(b84abd2f7b9de3b8:14626cc200000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=184.02 MB
Query(ea451792ac9bee90:6da1ca5700000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=3.70 MB
Query(ad4291ca45c30b72:72c849da00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB
Query(854f135a9ff1232d:a860d64c00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(5c44fc88f7b2e114:b131332800000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(694103942e16a154:a46ce0200000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB
Query(1346ba66f8f7e87f:ad4c9bf300000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(1844f51724fb7f86:f79bef5300000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.63 MB
Query(7340aed6f03244c4:d30c084700000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB
Query(f342689556b11af2:cd491d3500000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(8f4218b2cd8989d4:d289ac2e00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=183.58 MB
Query(e94c528f3e24c46b:fb0dec5c00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(e44ee4c15fff0e4:e0bf0c0a00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.78 MB
Query(6948d40a7e5cee1f:a8045e400000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB
Query(d74d9bab649c2943:f58280e00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=183.58 MB
Query(11460274d6a89fc6:b127d44100000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.63 MB
Query(f647e01bc54fc479:d0a16cff00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=183.58 MB
Query(e4e1ed9999af822:cb73767900000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=3.70 MB
Query(f340d513d6f03139:31d6fcde00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(424a1ea080b57bc6:a65f71aa00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.63 MB
Query(f442cce55a73b0b6:52e1703600000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=3.70 MB
Query(bf4fe233bc23e43d:f68e7bfb00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=184.02 MB
Query(d04f91d319a8948d:7645778700000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB
Query(6c4233f5cade6a9d:a642e31600000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=183.58 MB
Query(16455574b50ec681:cd7d152d00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.63 MB
Query(9440174609ea6b40:668959e000000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(a34f01835217010f:b68ab44c00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(394586f404adaa17:29aebc4300000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=183.58 MB
Query(ec4c37a2edec98c8:1310d64100000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(7644e786a923464f:557eed4900000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(4c4c5612214b5204:95b1291000000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.78 MB
Query(6c41a54b130da058:3d6571e100000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(e54f1ebb0b0c29de:388be7a800000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(f246941730e391be:2ce6a06200000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(bc46de645ccfac17:fadefd1d00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(6c4a5f3d6e626a42:3a5a652900000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB
Query(a5499684fa54ce57:a76c15d00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB
Query(3748e11b8da2f89f:6e92176000000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(b846621ac3ed8e8d:c909cdf500000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.63 MB
Query(cc4dd05bf1168ac4:3c269c6000000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=184.02 MB
Query(344a178076b9e2fd:281013ee00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB
Query(3e41d90b06cc8abf:55b960fa00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(884aaa642f96b5f6:afb986ea00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.63 MB
Query(bd4e0c6c49578a24:619c3fe800000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(b244ef8dffba32e3:fd52bce500000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(bb4b7960d1d5642f:c7c412ef00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB
Query(9d4c71f22e0cffa5:a13ef4db00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(d7400ba442a83dc8:f531c76b00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=184.02 MB
Query(504640f25e2ba657:373a607200000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=3.70 MB
Query(5461167949fe9e0:3c03f09500000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(ff49902873776e13:ae53d5c200000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB
Query(9d4a9bf18f99a438:51f58d2b00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=183.58 MB
Query(fd4f70dfdae1d4ae:f45b869900000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(ee45c15a953eceba:17f3bf8800000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.63 MB
Query(de4831a414f5bf7f:c12fff3c00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB
Query(364624a998401594:aac9fdac00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB
Query(40427760aa27509a:d70532a500000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(6b45462cffb4e613:b3bce2c700000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(fc4cfc0a1b7452e7:8686849600000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB
Query(14441749087ea961:7b05e57900000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(db4795ce1ece7828:c0aa533c00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(7f4d8fe3a4a799fd:aa7f584900000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.63 MB
Query(e849fe3d0f4f4691:e06c79b000000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=3.70 MB
Query(d3419cba926a5b08:1a93eab00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.78 MB
Query(ac4bcb1729e5df89:912d4a2100000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(554f04067fc90390:ba205e7300000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(5408bd49f6bac41:ac1c4bf200000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(a446a61c499f98e8:c061887e00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(5041327d97964ba5:3559e8d400000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=3.70 MB
Query(634807cdfcd7b150:f9657f9c00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(d54b7a01ffb78980:9fd96e3d00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.63 MB
Query(964bcd0f07aea38c:bd85990300000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=184.02 MB
Query(fc43006361b6c0b3:5a4ae6f100000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.63 MB
Query(cc4660d624fa6066:550f1e5b00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(e4ed4ee9a7c393c:310fbe7c00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.63 MB
Query(814072a748c93b25:b0144a200000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(f0473d41449d27c7:d4b431a800000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(3f4333ef53a760d2:61492de600000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.78 MB
Query(e841b8c2df129a50:a5a6223400000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.63 MB
Query(ae4484191e16018e:77f3f81100000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.78 MB
Query(a14ed52a210f833d:a91dfcc800000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.63 MB
Query(954592b194bfba10:44c9163700000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB
Query(9d4ce2d0dfee0fa6:3bbeffb100000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=183.58 MB
Query(dc45d0c88d847a6e:5d4f836e00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(3648b7e8c7f604ca:6456b57a00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.63 MB
Query(3549e5e1487c3157:56fa68c900000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB
Query(634cbd807aeedf8d:34196b9400000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(a4ac279688ae0d8:5dfd991200000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB
Query(7b42aeba14fe0594:4769e42a00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(f84e4d07c9a9e105:9047b87200000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(7846522233d40baa:f2b25bca00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.78 MB
Query(55469be2d4de9e26:722541e800000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=183.58 MB
Query(464b62a9faac0f3e:9446773100000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=183.58 MB
Query(9c4c22b3445eff38:7a56fed00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=3.70 MB
Query(ee451869fad3b405:b14d508a00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(a74f78556e504bf7:a521067700000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(584a35e51dd45338:1d1651cb00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(524f113ac4153334:e7cb37300000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(7b4c51808e21a05a:e0e1e9f600000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(524ee7797f8c366c:1a57d7a00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(4c4425ef169a0eab:7e98714300000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=184.02 MB
Query(55495f874653633d:cb37eec700000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(a04921a8a43861f0:8eb7edaf00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.63 MB
Query(d24eb199e31a2d3c:2d46f9ec00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=183.58 MB
Query(544a235340d32847:81ffd55f00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=3.70 MB
Query(6247be727dc675db:99bf53a00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(804d736e0673b6bf:884671b000000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.63 MB
Query(de44e090fd551a93:82d0c6e200000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(de4183726ee9f599:b597acec00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(df4fab35a1c74814:edb2f47b00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(124ce555361d1b42:7865990400000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.78 MB
Query(874c3bfa9bd67bd5:63a25d8500000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(ec466c9c49fb9fd4:8cb325c100000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.54 MB
Query(b3497a2a76e12018:e3da6ff00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(164e96bd888c1a90:491fbbec00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(2d4971784b4e89ce:d1bf6fa800000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.63 MB
Query(75454b8a436074aa:17143b3400000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=183.58 MB
Query(6f409e31d90482dc:ad138fd100000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(e1467b4e90e5004b:301db59b00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(ad4b33779be029bf:457a9be800000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(194129fe19af1a87:3ab5845d00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(274f9a32c7c658a8:5d31d68200000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=183.58 MB
Query(544eb6720c1dac92:3198af7f00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(684ddc67c6fad15a:c3ba3bea00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.78 MB
Query(f54bb563ad4e05dd:b1e0b3b900000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(764f75de17401afc:a2e3bd2a00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.48 MB
Query(a748aa9a18756667:a92b3cc600000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=73.78 MB
Query(44ec94a03ffdff:16c74f9e00000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=8.00 KB Total=8.00 KB Peak=73.54 MB
Query(fb4b7c060dfbdf68:9d92da5100000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=8.00 KB Total=8.00 KB Peak=73.48 MB
Query(3f401bd76a149018:b428f48800000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=8.00 KB Total=8.00 KB Peak=73.48 MB
Query(9f41406bd4a3be12:3a7829f600000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=16.00 KB Total=16.00 KB Peak=3.70 MB
Query(e54a3b7807962ac2:63f87ca500000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=0
Query(634dddb83155d4ad:4a81c27400000000): Reservation=68.00 MB ReservationLimit=8.00 GB OtherMemory=5.54 MB Total=73.54 MB Peak=73.54 MB
Query(11408c9da832da7a:abd592d500000000): Reservation=0 ReservationLimit=8.00 GB OtherMemory=0 Total=0 Peak=0
Untracked Memory: Total=9.90 GB
... View more
Labels:
- Labels:
-
Apache Impala
08-30-2018
10:08 AM
Hello, We have notice that Kudu has a limitation of 300 columns per table, but unfortunately we have a table with 396 columns in our system, which violates this. We already tried to increase the column limit to 400 and ingest data into that table and haven't notice any issues and the performance looked good for us. What is the underlying reason for this limit? Could you please share some information about the known implications of exceeding this limit and the circumstances in which they can become more apparent? Do you have an example of test case (be it regarding performance or functional) which fails due to the use of tables with more than 300 columns? Is it in the Kudu roadmap to provide support for higher number of columns in the future? Are the any workarounds to use Kudu with more than 300 columns besides setting a higher maximum value? Thanks, Paulo.
... View more
Labels:
- Labels:
-
Apache Kudu
