Reply
New Contributor
Posts: 5
Registered: ‎03-08-2019

impala/kudu long query duration on small table

[ Edited ]

I m running a simple request with Impala (CDH 5.14) on a simple Kudu table (47 columns and rows number= 87):

 

[xxxx:21000] > select count(1) from MYTABLE;

87

Fetched 1 row(s) in 3.63s

 

Looking at the profile, I see high those 3s spent into KUDU MaterializeTupleTime

KUDU_SCAN_NODE (id=0):(Total: 3s539ms, non-child: 3s539ms, % non-child: 100.00%)

         - BytesRead: 0

         - CollectionItemsRead: 0 (0)

         - KuduRemoteScanTokens: 0 (0)

         - NumScannerThreadsStarted: 1 (1)

         - PeakMemoryUsage: 8.00 KB (8192)

         - RowsRead: 87 (87)

         - RowsReturned: 87 (87)

         - RowsReturnedRate: 24.00 /sec

         - ScanRangesComplete: 36 (36)

         - ScannerThreadsInvoluntaryContextSwitches: 1 (1)

         - ScannerThreadsTotalWallClockTime: 3s538ms

           - MaterializeTupleTime(*): 3s538ms

           - ScannerThreadsSysTime: 1.507ms

           - ScannerThreadsUserTime: 4.521ms

         - ScannerThreadsVoluntaryContextSwitches: 76 (76)

         - TotalKuduScanRoundTrips: 2 (2)

         - TotalReadThroughput: 0.00 /sec

 

Note that I have the following recurrent logs:

in /var/log/kudu/kudu-tserver.INFO 

I0308 21:30:04.906582 48844 maintenance_manager.cc:417] P 3da0c36839de4ba5a7500981e1b252a2: System under memory pressure (80.82% of limit used). However, there are no ops currently runnable which would free memory.

 

In /var/log/kudu/kudu-master.INFO 

I0308 22:30:40.466732 122883 catalog_manager.cc:3234] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 46071113f7194bb79707e6f8a933b61b (attempt 867)

I0308 22:30:40.466745 122883 catalog_manager.cc:2700] Scheduling retry of ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 46071113f7194bb79707e6f8a933b61b with cas_config_opid_index -1 with a delay of 60010 ms (attempt = 867)

 

But note also that if I duplicate the same  Kudu table with same data , this gives:

[xxxx:21000] > select count(1) from MYTABLE2;

87

Fetched 1 row(s) in 0.63s

 

Could these Kudu master/tablets warnings be related to my long query duration ? but why not on my duplicated table ?

Cloudera Employee
Posts: 38
Registered: ‎11-12-2013

Re: impala/kudu long query duration on small table

The warning about the system being under memory pressure but unable to free memory is concerning. What is Kudu's configured memory limit? What are the contents of /memz in the affected tserver's web UI?
New Contributor
Posts: 5
Registered: ‎03-08-2019

Re: impala/kudu long query duration on small table

Kudu's configured memory limit:

------

kudu tablet server hard memory limit: memory_limit_hard_bytes : 8 GB

kudu tablet server block cache capacity: block_cache_capacity_mb : 512 MB

Cgroup Memory Soft Limit: memory.soft_limit_in_bytes: -1 MB

Cgroup Memory Hard Limit: memory.limit_in_bytes: -1 MB

-----
here is the memz of one kudu tablet server:

------------------------------------------------
MALLOC: 6821491632 ( 6505.5 MiB) Bytes in use by application
MALLOC: + 1568858112 ( 1496.2 MiB) Bytes in page heap freelist
MALLOC: + 280368120 ( 267.4 MiB) Bytes in central cache freelist
MALLOC: + 1088000 ( 1.0 MiB) Bytes in transfer cache freelist
MALLOC: + 24813144 ( 23.7 MiB) Bytes in thread cache freelists
MALLOC: + 66846720 ( 63.8 MiB) Bytes in malloc metadata
MALLOC: ------------
MALLOC: = 8763465728 ( 8357.5 MiB) Actual memory used (physical + swap)
MALLOC: + 38330368 ( 36.6 MiB) Bytes released to OS (aka unmapped)
MALLOC: ------------
MALLOC: = 8801796096 ( 8394.0 MiB) Virtual address space used
MALLOC:
MALLOC: 324195 Spans in use
MALLOC: 127 Thread heaps in use
MALLOC: 8192 Tcmalloc page size
------------------------------------------------
Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()).
Bytes released to the OS take up virtual address space but no physical memory.
------------------------------------------------
Total size of freelists for per-thread caches,
transfer cache, and central cache, by size class
------------------------------------------------
class 1 [ 8 bytes ] : 1494642 objs; 11.4 MiB; 11.4 **bleep** MiB
class 2 [ 16 bytes ] : 120351 objs; 1.8 MiB; 13.2 **bleep** MiB
class 3 [ 32 bytes ] : 404732 objs; 12.4 MiB; 25.6 **bleep** MiB
class 4 [ 48 bytes ] : 682384 objs; 31.2 MiB; 56.8 **bleep** MiB
class 5 [ 64 bytes ] : 842967 objs; 51.5 MiB; 108.3 **bleep** MiB
class 6 [ 80 bytes ] : 42433 objs; 3.2 MiB; 111.5 **bleep** MiB
class 7 [ 96 bytes ] : 251924 objs; 23.1 MiB; 134.6 **bleep** MiB
class 8 [ 112 bytes ] : 61161 objs; 6.5 MiB; 141.1 **bleep** MiB
class 9 [ 128 bytes ] : 65254 objs; 8.0 MiB; 149.1 **bleep** MiB
class 10 [ 144 bytes ] : 252404 objs; 34.7 MiB; 183.7 **bleep** MiB
class 11 [ 160 bytes ] : 12018 objs; 1.8 MiB; 185.6 **bleep** MiB
class 12 [ 176 bytes ] : 23561 objs; 4.0 MiB; 189.5 **bleep** MiB
class 13 [ 192 bytes ] : 7449 objs; 1.4 MiB; 190.9 **bleep** MiB
class 14 [ 208 bytes ] : 7221 objs; 1.4 MiB; 192.3 **bleep** MiB
class 15 [ 224 bytes ] : 4839 objs; 1.0 MiB; 193.4 **bleep** MiB
class 16 [ 240 bytes ] : 24532 objs; 5.6 MiB; 199.0 **bleep** MiB
class 17 [ 256 bytes ] : 13242 objs; 3.2 MiB; 202.2 **bleep** MiB
class 18 [ 288 bytes ] : 5592 objs; 1.5 MiB; 203.7 **bleep** MiB
class 19 [ 320 bytes ] : 3453 objs; 1.1 MiB; 204.8 **bleep** MiB
class 20 [ 352 bytes ] : 2278 objs; 0.8 MiB; 205.6 **bleep** MiB
class 21 [ 384 bytes ] : 999 objs; 0.4 MiB; 205.9 **bleep** MiB
class 22 [ 416 bytes ] : 12805 objs; 5.1 MiB; 211.0 **bleep** MiB
class 23 [ 448 bytes ] : 1048 objs; 0.4 MiB; 211.5 **bleep** MiB
class 24 [ 480 bytes ] : 983 objs; 0.4 MiB; 211.9 **bleep** MiB
class 25 [ 512 bytes ] : 29620 objs; 14.5 MiB; 226.4 **bleep** MiB
class 26 [ 576 bytes ] : 2825 objs; 1.6 MiB; 227.9 **bleep** MiB
class 27 [ 640 bytes ] : 10795 objs; 6.6 MiB; 234.5 **bleep** MiB
class 28 [ 704 bytes ] : 757 objs; 0.5 MiB; 235.0 **bleep** MiB
class 29 [ 768 bytes ] : 439 objs; 0.3 MiB; 235.3 **bleep** MiB
class 30 [ 896 bytes ] : 1092 objs; 0.9 MiB; 236.3 **bleep** MiB
class 31 [ 1024 bytes ] : 1499 objs; 1.5 MiB; 237.7 **bleep** MiB
class 32 [ 1152 bytes ] : 588 objs; 0.6 MiB; 238.4 **bleep** MiB
class 33 [ 1280 bytes ] : 353 objs; 0.4 MiB; 238.8 **bleep** MiB
class 34 [ 1408 bytes ] : 164 objs; 0.2 MiB; 239.0 **bleep** MiB
class 35 [ 1536 bytes ] : 212 objs; 0.3 MiB; 239.3 **bleep** MiB
class 36 [ 1792 bytes ] : 1318 objs; 2.3 MiB; 241.6 **bleep** MiB
class 37 [ 2048 bytes ] : 500 objs; 1.0 MiB; 242.6 **bleep** MiB
class 38 [ 2304 bytes ] : 265 objs; 0.6 MiB; 243.2 **bleep** MiB
class 39 [ 2560 bytes ] : 270 objs; 0.7 MiB; 243.8 **bleep** MiB
class 40 [ 2816 bytes ] : 81 objs; 0.2 MiB; 244.0 **bleep** MiB
class 41 [ 3072 bytes ] : 54 objs; 0.2 MiB; 244.2 **bleep** MiB
class 42 [ 3328 bytes ] : 121 objs; 0.4 MiB; 244.6 **bleep** MiB
class 43 [ 4096 bytes ] : 552 objs; 2.2 MiB; 246.7 **bleep** MiB
class 44 [ 4608 bytes ] : 2507 objs; 11.0 MiB; 257.7 **bleep** MiB
class 45 [ 5120 bytes ] : 110 objs; 0.5 MiB; 258.3 **bleep** MiB
class 46 [ 6144 bytes ] : 51 objs; 0.3 MiB; 258.6 **bleep** MiB
class 47 [ 6656 bytes ] : 37 objs; 0.2 MiB; 258.8 **bleep** MiB
class 48 [ 8192 bytes ] : 138 objs; 1.1 MiB; 259.9 **bleep** MiB
class 49 [ 9216 bytes ] : 22 objs; 0.2 MiB; 260.1 **bleep** MiB
class 50 [ 10240 bytes ] : 134 objs; 1.3 MiB; 261.4 **bleep** MiB
class 51 [ 12288 bytes ] : 47 objs; 0.6 MiB; 261.9 **bleep** MiB
class 52 [ 13312 bytes ] : 2 objs; 0.0 MiB; 262.0 **bleep** MiB
class 53 [ 16384 bytes ] : 5 objs; 0.1 MiB; 262.1 **bleep** MiB
class 54 [ 20480 bytes ] : 51 objs; 1.0 MiB; 263.0 **bleep** MiB
class 55 [ 24576 bytes ] : 5 objs; 0.1 MiB; 263.2 **bleep** MiB
class 56 [ 26624 bytes ] : 6 objs; 0.2 MiB; 263.3 **bleep** MiB
class 57 [ 32768 bytes ] : 48 objs; 1.5 MiB; 264.8 **bleep** MiB
class 58 [ 40960 bytes ] : 11 objs; 0.4 MiB; 265.2 **bleep** MiB
class 60 [ 57344 bytes ] : 2 objs; 0.1 MiB; 265.4 **bleep** MiB
class 61 [ 65536 bytes ] : 3 objs; 0.2 MiB; 265.5 **bleep** MiB
class 62 [ 73728 bytes ] : 8 objs; 0.6 MiB; 266.1 **bleep** MiB
class 63 [ 81920 bytes ] : 5 objs; 0.4 MiB; 266.5 **bleep** MiB
class 65 [ 98304 bytes ] : 1 objs; 0.1 MiB; 266.6 **bleep** MiB
class 66 [ 106496 bytes ] : 2 objs; 0.2 MiB; 266.8 **bleep** MiB
class 67 [ 114688 bytes ] : 5 objs; 0.5 MiB; 267.3 **bleep** MiB
class 68 [ 122880 bytes ] : 1 objs; 0.1 MiB; 267.5 **bleep** MiB
class 69 [ 131072 bytes ] : 3 objs; 0.4 MiB; 267.8 **bleep** MiB
class 71 [ 147456 bytes ] : 1 objs; 0.1 MiB; 268.0 **bleep** MiB
class 72 [ 155648 bytes ] : 1 objs; 0.1 MiB; 268.1 **bleep** MiB
class 79 [ 212992 bytes ] : 1 objs; 0.2 MiB; 268.3 **bleep** MiB
class 80 [ 221184 bytes ] : 1 objs; 0.2 MiB; 268.5 **bleep** MiB
class 85 [ 262144 bytes ] : 1 objs; 0.2 MiB; 268.8 **bleep** MiB
------------------------------------------------
PageHeap: 86 sizes; 1496.2 MiB free; 36.6 MiB unmapped
------------------------------------------------
1 pages * 5545 spans ~ 43.3 MiB; 43.3 MiB **bleep**; unmapped: 0.1 MiB; 0.1 MiB **bleep**
2 pages * 2062 spans ~ 32.2 MiB; 75.5 MiB **bleep**; unmapped: 8.0 MiB; 8.1 MiB **bleep**
3 pages * 1967 spans ~ 46.1 MiB; 121.6 MiB **bleep**; unmapped: 11.8 MiB; 19.9 MiB **bleep**
4 pages * 1271 spans ~ 39.7 MiB; 161.4 MiB **bleep**; unmapped: 0.0 MiB; 19.9 MiB **bleep**
5 pages * 919 spans ~ 35.9 MiB; 197.3 MiB **bleep**; unmapped: 0.2 MiB; 20.1 MiB **bleep**
6 pages * 422 spans ~ 19.8 MiB; 217.0 MiB **bleep**; unmapped: 0.1 MiB; 20.2 MiB **bleep**
7 pages * 239 spans ~ 13.1 MiB; 230.1 MiB **bleep**; unmapped: 0.3 MiB; 20.5 MiB **bleep**
8 pages * 260 spans ~ 16.2 MiB; 246.4 MiB **bleep**; unmapped: 0.2 MiB; 20.7 MiB **bleep**
9 pages * 178 spans ~ 12.5 MiB; 258.9 MiB **bleep**; unmapped: 0.0 MiB; 20.7 MiB **bleep**
10 pages * 173 spans ~ 13.5 MiB; 272.4 MiB **bleep**; unmapped: 0.0 MiB; 20.7 MiB **bleep**
11 pages * 73 spans ~ 6.3 MiB; 278.7 MiB **bleep**; unmapped: 0.0 MiB; 20.7 MiB **bleep**
12 pages * 103 spans ~ 9.7 MiB; 288.3 MiB **bleep**; unmapped: 0.0 MiB; 20.7 MiB **bleep**
13 pages * 70 spans ~ 7.1 MiB; 295.4 MiB **bleep**; unmapped: 0.0 MiB; 20.7 MiB **bleep**
14 pages * 41 spans ~ 4.5 MiB; 299.9 MiB **bleep**; unmapped: 0.0 MiB; 20.7 MiB **bleep**
15 pages * 96 spans ~ 11.2 MiB; 311.2 MiB **bleep**; unmapped: 0.0 MiB; 20.7 MiB **bleep**
16 pages * 51 spans ~ 6.4 MiB; 317.5 MiB **bleep**; unmapped: 0.0 MiB; 20.7 MiB **bleep**
17 pages * 46 spans ~ 6.1 MiB; 323.6 MiB **bleep**; unmapped: 0.0 MiB; 20.7 MiB **bleep**
18 pages * 18 spans ~ 2.5 MiB; 326.2 MiB **bleep**; unmapped: 0.0 MiB; 20.7 MiB **bleep**
19 pages * 10 spans ~ 1.5 MiB; 327.7 MiB **bleep**; unmapped: 0.0 MiB; 20.7 MiB **bleep**
20 pages * 3 spans ~ 0.5 MiB; 328.1 MiB **bleep**; unmapped: 0.0 MiB; 20.7 MiB **bleep**
21 pages * 6 spans ~ 1.0 MiB; 329.1 MiB **bleep**; unmapped: 0.0 MiB; 20.7 MiB **bleep**
22 pages * 3 spans ~ 0.5 MiB; 329.6 MiB **bleep**; unmapped: 0.0 MiB; 20.7 MiB **bleep**
24 pages * 2 spans ~ 0.4 MiB; 330.0 MiB **bleep**; unmapped: 0.0 MiB; 20.7 MiB **bleep**
25 pages * 1 spans ~ 0.2 MiB; 330.2 MiB **bleep**; unmapped: 0.0 MiB; 20.7 MiB **bleep**
26 pages * 3 spans ~ 0.6 MiB; 330.8 MiB **bleep**; unmapped: 0.4 MiB; 21.1 MiB **bleep**
27 pages * 2 spans ~ 0.4 MiB; 331.2 MiB **bleep**; unmapped: 0.0 MiB; 21.1 MiB **bleep**
29 pages * 1 spans ~ 0.2 MiB; 331.5 MiB **bleep**; unmapped: 0.0 MiB; 21.1 MiB **bleep**
30 pages * 1 spans ~ 0.2 MiB; 331.7 MiB **bleep**; unmapped: 0.0 MiB; 21.1 MiB **bleep**
32 pages * 16 spans ~ 4.0 MiB; 335.7 MiB **bleep**; unmapped: 0.0 MiB; 21.1 MiB **bleep**
33 pages * 13 spans ~ 3.4 MiB; 339.0 MiB **bleep**; unmapped: 0.0 MiB; 21.1 MiB **bleep**
34 pages * 10 spans ~ 2.7 MiB; 341.7 MiB **bleep**; unmapped: 0.0 MiB; 21.1 MiB **bleep**
35 pages * 1 spans ~ 0.3 MiB; 342.0 MiB **bleep**; unmapped: 0.0 MiB; 21.1 MiB **bleep**
36 pages * 15 spans ~ 4.2 MiB; 346.2 MiB **bleep**; unmapped: 0.0 MiB; 21.1 MiB **bleep**
37 pages * 5 spans ~ 1.4 MiB; 347.6 MiB **bleep**; unmapped: 0.0 MiB; 21.1 MiB **bleep**
38 pages * 7 spans ~ 2.1 MiB; 349.7 MiB **bleep**; unmapped: 0.0 MiB; 21.1 MiB **bleep**
39 pages * 3 spans ~ 0.9 MiB; 350.6 MiB **bleep**; unmapped: 0.0 MiB; 21.1 MiB **bleep**
40 pages * 13 spans ~ 4.1 MiB; 354.7 MiB **bleep**; unmapped: 0.0 MiB; 21.1 MiB **bleep**
44 pages * 12 spans ~ 4.1 MiB; 358.8 MiB **bleep**; unmapped: 0.0 MiB; 21.1 MiB **bleep**
45 pages * 4 spans ~ 1.4 MiB; 360.2 MiB **bleep**; unmapped: 0.0 MiB; 21.1 MiB **bleep**
46 pages * 5 spans ~ 1.8 MiB; 362.0 MiB **bleep**; unmapped: 0.0 MiB; 21.1 MiB **bleep**
47 pages * 4 spans ~ 1.5 MiB; 363.5 MiB **bleep**; unmapped: 0.0 MiB; 21.1 MiB **bleep**
48 pages * 13 spans ~ 4.9 MiB; 368.4 MiB **bleep**; unmapped: 0.0 MiB; 21.1 MiB **bleep**
49 pages * 7 spans ~ 2.7 MiB; 371.0 MiB **bleep**; unmapped: 0.0 MiB; 21.1 MiB **bleep**
50 pages * 5 spans ~ 2.0 MiB; 373.0 MiB **bleep**; unmapped: 0.0 MiB; 21.1 MiB **bleep**
51 pages * 4 spans ~ 1.6 MiB; 374.6 MiB **bleep**; unmapped: 0.0 MiB; 21.1 MiB **bleep**
52 pages * 5 spans ~ 2.0 MiB; 376.6 MiB **bleep**; unmapped: 0.0 MiB; 21.1 MiB **bleep**
53 pages * 7 spans ~ 2.9 MiB; 379.5 MiB **bleep**; unmapped: 0.0 MiB; 21.1 MiB **bleep**
54 pages * 7 spans ~ 3.0 MiB; 382.5 MiB **bleep**; unmapped: 0.0 MiB; 21.1 MiB **bleep**
55 pages * 4 spans ~ 1.7 MiB; 384.2 MiB **bleep**; unmapped: 0.0 MiB; 21.1 MiB **bleep**
56 pages * 6 spans ~ 2.6 MiB; 386.8 MiB **bleep**; unmapped: 0.0 MiB; 21.1 MiB **bleep**
57 pages * 3 spans ~ 1.3 MiB; 388.2 MiB **bleep**; unmapped: 0.0 MiB; 21.1 MiB **bleep**
58 pages * 4 spans ~ 1.8 MiB; 390.0 MiB **bleep**; unmapped: 0.0 MiB; 21.1 MiB **bleep**
59 pages * 1 spans ~ 0.5 MiB; 390.4 MiB **bleep**; unmapped: 0.0 MiB; 21.1 MiB **bleep**
60 pages * 8 spans ~ 3.8 MiB; 394.2 MiB **bleep**; unmapped: 0.5 MiB; 21.6 MiB **bleep**
61 pages * 7 spans ~ 3.3 MiB; 397.5 MiB **bleep**; unmapped: 0.5 MiB; 22.0 MiB **bleep**
62 pages * 2 spans ~ 1.0 MiB; 398.5 MiB **bleep**; unmapped: 0.0 MiB; 22.0 MiB **bleep**
64 pages * 2 spans ~ 1.0 MiB; 399.5 MiB **bleep**; unmapped: 0.0 MiB; 22.0 MiB **bleep**
65 pages * 2 spans ~ 1.0 MiB; 400.5 MiB **bleep**; unmapped: 0.0 MiB; 22.0 MiB **bleep**
66 pages * 1 spans ~ 0.5 MiB; 401.0 MiB **bleep**; unmapped: 0.0 MiB; 22.0 MiB **bleep**
69 pages * 2 spans ~ 1.1 MiB; 402.1 MiB **bleep**; unmapped: 0.0 MiB; 22.0 MiB **bleep**
73 pages * 1 spans ~ 0.6 MiB; 402.7 MiB **bleep**; unmapped: 0.0 MiB; 22.0 MiB **bleep**
76 pages * 1 spans ~ 0.6 MiB; 403.3 MiB **bleep**; unmapped: 0.0 MiB; 22.0 MiB **bleep**
78 pages * 1 spans ~ 0.6 MiB; 403.9 MiB **bleep**; unmapped: 0.0 MiB; 22.0 MiB **bleep**
85 pages * 3 spans ~ 2.0 MiB; 405.9 MiB **bleep**; unmapped: 0.0 MiB; 22.0 MiB **bleep**
86 pages * 1 spans ~ 0.7 MiB; 406.5 MiB **bleep**; unmapped: 0.0 MiB; 22.0 MiB **bleep**
88 pages * 3 spans ~ 2.1 MiB; 408.6 MiB **bleep**; unmapped: 0.0 MiB; 22.0 MiB **bleep**
89 pages * 4 spans ~ 2.8 MiB; 411.4 MiB **bleep**; unmapped: 2.1 MiB; 24.1 MiB **bleep**
90 pages * 2 spans ~ 1.4 MiB; 412.8 MiB **bleep**; unmapped: 1.4 MiB; 25.5 MiB **bleep**
92 pages * 1 spans ~ 0.7 MiB; 413.5 MiB **bleep**; unmapped: 0.7 MiB; 26.3 MiB **bleep**
93 pages * 3 spans ~ 2.2 MiB; 415.7 MiB **bleep**; unmapped: 0.7 MiB; 27.0 MiB **bleep**
94 pages * 2 spans ~ 1.5 MiB; 417.1 MiB **bleep**; unmapped: 0.7 MiB; 27.7 MiB **bleep**
96 pages * 3 spans ~ 2.2 MiB; 419.4 MiB **bleep**; unmapped: 0.0 MiB; 27.7 MiB **bleep**
97 pages * 4 spans ~ 3.0 MiB; 422.4 MiB **bleep**; unmapped: 3.0 MiB; 30.8 MiB **bleep**
100 pages * 5 spans ~ 3.9 MiB; 426.3 MiB **bleep**; unmapped: 0.8 MiB; 31.5 MiB **bleep**
101 pages * 1 spans ~ 0.8 MiB; 427.1 MiB **bleep**; unmapped: 0.8 MiB; 32.3 MiB **bleep**
104 pages * 5 spans ~ 4.1 MiB; 431.2 MiB **bleep**; unmapped: 0.8 MiB; 33.1 MiB **bleep**
106 pages * 1 spans ~ 0.8 MiB; 432.0 MiB **bleep**; unmapped: 0.8 MiB; 34.0 MiB **bleep**
110 pages * 2 spans ~ 1.7 MiB; 433.7 MiB **bleep**; unmapped: 1.7 MiB; 35.7 MiB **bleep**
111 pages * 1 spans ~ 0.9 MiB; 434.6 MiB **bleep**; unmapped: 0.0 MiB; 35.7 MiB **bleep**
112 pages * 2 spans ~ 1.8 MiB; 436.4 MiB **bleep**; unmapped: 0.9 MiB; 36.6 MiB **bleep**
113 pages * 1 spans ~ 0.9 MiB; 437.2 MiB **bleep**; unmapped: 0.0 MiB; 36.6 MiB **bleep**
115 pages * 2 spans ~ 1.8 MiB; 439.0 MiB **bleep**; unmapped: 0.0 MiB; 36.6 MiB **bleep**
119 pages * 2 spans ~ 1.9 MiB; 440.9 MiB **bleep**; unmapped: 0.0 MiB; 36.6 MiB **bleep**
123 pages * 1 spans ~ 1.0 MiB; 441.9 MiB **bleep**; unmapped: 0.0 MiB; 36.6 MiB **bleep**
125 pages * 2 spans ~ 2.0 MiB; 443.8 MiB **bleep**; unmapped: 0.0 MiB; 36.6 MiB **bleep**
127 pages * 1 spans ~ 1.0 MiB; 444.8 MiB **bleep**; unmapped: 0.0 MiB; 36.6 MiB **bleep**
>255 large * 36 spans ~ 1087.9 MiB; 1532.7 MiB **bleep**; unmapped: 0.0 MiB; 36.6 MiB **bleep**

 

Cloudera Employee
Posts: 38
Registered: ‎11-12-2013

Re: impala/kudu long query duration on small table

What are the contents of the "FS layout report", dumped to the tserver log when it starts up?
Highlighted
New Contributor
Posts: 5
Registered: ‎03-08-2019

Re: impala/kudu long query duration on small table

After Kudu restart:

 

Tablet server version:
kudu 1.7.0-cdh5.16.1
revision 4ad36868c5d782efb3eb5339c12a33b6824cce79
build type RELEASE
built by jenkins at 21 Nov 2018 22:13:38 PST on impala-ec2-pkg-centos-7-031c.vpc.cloudera.com
build id 2018-11-21_21-08-25
I0312 16:01:23.652084 249898 minidump.cc:237] Setting minidump size limit to 20M
I0312 16:01:23.652737 249898 tablet_server_main.cc:79] Initializing tablet server...
I0312 16:01:23.944438 249898 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 1102679us
I0312 16:01:23.944476 249898 fs_manager.cc:260] Metadata directory not provided
I0312 16:01:23.944489 249898 fs_manager.cc:263] Using existing metadata directory in first data directory
I0312 16:01:24.058754 249898 fs_manager.cc:397] Time spent opening directory manager: real 0.100s user 0.000s sys 0.000s
I0312 16:01:29.870604 249898 env_posix.cc:1634] Not raising this process' open files per process limit of 65536; it is already as high as it can go
I0312 16:01:29.870658 249898 file_cache.cc:470] Constructed file cache lbm with capacity 26214
I0312 16:01:39.967597 249961 log_block_manager.cc:2367] Opened 684 log block containers in /data/kudu/data
I0312 16:01:50.170301 249961 log_block_manager.cc:2367] Opened 791 log block containers in /data/kudu/data
I0312 16:02:00.308365 249961 log_block_manager.cc:2367] Opened 852 log block containers in /data/kudu/data
I0312 16:02:10.370048 249961 log_block_manager.cc:2367] Opened 921 log block containers in /data/kudu/data
I0312 16:02:20.427687 249961 log_block_manager.cc:2367] Opened 973 log block containers in /data/kudu/data
I0312 16:02:30.589460 249961 log_block_manager.cc:2367] Opened 1034 log block containers in /data/kudu/data
I0312 16:02:30.610613 249961 log_block_manager.cc:2518] Deleted 4 dead containers (6899 metadata bytes)
I0312 16:02:30.875131 249961 log_block_manager.cc:2663] Compacted 31 metadata files (15999442 metadata bytes)
I0312 16:02:31.130447 249898 fs_manager.cc:417] Time spent opening block manager: real 61.259s user 0.002s sys 0.001s
I0312 16:02:31.130508 249898 fs_manager.cc:428] Opened local filesystem: /data/kudu
uuid: "e1491b634e904b5faead1eeb3e09b6bf"
format_stamp: "Formatted at 2017-12-20 17:21:05 on xxx.domain.com"
I0312 16:02:31.130543 249898 fs_report.cc:347] Block manager report
--------------------
1 data directories: /data/kudu/data
Total live blocks: 1695972
Total live bytes: 59203893146
Total live bytes (after alignment): 64107425792
Total number of LBM containers: 1035 (210 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 18 (18 repaired)
Total full LBM container extra space in bytes: 558772224 (558772224 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
I0312 16:02:31.279477 249898 env_posix.cc:1634] Not raising this process' running threads per effective uid limit of 65536; it is already as high as it can go
I0312 16:02:31.286424 249898 ts_tablet_manager.cc:287] Loading tablet metadata (0/13200 complete)
I0312 16:02:31.286433 249898 ts_tablet_manager.cc:915] T fb8ae16512364c5989659150b491d552 P e1491b634e904b5faead1eeb3e09b6bf: Loading tablet metadata
...

I0312 16:04:02.484444 249898 ts_tablet_manager.cc:915] T 797f60a25e1a4bfab21c014240dab41e P e1491b634e904b5faead1eeb3e09b6bf: Loading tablet metadata
I0312 16:04:02.489385 249898 ts_tablet_manager.cc:915] T 59a8e8a4c30945cf8bd89a855ee63415 P e1491b634e904b5faead1eeb3e09b6bf: Loading tablet metadata
I0312 16:04:02.499734 249898 ts_tablet_manager.cc:299] Loaded tablet metadata (13200 live tablets)
...
I0312 16:04:06.421217 249898 ts_tablet_manager.cc:1080] T 59a8e8a4c30945cf8bd89a855ee63415 P e1491b634e904b5faead1eeb3e09b6bf: Registered tablet (data state: TABLET_DATA_READY)
....

and then again,

I0312 16:19:33.427217 251822 maintenance_manager.cc:417] P e1491b634e904b5faead1eeb3e09b6bf: System under memory pressure (60.00% of limit used). However, there are no ops currently runnable which would free memory.
.....
I0312 16:27:33.802908 251822 maintenance_manager.cc:302] P e1491b634e904b5faead1eeb3e09b6bf: Scheduling FlushDeltaMemStoresOp(b9f1ff7b72334aeba2816bc114f119d1): under memory pressure (79.73% used, can flush 127255 bytes)