You can connect to multiple masters by providing a list of master addresses. ... View more

The memory is being used somewhere other than in places where the usage is tracked by the memtrackers. The first possibility that comes to mind is that it is being used by the block manager. Try running   kudu fs check -fs_wal_dir=<wal dir> -fs_data_dirs=<data dirs>   and posting the output here. You may need to run the command as the kudu user (usually 'kudu').   Another helpful thing would be to gather heap samples. See https://kudu.apache.org/docs/troubleshooting.html#heap_sampling. If you attach the resulting SVG file it would be very helpful in locating the memory usage. ... View more

Yes, KUDU-1400 is a reasonable explanation here. It's not exactly a bug, just a lack of a feature. In any case, the next release of Kudu has an enhanced compaction procedure that will handle this case. ... View more

One thing that is clearly happening here is that Kudu is sending much more data than is necessary back to Impala. You specified LIMIT 7, but Kudu doesn't support server-side limits until CDH6.1. For such a small query, this might make things quite a bit faster.   Beyond that, I honestly don't see anything suspicious about the numbers from the Kudu side other than the scan took a long time given the amount of work involved. The trace shows nothing out of the ordinary; the metrics are fine. There weren't even cache misses, so everything came out of cache decoded + decompressed.   From the Impala profile, the only suspicious-looking thing is that two round trips were required. That shouldn't have been the case with LIMIT 7 as the first batch should have had more than 7 records in it.   If you run the scan a couple times in a row, does it get much faster? How does the time vary with the LIMIT amount? ... View more

You might be able to find a trace of the ScanRequest RPC in the /rpcz endpoint of the web UI. Try running the scan and, if you observe it is slow, looking at /rpcz for the matching trace.   If that's too difficult, you can get all the traces dumped into the INFO log using the flag -rpc_dump_all_traces. Naturally this is very verbose on a busy server, but if you use the kudu tserver set_flag command you can turn the flag on and then turn it off a few seconds later, giving just enough time to capture the scan if you coordinate running the scan with turning on tracing. It's safe to change the flag at runtime in this way.   If you get a trace, please copy paste it in a reply here. It will help us understand where time is being spent.   You could also try gathering a trace of the tablet server when the slow scan runs. If you upload the trace file I can take a look at it.   See https://kudu.apache.org/docs/troubleshooting.html#kudu_tracing for more information on how to gather traces.     ... View more

Those are metrics for a write, not a scan. A scan RPC trace looks like { "method_name": "kudu.tserver.ScanRequestPB", "samples": [ { "header": { "call_id": 7, "remote_method": { "service_name": "kudu.tserver.TabletServerService", "method_name": "Scan" }, "timeout_millis": 29999 }, "trace": "1005 10:27:46.216542 (+ 0us) service_pool.cc:162] Inserting onto call queue\n1005 10:27:46.216573 (+ 31us) service_pool.cc:221] Handling call\n1005 10:27:46.216712 (+ 139us) tablet_service.cc:1796] Created scanner 9c3aaa87517f4832aa81ff0dc0d71284 for tablet 42483058124f48c685943bef52f3b625\n1005 10:27:46.216839 (+ 127us) tablet_service.cc:1872] Creating iterator\n1005 10:27:46.216874 (+ 35us) tablet_service.cc:2209] Waiting safe time to advance\n1005 10:27:46.216894 (+ 20us) tablet_service.cc:2217] Waiting for operations to commit\n1005 10:27:46.216917 (+ 23us) tablet_service.cc:2231] All operations in snapshot committed. Waited for 32 microseconds\n1005 10:27:46.216937 (+ 20us) tablet_service.cc:1902] Iterator created\n1005 10:27:46.217231 (+ 294us) tablet_service.cc:1916] Iterator init: OK\n1005 10:27:46.217250 (+ 19us) tablet_service.cc:1965] has_more: true\n1005 10:27:46.217258 (+ 8us) tablet_service.cc:1980] Continuing scan request\n1005 10:27:46.217291 (+ 33us) tablet_service.cc:2033] Found scanner 9c3aaa87517f4832aa81ff0dc0d71284 for tablet 42483058124f48c685943bef52f3b625\n1005 10:27:46.218143 (+ 852us) inbound_call.cc:162] Queueing success response\n", "duration_ms": 1, "metrics": [ { "key": "rowset_iterators", "value": 1 }, { "key": "threads_started", "value": 1 }, { "key": "thread_start_us", "value": 64 }, { "key": "compiler_manager_pool.run_cpu_time_us", "value": 117013 }, { "key": "compiler_manager_pool.run_wall_time_us", "value": 127378 }, { "key": "compiler_manager_pool.queue_time_us", "value": 114 } ] } ] }, The profile is pointing to this server having a lot of data blocks. What is your workload like? Does it involve a lot of updates and deletes? How many tablet replicas are on this server? Attaching the output of the following commands will help investigate further. All should be run on the tablet server where you're seeing the memory problem: sudo -u kudu kudu fs check --fs_wal_dir=<wal dir> --fs_data_dirs=<data dirs> This should be fine to run while the server is running. You'll see a benign error message about not being able to acquire a lock and proceeding in read only mode. sudo -u kudu kudu local_replica data_size <tablet id> --fs_wal_dir=<wal dir> --fs_data_dirs=<data dirs> Try running this for a few tablets of your most active tables. ... View more