Could you advise if there is a solution to the problem, when Impala assigns heavy query parts to busy executors. For example the following was faced at CDH 5.16 with Impala 2.12.0:   Impala has several (let's say 5) executors each having ~100GB RAM. Impala admission control is used. The mem_limit is set default (or about default ~80%), e.g. 80GB. The first relatively long and heavy query (let's name it Query1) comes and one of its steps take ~70GB RAM at executor1, i.e. there is ~10GB available RAM at this executor for reservation. Other 4 executor servers are nearly idle. At the same time the second query (let's name Query2) comes, which requires 40GB RAM, and it might happen the Query2 is assigned to the executor1, which is busy. So the Query2 fails due to it cannot allocate/reserve the memory.   Is there a way to configure Impala to assign fragments/query parts to less busy executors? So far the concurrency reduction or reservation removal (since reserved memory amount usually is larger than really used) might work, but I see it too inefficient to use only 1-2 executors out of 5. Impala on YARN potentially might help, but as far as I see, it requires Llama, which is deprecated and is going to be removed soon. ... View more

Yes, that is for limiting the query in order not to reduce accidental influence on other users (i.e. by occupying all available resources). One more point: impala may have default query memory limits set, so you may wish to overwrite it. ... View more

Yes, 'query1' always returned data, but the 'query2' always returned empty output. Did you mean by "now" - after some time, during which some inserts and timestamp move was done? Unfortunately, we've recreated the table after some time and I can't say if the writing helped.     Thanks for the link to ticket KUDU-2463. 1. There were inserts previous evening - the table was populated. Since then unlikely there were changes. 2. I had a look into logs: all tservers were restarted a few hours before the select queries. So far there are more evidences for this Kudu-2463. If it appears again, I will try to write to the table.   I've checked with developers, so summing up with your comments: previous day evening there was deletion (all records, but table remained) and insertion of data (via spark). Next morning tservers were restarted. After ~8h or more after insert has finished the inconsistency was found.   ... View more

These queries were executed multiple times in order something like "query1", "query2", "query1", "query2",... . For about half an hour. Kudu cluster ksck was checked as well and it was healthy. ... View more

mpercy, reviewing the cluster size is in progress, but it takes long time. As it usually happen with on-premises hardware 🙂 However, the tablet number most likely will be a bottleneck in future too, so any related performance improvement is worth. I see there are improvements with optimization for deletion in 1.7.1, so yes, might be worth to consider too. Raft consensus timeouts - probably may help to avoid avelanches, but it sounds to be rather as fighting vs consequences.   Alexey1c, 1. /etc/hosts is in place again after the issue appeared. It is the first point to check in the nswitch.conf 2. THP were disabled initially according to Cloudera recommendations. 3. Tablets were rebalanced. Thanks for the link - time to replace custom script.   Basically, reduction of tablets amount (but still above recommendations), rebalancing and population of /etc/hosts - these were the first action points and they helped to reduce the occurance significantly. But "slow dns lookup" and "couldn't get scanned data" still appear from time to time. ... View more

Yes, I understand that.   Unfortunately, the usecase dictates conditions we're hitting the limits: 1) small amount of large servers -> not so large number of tablets available 2) several dozens of systems * hundred of tables each * 3-50 tablets per table * replication factor -> quite large number of tablets required   May parameters tuning improve the situation with appearing backpressure, e.g. the default maintenance_manager_num_threads = 1 Does it make sense to change to 2-4? Maybe any other advises?     ... View more

Great news, thanks! Since it is Kudu client, does it mean it will be available when a new version of Impala is released with fixed Kudu client? ... View more

We've reduced the amount of tablets to 4-4.5k per tserver and added populated /etc/hosts - the frequency has significantly dropped (previously it happened sometimes for 3 map attempts in a row and failed the job, now it happens rarely and handled by the second attempt).   Application writes asynchronously, but shouldn't wait for such long. But I guess at OS level it still may be interrupted. I haven't seen the scanner ID in that tservers' logs. However, previously there were cases some time ago, when scanner ID wasn't found error appeared right after scanner creation and nearly after 60 seconds at one of tservers appeared this scanners timeout message.   Regarding cluster check: During the backpressure issue the tservers may become inavailable in kudu (including "cluster ksck") and consensus is lost. After some time kudu returns to normal operational state.   Are there any recommendations to reduce backpressure? Is it worth to increase stack from 50 items to larger? Or maybe any recommendations to tune kudu for larger tablet amount? ... View more

Unfortunately, the logrotation policies have just removed the logs for the case in the first post.   Are there any ideas/suggestions based on difference in table types - external and not? Both were based on Kudu, but external continued to work properly.   ... View more

Thanks for fast replies and noting about version - forgot to mention it: kudu 1.6.0-cdh5.14.4 Impala Shell v2.11.0-cdh5.14.4   mpercy, 1. Here is impala-shell (I have saved a piece of failed queries with a timestamp, so I'll try to provide details basing on it): [<coord+executor_dns>>:21000] > select * from enabled_vars where pid="3020"; Query: select * from enabled_vars where pid="3020" Query submitted at: 2018-08-01 13:47:35 (Coordinator: http://<coord+executor_dns>>:25000) Query progress can be monitored at: http://<coord+executor_dns>>:25000/query_plan?query_id=2a43f7388ed44c8d:92ff202f00000000 WARNINGS: Unable to open Kudu table: Timed out: GetTableSchema timed out after deadline expired [<coord+executor_dns>>:21000] > invalidate metadata; Query: invalidate metadata Query submitted at: 2018-08-01 13:48:45 (Coordinator: http://<coord+executor_dns>>:25000) Query progress can be monitored at: http://<coord+executor_dns>>:25000/query_plan?query_id=e34f104534f46bd9:a22d22e200000000 Fetched 0 row(s) in 42.06s 2. Nothing much in catalog daemon log at that time: I0801 13:46:19.166558 58677 catalog-server.cc:241] Catalog Version: 82226 Last Catalog Version: 82226 I0801 13:48:42.336992 47009 HdfsTable.java:1197] Incrementally loading table metadata for: <some_table>     Alexey1c, 1. 3 kudu masters 2. yes, all 3 show create table enabled_vars; Query: show create table enabled_vars +------------------------------------------------------------------------------------------------------------------------------------------------+ | result | +------------------------------------------------------------------------------------------------------------------------------------------------+ | CREATE TABLE <desired_database>.enabled_vars ( | | pid STRING NOT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION, | | var_code STRING NOT NULL ENCODING AUTO_ENCODING COMPRESSION DEFAULT_COMPRESSION, | | PRIMARY KEY (pid, var_code) | | ) | | PARTITION BY HASH (pid, var_code) PARTITIONS 2 | | STORED AS KUDU | | TBLPROPERTIES ('STATS_GENERATED_VIA_STATS_TASK'='true', 'kudu.master_addresses'='<master1>,<master2>,<master3>') | +------------------------------------------------------------------------------------------------------------------------------------------------+ Fetched 1 row(s) in 0.07s 3. Yes, it is met at the failing impala-daemon, but only more than a day before issue (see timestamp): I0731 09:13:59.293942 4640 client-internal.cc:233] Reconnecting to the cluster for a new authn token I0731 09:13:59.293975 4637 client-internal.cc:233] Reconnecting to the cluster for a new authn token I0731 09:13:59.294001 4625 client-internal.cc:233] Reconnecting to the cluster for a new authn token I0731 09:13:59.294025 4632 client-internal.cc:233] Reconnecting to the cluster for a new authn token I0731 09:13:59.294055 4622 client-internal.cc:233] Reconnecting to the cluster for a new authn token I0731 09:13:59.294075 4643 client-internal.cc:233] Reconnecting to the cluster for a new authn token I0731 09:13:59.294252 4628 client-internal.cc:233] Reconnecting to the cluster for a new authn token I0731 09:13:59.294342 4648 client-internal.cc:233] Reconnecting to the cluster for a new authn token I0731 09:13:59.294668 4633 client-internal.cc:233] Reconnecting to the cluster for a new authn token I0731 09:13:59.296530 4652 client-internal.cc:233] Reconnecting to the cluster for a new authn token I0731 09:13:59.304276 4661 client-internal.cc:233] Reconnecting to the cluster for a new authn token I0731 09:13:59.394750 29050 impala-internal-service.cc:44] ExecQueryFInstances(): query_id=464265b6715c2634:cdc2ab0500000000 I0731 09:13:59.395293 29050 query-exec-mgr.cc:46] StartQueryFInstances() query_id=464265b6715c2634:cdc2ab0500000000 coord=<another_coordinator_node>:22000 I0731 09:13:59.395375 29050 query-state.cc:173] Buffer pool limit for 464265b6715c2634:cdc2ab0500000000: 68719476736 I0731 09:13:59.396725 29050 initial-reservations.cc:60] Successfully claimed initial reservations (0) for query 464265b6715c2634:cdc2ab050 0000000 Do not see any errors closely after it. 4. No, can't find at the failing impala-daemon ... View more

The issue appeared again: this time it was at nodes, which are both "coordinator only" and "coordinator+executor". Coordinator+executor didn't have LDAP authentication.   Recent changes before the issue was noticed: there was added one new impala-coordinator-only instance.   One more observation: impala external tables (on top of Kudu) continue working fine impala tables stored as Kudu were failing with the warning (in impala CLI): WARNINGS: Unable to open Kudu table: Timed out: GetTableSchema timed out after deadline expired     Kudu-master log file:   negotiation.cc:320] Unauthorized connection attempt: Server connection negotiation failed: server connection from <impala-coord-executor-IP-address>:48648: authentication token expired ...   Impala-coord+executor log file: I0801 13:46:10.203364 39650 coordinator.cc:370] started execution on 2 backends for query 21448fac89ef4281:3bffcd4800000000 I0801 13:46:10.238082 39926 query-exec-mgr.cc:149] ReleaseQueryState(): query_id=21448fac89ef4281:3bffcd4800000000 refcnt=4 I0801 13:46:10.310528 39928 client-internal.cc:281] Determining the new leader Master and retrying... I0801 13:46:10.347209 39928 client-internal.cc:283] Unable to determine the new leader Master: Not authorized: Client connection negotiation failed: client connection to <kudu-leader-IP>:7051: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: authentication token expired I0801 13:46:10.374269 39928 client-internal.cc:283] Unable to determine the new leader Master: Not authorized: Client connection negotiation failed: client connection to <kudu-leader-IP>:7051: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: authentication token expired   Restart of a single impala coordinator+executor instance (where the issue happened) didn't help. Restart of all impala daemon instances resolved the issue. ... View more

Oh, the issue appeared to be even in functions. Thanks for fast reply and raising a ticket. ... View more

mpercy, t hanks for the link. Currently the number of tablets per server at some tservers is exceeding recommended up to 10 times. Others criteria are fine.   Does it mean the default limit number of file handlers should be increased 10 times, e.g. from 32k to 320k? ... View more

"-1" is an option, of course, but it would be good to have better understanding is it normal behaviour, why does it grow and how large might it raise. As well as having some limits (in the case of sudden grow) would be useful to limit the effect on other cluster services.   Hash Partitions - majority 50, a couple tables - 128. Range partitioning unbounded. Encoding attribute is not used in table creation (auto_encoding). Block size is not used in table creation (default is taken) ... View more

The cluster was stopped during upgrade (from 5.13.0), so impala (as well as other services) is expected to be started with new configs since then.   From cluster inspect the only thing remained running from 5.13 is supervisord process.   I'll share more details if it appears again. ... View more

Answering your questions: 1) kudu -version kudu 1.6.0-cdh5.14.2 However, not sure if Kudu client is used in communication between Impala and Kudu. 2) Impala coordinators use LDAP authentication, impala executors - don't. Kerberos is not used. Moreover, the new session is initiated and the same issue appears (this happens till Impala is restarted).   The sequence was: Normally the cluster works, Impala-users connect and execute queries through coordinator. There was Kudu leader change from one master to another (possibly also masterA->masterC->masterA). The new Impala-user runs a query and receives this issue. Impala restart has removed the effect and cluster returned to usual operating mode. ... View more

This property is left default, i.e. --block_manager_max_open_files=-1  The properties are left default, except for 11:   - 4 related to dirs (fs_wal_dirs and fs_data_dirs for master and tablet servers) - Kudu Tablet Server Hard Memory Limit = 20G - Kudu Tablet Server Block Cache Capacity = 5G - Automatically Restart Process - Process Swap Memory Thresholds - Maximum Process File Descriptors = 65k - Cgroup CPU Shares=5000 - Cgroup I/O Weight=512 ... View more

3 Masters 7 Tablet Servers (plans are to double the number of nodes in the nearest future)   Kudu comes from CDH: kudu 1.6.0-cdh5.14.2 ... View more