DNS Error when trying to insert data in Kudu table

albers2n — Fri, 16 Sep 2022 11:59:53 GMT

Hi,

We have a Kudu installation with 3 tablet servers and when we try to insert data using the C++ Kudu API we get the following errors:

W0728 10:10:49.129612  8707 meta_cache.cc:207] Tablet ee4f630ffb8b440faa029c106df6c058: Replica 4c09e6d53a8248a3bc6e1f6408c0d0a6 (bigdata07dev:7050) has failed: Network error: Failed to resolve address for TS 4c09e6d53a8248a3bc6e1f6408c0d0a6: Unable to resolve address 'bigdata07dev': Name or service not known
W0728 10:10:50.142993  8707 meta_cache.cc:207] Tablet 04f38c93f4c14fa29a581cb2b70aa982: Replica 0cf3c1866a094ee0b2305bca770f5e70 (bigdata09dev:7050) has failed: Network error: Failed to resolve address for TS 0cf3c1866a094ee0b2305bca770f5e70: Unable to resolve address 'bigdata09dev': Name or service not known
W0728 10:10:51.165246  8707 meta_cache.cc:207] Tablet 4e8f82788d0d4442bd91f39b3a09f5ca: Replica 5a6eb71e57554893b09acbd0a052aa9e (bigdata08dev:7050) has failed: Network error: Failed to resolve address for TS 5a6eb71e57554893b09acbd0a052aa9e: Unable to resolve address 'bigdata08dev': Name or service not known
W0728 10:10:52.165480  8707 meta_cache.cc:207] Tablet a992ebf9cc3c40bda5c458cabeb2cabf: Replica 0cf3c1866a094ee0b2305bca770f5e70 (bigdata09dev:7050) has failed: Network error: Failed to resolve address for TS 0cf3c1866a094ee0b2305bca770f5e70: Unable to resolve address 'bigdata09dev': Name or service not known
W0728 10:10:53.185971  8707 meta_cache.cc:207] Tablet a56b25d77bd7406099533cbcff6b59a5: Replica 5a6eb71e57554893b09acbd0a052aa9e (bigdata08dev:7050) has failed: Network error: Failed to resolve address for TS 5a6eb71e57554893b09acbd0a052aa9e: Unable to resolve address 'bigdata08dev': Name or service not known
W0728 10:10:54.191416  8707 meta_cache.cc:207] Tablet 57129396125e4db2ba9374acfe013682: Replica 5a6eb71e57554893b09acbd0a052aa9e (bigdata08dev:7050) has failed: Network error: Failed to resolve address for TS 5a6eb71e57554893b09acbd0a052aa9e: Unable to resolve address 'bigdata08dev': Name or service not known
W0728 10:10:55.212915  8707 meta_cache.cc:207] Tablet 3cc7d0b868834cdd883768b5d5afed68: Replica 0cf3c1866a094ee0b2305bca770f5e70 (bigdata09dev:7050) has failed: Network error: Failed to resolve address for TS 0cf3c1866a094ee0b2305bca770f5e70: Unable to resolve address 'bigdata09dev': Name or service not known
W0728 10:10:56.219542  8707 meta_cache.cc:207] Tablet 91f1452cbb0942d3a3ef1bf60b1d97ab: Replica 0cf3c1866a094ee0b2305bca770f5e70 (bigdata09dev:7050) has failed: Network error: Failed to resolve address for TS 0cf3c1866a094ee0b2305bca770f5e70: Unable to resolve address 'bigdata09dev': Name or service not known
W0728 10:10:57.220916  8707 meta_cache.cc:207] Tablet 2ff810ab35724222a01e5db584c87c35: Replica 4c09e6d53a8248a3bc6e1f6408c0d0a6 (bigdata07dev:7050) has failed: Network error: Failed to resolve address for TS 4c09e6d53a8248a3bc6e1f6408c0d0a6: Unable to resolve address 'bigdata07dev': Name or service not known
W0728 10:11:02.837929  8707 net_util.cc:129] Time spent resolving address for bigdata07dev: real 5.012s user 0.000s sys 0.000s
W0728 10:11:02.838008  8707 meta_cache.cc:207] Tablet a56b25d77bd7406099533cbcff6b59a5: Replica 4c09e6d53a8248a3bc6e1f6408c0d0a6 (bigdata07dev:7050) has failed: Network error: Failed to resolve address for TS 4c09e6d53a8248a3bc6e1f6408c0d0a6: Unable to resolve address 'bigdata07dev': Name or service not known
W0728 10:11:02.857394  8703 batcher.cc:329] Timed out: Failed to write batch of 805 ops to tablet a56b25d77bd7406099533cbcff6b59a5 after 15 attempt(s): Failed to write to server: (no server available): Write(tablet: a56b25d77bd7406099533cbcff6b59a5, num_ops: 805, num_attempts: 15) passed its deadline: Network error: Failed to resolve address for TS 5a6eb71e57554893b09acbd0a052aa9e: Unable to resolve address 'bigdata08dev': Name or service not known
Check failed: _s.ok() Bad status: IO error: Some errors occurred

We don't have DNS to resolve the names, instead we use the hosts file of the OS. Is there a way to configure Kudu to use the hosts file?

Thank you very much and best regards

Re: DNS Error when trying to insert data in Kudu table

albers2n — Mon, 31 Jul 2017 11:50:00 GMT

UP!!

Re: DNS Error when trying to insert data in Kudu table

J-D — Mon, 31 Jul 2017 16:21:00 GMT

Kudu doesn't do anything special to resolve hostnames, so it should use the hosts file by way of doing the resolving. You can also see a 5s delay when trying to resolve the address, I would recommend googling "dns 5 seconds delay" and debug your network situation from there.

Re: DNS Error when trying to insert data in Kudu table

albers2n — Mon, 07 Aug 2017 11:21:32 GMT

If it happens to anyone else, we fixed this by deactivating IPv6 (seems that when resolving an address goes to the /etc/hosts for it, and if it only finds the IPv4 address then tries to get the IPv6 through DNS. Deactivating IPv6 in the servers avoids the DNS search).

question DNS Error when trying to insert data in Kudu table in Support Questions

DNS Error when trying to insert data in Kudu table

Re: DNS Error when trying to insert data in Kudu table

Re: DNS Error when trying to insert data in Kudu table

Re: DNS Error when trying to insert data in Kudu table