DNS Error when trying to insert data in Kudu table

[#1]

   Hi,

   We have a Kudu installation with 3 tablet servers and when we try to insert data using the C++ Kudu API we get the following errors:

W0728 10:10:49.129612  8707 meta_cache.cc:207] Tablet ee4f630ffb8b440faa029c106df6c058: Replica 4c09e6d53a8248a3bc6e1f6408c0d0a6 (bigdata07dev:7050) has failed: Network error: Failed to resolve address for TS 4c09e6d53a8248a3bc6e1f6408c0d0a6: Unable to resolve address 'bigdata07dev': Name or service not known
W0728 10:10:50.142993  8707 meta_cache.cc:207] Tablet 04f38c93f4c14fa29a581cb2b70aa982: Replica 0cf3c1866a094ee0b2305bca770f5e70 (bigdata09dev:7050) has failed: Network error: Failed to resolve address for TS 0cf3c1866a094ee0b2305bca770f5e70: Unable to resolve address 'bigdata09dev': Name or service not known
W0728 10:10:51.165246  8707 meta_cache.cc:207] Tablet 4e8f82788d0d4442bd91f39b3a09f5ca: Replica 5a6eb71e57554893b09acbd0a052aa9e (bigdata08dev:7050) has failed: Network error: Failed to resolve address for TS 5a6eb71e57554893b09acbd0a052aa9e: Unable to resolve address 'bigdata08dev': Name or service not known
W0728 10:10:52.165480  8707 meta_cache.cc:207] Tablet a992ebf9cc3c40bda5c458cabeb2cabf: Replica 0cf3c1866a094ee0b2305bca770f5e70 (bigdata09dev:7050) has failed: Network error: Failed to resolve address for TS 0cf3c1866a094ee0b2305bca770f5e70: Unable to resolve address 'bigdata09dev': Name or service not known
W0728 10:10:53.185971  8707 meta_cache.cc:207] Tablet a56b25d77bd7406099533cbcff6b59a5: Replica 5a6eb71e57554893b09acbd0a052aa9e (bigdata08dev:7050) has failed: Network error: Failed to resolve address for TS 5a6eb71e57554893b09acbd0a052aa9e: Unable to resolve address 'bigdata08dev': Name or service not known
W0728 10:10:54.191416  8707 meta_cache.cc:207] Tablet 57129396125e4db2ba9374acfe013682: Replica 5a6eb71e57554893b09acbd0a052aa9e (bigdata08dev:7050) has failed: Network error: Failed to resolve address for TS 5a6eb71e57554893b09acbd0a052aa9e: Unable to resolve address 'bigdata08dev': Name or service not known
W0728 10:10:55.212915  8707 meta_cache.cc:207] Tablet 3cc7d0b868834cdd883768b5d5afed68: Replica 0cf3c1866a094ee0b2305bca770f5e70 (bigdata09dev:7050) has failed: Network error: Failed to resolve address for TS 0cf3c1866a094ee0b2305bca770f5e70: Unable to resolve address 'bigdata09dev': Name or service not known
W0728 10:10:56.219542  8707 meta_cache.cc:207] Tablet 91f1452cbb0942d3a3ef1bf60b1d97ab: Replica 0cf3c1866a094ee0b2305bca770f5e70 (bigdata09dev:7050) has failed: Network error: Failed to resolve address for TS 0cf3c1866a094ee0b2305bca770f5e70: Unable to resolve address 'bigdata09dev': Name or service not known
W0728 10:10:57.220916  8707 meta_cache.cc:207] Tablet 2ff810ab35724222a01e5db584c87c35: Replica 4c09e6d53a8248a3bc6e1f6408c0d0a6 (bigdata07dev:7050) has failed: Network error: Failed to resolve address for TS 4c09e6d53a8248a3bc6e1f6408c0d0a6: Unable to resolve address 'bigdata07dev': Name or service not known
W0728 10:11:02.837929  8707 net_util.cc:129] Time spent resolving address for bigdata07dev: real 5.012s user 0.000s sys 0.000s
W0728 10:11:02.838008  8707 meta_cache.cc:207] Tablet a56b25d77bd7406099533cbcff6b59a5: Replica 4c09e6d53a8248a3bc6e1f6408c0d0a6 (bigdata07dev:7050) has failed: Network error: Failed to resolve address for TS 4c09e6d53a8248a3bc6e1f6408c0d0a6: Unable to resolve address 'bigdata07dev': Name or service not known
W0728 10:11:02.857394  8703 batcher.cc:329] Timed out: Failed to write batch of 805 ops to tablet a56b25d77bd7406099533cbcff6b59a5 after 15 attempt(s): Failed to write to server: (no server available): Write(tablet: a56b25d77bd7406099533cbcff6b59a5, num_ops: 805, num_attempts: 15) passed its deadline: Network error: Failed to resolve address for TS 5a6eb71e57554893b09acbd0a052aa9e: Unable to resolve address 'bigdata08dev': Name or service not known
Check failed: _s.ok() Bad status: IO error: Some errors occurred

   We don't have DNS to resolve the names, instead we use the hosts file of the OS. Is there a way to configure Kudu to use the hosts file?

   Thank you very much and best regards

[#2]

UP!!

[#3]

Kudu doesn't do anything special to resolve hostnames, so it should use the hosts file by way of doing the resolving. You can also see a 5s delay when trying to resolve the address, I would recommend googling "dns 5 seconds delay" and debug your network situation from there.

[#4 · Accepted solution]

If it happens to anyone else, we fixed this by deactivating IPv6 (seems that when resolving an address goes to the /etc/hosts for it, and if it only finds the IPv4 address then tries to get the IPv6 through DNS. Deactivating IPv6 in the servers avoids the DNS search).