Support Questions
Find answers, ask questions, and share your expertise

archive.cloudera.com Ubuntu precise repo hangs for a long time on apt-get update

archive.cloudera.com Ubuntu precise repo hangs for a long time on apt-get update

New Contributor

We (unfortunately) still have some Ubuntu 12.04 nodes, and these nodes rely on Cloudera's Ubuntu apt repos for some packages. We noticed starting maybe 3 or 4 weeks ago that the repo hosted at archive.cloudera.com has started hanging during apt-get update. It finishes eventually, after maybe 15–30 mins, but this is a significant disruption to operational maintenance. This is our sources list entry:

 

deb [arch=amd64] http://archive.cloudera.com/cm5/ubuntu/precise/amd64/cm precise-cm5.4.5 contrib

 

This is what the output looks like; it just sits here like this for a long time before eventually finishing:

 

Hit http://archive.ubuntu.com precise-backports/restricted Translation-en
Hit http://archive.ubuntu.com precise-backports/universe Translation-en
Ign http://archive.cloudera.com precise-cm5.4.5/contrib TranslationIndex
100% [Waiting for headers]

 

stracing apt-get confirmed that it was repeatedly reaching out for files from archive.cloudera.com. Has anything changed on Cloudera's side in the Ubuntu repos that may have started causing this behavior?

2 REPLIES 2

Re: archive.cloudera.com Ubuntu precise repo hangs for a long time on apt-get update

Rising Star

@Skyler ,

 

There were no changes to archive.cloudera.com that we are aware of that should have brought this type of performance problem about.   I've done a few time curls of various sections of that repo, but of course, I am internal to our network and performance looks fine.   When this problem happens, could you please run a few

 

time curl https://archive.cloudera.com/cm5/ubuntu/precise/amd64/cm/pool/contrib/e/enterprise/

 

in addition to files in the archive like:

 

time curl https://archive.cloudera.com/cm5/ubuntu/precise/amd64/cm/pool/contrib/e/enterprise/cloudera-manager-...

 

ping archive.cloudera.com 

nslookup (or dig) archive.cloudera.com

 

and let us know what the response rates are?   One off the top of my head suggestions that I have is please make sure DNS resolution of archive.cloudera.com is working and that you don't have a broken DNS server in /etc/resolv.conf which is producing a DNS timeout lag everytime apt-get update gets a file.



Robert Justice, Technical Resolution Manager


Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Learn more about the Cloudera Community:

Terms of Service

Re: archive.cloudera.com Ubuntu precise repo hangs for a long time on apt-get update

New Contributor

Hi Robert, thanks for replying! The response rates for the curls you posted are very fast (sub-second), and DNS resolution works just fine

 

root@ip-10-71-8-214:~# ping archive.cloudera.com
PING prod.cloudera.map.fastly.net (151.101.188.167) 56(84) bytes of data.
64 bytes from 151.101.188.167: icmp_req=1 ttl=52 time=2.11 ms
64 bytes from 151.101.188.167: icmp_req=2 ttl=52 time=1.70 ms
64 bytes from 151.101.188.167: icmp_req=3 ttl=52 time=1.71 ms
64 bytes from 151.101.188.167: icmp_req=4 ttl=52 time=1.68 ms
64 bytes from 151.101.188.167: icmp_req=5 ttl=52 time=1.70 ms
^C
--- prod.cloudera.map.fastly.net ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4005ms
rtt min/avg/max/mdev = 1.682/1.784/2.117/0.166 ms
root@ip-10-71-8-214:~#
root@ip-10-71-8-214:~#
root@ip-10-71-8-214:~# dig archive.cloudera.com

; <<>> DiG 9.8.1-P1 <<>> archive.cloudera.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 64476
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;archive.cloudera.com. IN A

;; ANSWER SECTION:
archive.cloudera.com. 5 IN CNAME prod.cloudera.map.fastly.net.
prod.cloudera.map.fastly.net. 19 IN A 151.101.188.167

;; Query time: 0 msec
;; SERVER: 10.71.0.2#53(10.71.0.2)
;; WHEN: Mon Jan 6 18:47:41 2020
;; MSG SIZE rcvd: 96

 

Just judging by the output of strace -fs 256 apt-get update, I actually suspect that the issue is with the version of apt-get that precise uses. I should have mentioned in the first post that this issue doesn't happen on trusty or later. If we look at the output of strace, starting from when apt-get starts hanging and just saying "Waiting for headers...", we see:

 

[pid 22037] select(20, [5 6 7 9 11 13 15 17 19], [], NULL, {0, 500000}) = 0 (Timeout)
[pid 22037] gettimeofday({1578336552, 341881}, NULL) = 0
[pid 22037] select(20, [5 6 7 9 11 13 15 17 19], [], NULL, {0, 500000}) = 0 (Timeout)
[pid 22037] gettimeofday({1578336552, 842608}, NULL) = 0
[pid 22037] select(20, [5 6 7 9 11 13 15 17 19], [], NULL, {0, 500000}) = 0 (Timeout)
[pid 22037] gettimeofday({1578336553, 343340}, NULL) = 0
[pid 22037] select(20, [5 6 7 9 11 13 15 17 19], [], NULL, {0, 500000} <unfinished ...>
[pid 22048] <... select resumed> ) = 0 (Timeout)
[pid 22048] close(3) = 0
[pid 22048] read(0, 0x7ffc5bc919a0, 64000) = -1 EAGAIN (Resource temporarily unavailable)
[pid 22048] close(4294967295) = -1 EBADF (Bad file descriptor)
[pid 22048] write(1, "102 Status\nURI: http://archive.cloudera.com/cm5/ubuntu/precise/amd64/cm/dists/precise-cm5.4.5/contrib/i18n/Index/\nMessage: Connecting to archive.cloudera.com (151.101.188.167)\n\n", 177) = 177
[pid 22048] socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 3
[pid 22048] fcntl(3, F_GETFL) = 0x2 (flags O_RDWR)
[pid 22048] fcntl(3, F_SETFL, O_RDWR|O_NONBLOCK) = 0
[pid 22048] connect(3, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("151.101.188.167")}, 16) = -1 EINPROGRESS (Operation now in progress)
[pid 22048] select(4, NULL, [3], NULL, {120, 0} <unfinished ...>
[pid 22037] <... select resumed> ) = 1 (in [9], left {0, 342725})
[pid 22037] read(9, "102 Status\nURI: http://archive.cloudera.com/cm5/ubuntu/precise/amd64/cm/dists/precise-cm5.4.5/contrib/i18n/Index/\nMessage: Connecting to archive.cloudera.com (151.101.188.167)\n\n", 64000) = 177
[pid 22037] gettimeofday({1578336553, 501069}, NULL) = 0
[pid 22037] select(20, [5 6 7 9 11 13 15 17 19], [], NULL, {0, 500000} <unfinished ...>
[pid 22048] <... select resumed> ) = 1 (out [3], left {119, 998086})
[pid 22048] getsockopt(3, SOL_SOCKET, SO_ERROR, [0], [4]) = 0
[pid 22048] stat("/var/lib/apt/lists/partial/archive.cloudera.com_cm5_ubuntu_precise_amd64_cm_dists_precise-cm5.4.5_contrib_i18n_Index", 0x7ffc5bca0d80) = -1 ENOENT (No such file or directory)
[pid 22048] open("/etc/apt/auth.conf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 22048] open("/etc/apt/auth.conf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 22048] write(1, "102 Status\nURI: http://archive.cloudera.com/cm5/ubuntu/precise/amd64/cm/dists/precise-cm5.4.5/contrib/i18n/Index/\nMessage: Waiting for headers\n\n", 144) = 144
[pid 22048] select(4, [0 3], [3], NULL, {120, 0}) = 1 (out [3], left {119, 999997})
[pid 22048] write(3, "GET /cm5/ubuntu/precise/amd64/cm/dists/precise-cm5.4.5/contrib/i18n/Index/ HTTP/1.1\r\nHost: archive.cloudera.com\r\nConnection: keep-alive\r\nCache-Control: max-age=0\r\nAccept: text/*\r\nUser-Agent: Debian APT-HTTP/1.3 (0.8.16~exp12ubuntu10.16)\r\n\r\n", 240) = 240
[pid 22048] select(4, [0 3], [], NULL, {120, 0} <unfinished ...>
[pid 22037] <... select resumed> ) = 1 (in [9], left {0, 497912})
[pid 22037] read(9, "102 Status\nURI: http://archive.cloudera.com/cm5/ubuntu/precise/amd64/cm/dists/precise-cm5.4.5/contrib/i18n/Index/\nMessage: Waiting for headers\n\n", 64000) = 144
[pid 22037] gettimeofday({1578336553, 503490}, NULL) = 0
[pid 22037] select(20, [5 6 7 9 11 13 15 17 19], [], NULL, {0, 500000} <unfinished ...>
[pid 22048] <... select resumed> ) = 1 (in [3], left {119, 988654})
[pid 22048] read(3, "HTTP/1.1 404 Not Found\r\nServer: Varnish\r\nRetry-After: 0\r\nContent-Type: text/html\r\nAccept-Ranges: bytes\r\nContent-Length: 739\r\nAccept-Ranges: bytes\r\nDate: Mon, 06 Jan 2020 18:49:13 GMT\r\nVia: 1.1 varnish\r\nConnection: keep-alive\r\nX-Served-By: cache-pao17445-PA"..., 65536) = 1070
[pid 22048] gettimeofday({1578336553, 514918}, NULL) = 0
[pid 22048] read(3, 0x95fb0e, 64466) = -1 EAGAIN (Resource temporarily unavailable)
[pid 22048] write(1, "400 URI Failure\nURI: http://archive.cloudera.com/cm5/ubuntu/precise/amd64/cm/dists/precise-cm5.4.5/contrib/i18n/Index/\nMessage: 404 Not Found \nFailReason: HttpError404\n\n", 170) = 170
[pid 22048] open("/dev/null", O_RDWR) = 4
[pid 22048] fcntl(4, F_SETFD, FD_CLOEXEC) = 0
[pid 22048] select(5, [0 3], [4], NULL, {120, 0}) = 1 (out [4], left {119, 999997})
[pid 22048] write(4, "<!DOCTYPE html>\n<html>\n<head>\n<style>\n.error {\n margin: 0 auto;\n text-align: center;\n}\n\n.error-code {\n bottom: 60%;\n color: #2d353c;\n font-size: 96px;\n line-height: 100px;\n}\n\n.error-desc {\n font-size: 12px;\n color: #647788;\n}\n\n.m-b-10 {\n margin-bo"..., 739) = 739
[pid 22048] close(4) = 0
[pid 22048] select(1, [0], NULL, NULL, NULL <unfinished ...>
[pid 22037] <... select resumed> ) = 1 (in [9], left {0, 488432})
[pid 22037] read(9, "400 URI Failure\nURI: http://archive.cloudera.com/cm5/ubuntu/precise/amd64/cm/dists/precise-cm5.4.5/contrib/i18n/Index/\nMessage: 404 Not Found \nFailReason: HttpError404\n\n", 64000) = 170
[pid 22037] stat("/var/lib/apt/lists/archive.cloudera.com_cm5_ubuntu_precise_amd64_cm_dists_precise-cm5.4.5_contrib_i18n_Translation-en%5fUS", 0x7ffe0c0db4b0) = -1 ENOENT (No such file or directory)
[pid 22037] write(1, "Ign http://archive.cloudera.com precise-cm5.4.5/contrib TranslationIndex\n", 73Ign http://archive.cloudera.com precise-cm5.4.5/contrib TranslationIndex
) = 73
[pid 22037] gettimeofday({1578336553, 515779}, NULL) = 0
[pid 22037] select(20, [5 6 7 9 11 13 15 17 19], [14], NULL, {0, 500000}) = 1 (out [14], left {0, 499996})
[pid 22037] write(14, "600 URI Acquire\nURI: http://archive.cloudera.com/cm5/ubuntu/precise/amd64/cm/dists/precise-cm5.4.5/contrib/i18n/Translation-en_US.bz2\nFilename: /var/lib/apt/lists/partial/archive.cloudera.com_cm5_ubuntu_precise_amd64_cm_dists_precise-cm5.4.5_contrib_i18n_T"..., 311 <unfinished ...>
[pid 22048] <... select resumed> ) = 1 (in [0])
[pid 22048] read(0, "600 URI Acquire\nURI: http://archive.cloudera.com/cm5/ubuntu/precise/amd64/cm/dists/precise-cm5.4.5/contrib/i18n/Translation-en_US.bz2\nFilename: /var/lib/apt/lists/partial/archive.cloudera.com_cm5_ubuntu_precise_amd64_cm_dists_precise-cm5.4.5_contrib_i18n_T"..., 64000) = 311
[pid 22048] stat("/var/lib/apt/lists/partial/archive.cloudera.com_cm5_ubuntu_precise_amd64_cm_dists_precise-cm5.4.5_contrib_i18n_Translation-en%5fUS", 0x7ffc5bca0cc0) = -1 ENOENT (No such file or directory)
[pid 22048] open("/etc/apt/auth.conf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 22048] open("/etc/apt/auth.conf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 22048] read(0, 0x7ffc5bc919a0, 64000) = -1 EAGAIN (Resource temporarily unavailable)
[pid 22048] write(1, "102 Status\nURI: http://archive.cloudera.com/cm5/ubuntu/precise/amd64/cm/dists/precise-cm5.4.5/contrib/i18n/Translation-en_US.bz2\nMessage: Waiting for headers\n\n", 159) = 159
[pid 22048] select(4, [0 3], [3], NULL, {120, 0}) = 1 (out [3], left {119, 999998})
[pid 22048] write(3, "GET /cm5/ubuntu/precise/amd64/cm/dists/precise-cm5.4.5/contrib/i18n/Translation-en_US.bz2 HTTP/1.1\r\nHost: archive.cloudera.com\r\nConnection: keep-alive\r\nCache-Control: max-age=0\r\nUser-Agent: Debian APT-HTTP/1.3 (0.8.16~exp12ubuntu10.16)\r\n\r\n", 239) = 239
[pid 22048] select(4, [0 3], [], NULL, {120, 0} <unfinished ...>
[pid 22037] <... write resumed> ) = 311
[pid 22037] gettimeofday({1578336553, 516518}, NULL) = 0
[pid 22037] select(20, [5 6 7 9 11 13 15 17 19], [], NULL, {0, 500000}) = 1 (in [9], left {0, 499997})
[pid 22037] read(9, "102 Status\nURI: http://archive.cloudera.com/cm5/ubuntu/precise/amd64/cm/dists/precise-cm5.4.5/contrib/i18n/Translation-en_US.bz2\nMessage: Waiting for headers\n\n", 64000) = 159
[pid 22037] gettimeofday({1578336553, 516663}, NULL) = 0
[pid 22037] select(20, [5 6 7 9 11 13 15 17 19], [], NULL, {0, 500000} <unfinished ...>
[pid 22048] <... select resumed> ) = 1 (in [3], left {119, 997859})
[pid 22048] read(3, "HTTP/1.1 302 /cm5/ubuntu/precise/amd64/cm/dists/precise-cm5.4.5/contrib/i18n/Translation-en_US.bz2/\r\nServer: Varnish\r\nRetry-After: 0\r\nLocation: /cm5/ubuntu/precise/amd64/cm/dists/precise-cm5.4.5/contrib/i18n/Translation-en_US.bz2/\r\nVia: 1.1 varnish\r\nConten"..., 64466) = 513
[pid 22048] read(3, 0x95fd0f, 63953) = -1 EAGAIN (Resource temporarily unavailable)
[pid 22048] open("/dev/null", O_RDWR) = 4
[pid 22048] fcntl(4, F_SETFD, FD_CLOEXEC) = 0
[pid 22048] close(4) = 0
[pid 22048] write(1, "103 Redirect\nURI: http://archive.cloudera.com/cm5/ubuntu/precise/amd64/cm/dists/precise-cm5.4.5/contrib/i18n/Translation-en_US.bz2\nNew-URI: http://archive.cloudera.com/cm5/ubuntu/precise/amd64/cm/dists/precise-cm5.4.5/contrib/i18n/Translation-en_US.bz2/\n\n", 255) = 255
[pid 22048] read(0, 0x7ffc5bc919a0, 64000) = -1 EAGAIN (Resource temporarily unavailable)
[pid 22048] write(1, "102 Status\nURI: http://archive.cloudera.com/cm5/ubuntu/precise/amd64/cm/dists/precise-cm5.4.5/contrib/i18n/Translation-en_US.bz2/\nMessage: Waiting for headers\n\n", 160) = 160
[pid 22048] select(4, [0 3], [], NULL, {120, 0} <unfinished ...>
[pid 22037] <... select resumed> ) = 1 (in [9], left {0, 497715})
[pid 22037] read(9, "103 Redirect\nURI: http://archive.cloudera.com/cm5/ubuntu/precise/amd64/cm/dists/precise-cm5.4.5/contrib/i18n/Translation-en_US.bz2\nNew-URI: http://archive.cloudera.com/cm5/ubuntu/precise/amd64/cm/dists/precise-cm5.4.5/contrib/i18n/Translation-en_US.bz2/\n\n1"..., 64000) = 415
[pid 22037] gettimeofday({1578336553, 519282}, NULL) = 0
[pid 22037] select(20, [5 6 7 9 11 13 15 17 19], [], NULL, {0, 500000}) = 0 (Timeout)
[pid 22037] gettimeofday({1578336554, 20030}, NULL) = 0
[pid 22037] select(20, [5 6 7 9 11 13 15 17 19], [], NULL, {0, 500000}) = 0 (Timeout)
[pid 22037] gettimeofday({1578336554, 520421}, NULL) = 0

 

It looks like it's trying to download a file that isn't there; one of the particular endpoints it's trying to fetch is http://archive.cloudera.com/cm5/ubuntu/precise/amd64/cm/dists/precise-cm5.4.5/contrib/i18n/Index. This file does not exist, and in fact, the i18n directory doesn't exist. I'm not sure if apt-get just always tries to find these files, or if a redirect was recently put into place, but it seems like apt-get doesn't know what to do. After these files fail to fetch, strace shows it waiting, repeatedly printing

 

[pid 22037] select(20, [5 6 7 9 11 13 15 17 19], [], NULL, {0, 500000}) = 0 (Timeout)
[pid 22037] gettimeofday({1578336573, 546170}, NULL) = 0
[pid 22037] select(20, [5 6 7 9 11 13 15 17 19], [], NULL, {0, 500000}) = 0 (Timeout)
[pid 22037] gettimeofday({1578336574, 46926}, NULL) = 0
[pid 22037] select(20, [5 6 7 9 11 13 15 17 19], [], NULL, {0, 500000}) = 0 (Timeout)
[pid 22037] gettimeofday({1578336574, 547674}, NULL) = 0

 

until it shows the HTTP failures above again. This is why I wasn't sure if maybe some files moved on Cloudera's side and redirects were put into place that this old version of apt-get doesn't know how to handle.