Reply
New Contributor
Posts: 4
Registered: ‎06-15-2018

File size/checksum mismatches in Ubuntu xenial apt repo, CentOS 7 yum repo

[ Edited ]

Hello,

 

I am installing CDH 5 manually on Ubuntu xenial (16.04.x), following these instructions:

 

https://www.cloudera.com/documentation/enterprise/5-14-x/topics/cdh_ig_cdh5_install.html

 

Adding the CDH 5 repository:

 

 

sudo wget 'https://archive.cloudera.com/cdh5/ubuntu/xenial/amd64/cdh/cloudera.list' -O /etc/apt/sources.list.d/cloudera.list

 

 

works but cleaning the repository cache:

 

 

sudo apt-get update

 

 

does not. I get this error:

 

 

Get:1 http://archive.cloudera.com/cdh5/ubuntu/xenial/amd64/cdh xenial-cdh5 InRelease [1931 B]
Get:2 http://archive.cloudera.com/cdh5/ubuntu/xenial/amd64/cdh xenial-cdh5/contrib Sources [11.6 kB]                                      
Get:3 http://archive.cloudera.com/cdh5/ubuntu/xenial/amd64/cdh xenial-cdh5/contrib amd64 Packages [27.5 kB]
Ign:3 http://archive.cloudera.com/cdh5/ubuntu/xenial/amd64/cdh xenial-cdh5/contrib amd64 Packages
Get:3 http://archive.cloudera.com/cdh5/ubuntu/xenial/amd64/cdh xenial-cdh5/contrib amd64 Packages [110 kB]
Err:3 http://archive.cloudera.com/cdh5/ubuntu/xenial/amd64/cdh xenial-cdh5/contrib amd64 Packages
  Writing more data than expected (110106 > 110088)
Get:4 http://security.ubuntu.com/ubuntu xenial-security InRelease [107 kB]  
Hit:5 http://archive.ubuntu.com/ubuntu xenial InRelease
Hit:6 http://archive.ubuntu.com/ubuntu xenial-updates InRelease                    
Hit:7 http://archive.ubuntu.com/ubuntu xenial-backports InRelease                  
Fetched 107 kB in 1s (74.3 kB/s)                                                   
Reading package lists... Done
W: http://archive.cloudera.com/cdh5/ubuntu/xenial/amd64/cdh/dists/xenial-cdh5/InRelease: Signature by key F36A89E33CC1BD0F71079007327574EE02A818DD uses weak digest algorithm (SHA1)
E: Failed to fetch http://archive.cloudera.com/cdh5/ubuntu/xenial/amd64/cdh/dists/xenial-cdh5/contrib/binary-amd64/Packages  Writing more data than expected (110106 > 110088)
E: Some index files failed to download. They have been ignored, or old ones used instead.

 

The file http://archive.cloudera.com/cdh5/ubuntu/xenial/amd64/cdh/dists/xenial-cdh5/InRelease has a Last-Modified date of Mon, 09 Apr 2018 22:20:38 GMT and starts with:

 

 

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Origin: Cloudera
Label: Cloudera
Suite: xenial-cdh5
Codename: xenial-cdh5.14.2
Version: 16.04
Date: Fri, 06 Apr 2018 19:19:16 UTC
Architectures: amd64
Components: contrib
Description: Cloudera nightly packages for xenial
MD5Sum:
 900fc40716308cb931c53d54ec06c21e 110088 contrib/binary-amd64/Packages
 7d78d4715173312ae5c0229664a6ea9e 27517 contrib/binary-amd64/Packages.gz
 3223aa2e1d0ce6b3a8ee0368a26fe87e 158 contrib/binary-amd64/Release
 b4c4d5f33777a5ac04a404c63c1b8ff4 44120 contrib/source/Sources
 90d9281874a28838d14c3d7cdd181f3e 11614 contrib/source/Sources.gz
 9ac635a7d0757f06b77e327377723704 159 contrib/source/Release
SHA1:
 84611939e0e78e6a96ad06cc090562020c9126df 110088 contrib/binary-amd64/Packages
 e28fb7c396bd4d69ce9e104f7746f3ac453cf4aa 27517 contrib/binary-amd64/Packages.gz
 f34a4af27e9da867fd9d08930ecc0754ce88969a 158 contrib/binary-amd64/Release
 77f3da7364c414cfd0ecff8996d5c996acd71594 44120 contrib/source/Sources
 b90a1fb6a712a7140c704608a57e91fb32f40084 11614 contrib/source/Sources.gz
 1f04a06a4b65f0c69f9d2d4cef99a41cfff6faaf 159 contrib/source/Release
SHA256:
 a1284e3873ea8415d491282b74062b8363816b4f0feb9096c30c15b2fa9d5c75 110088 contrib/binary-amd64/Packages
 f363c24fe3f45431fee2e4cfd8ef58051e59c2e9a71ae5e2d5f36e25d04ecf99 27517 contrib/binary-amd64/Packages.gz
 5b4244fa17696fe27a0e813ec21e50a1a44cbef3b35c9cf554dab20454885a41 158 contrib/binary-amd64/Release
 43b3a25ddcaab9d22d5bb0e54ec4bd176b207a80ad2e25a66f6b97e852f61e7b 44120 contrib/source/Sources
 72943765a1308da7076580862ef447db12bfc71f4ad961d2115286da2b74ed0a 11614 contrib/source/Sources.gz
 62232b1572d80068acdff02552b09b0c30248620a61ac976a5f28347166ca4bf 159 contrib/source/Release

 

It declares that http://archive.cloudera.com/cdh5/ubuntu/xenial/amd64/cdh/dists/xenial-cdh5/contrib/binary-amd64/Pack... is 110,088 bytes. But http://archive.cloudera.com/cdh5/ubuntu/xenial/amd64/cdh/dists/xenial-cdh5/contrib/binary-amd64/Pack... has a Last-Modified date of Thu, 14 Jun 2018 18:34:19 GMT and is now 110,106 bytes:

 

$ wget -S http://archive.cloudera.com/cdh5/ubuntu/xenial/amd64/cdh/dists/xenial-cdh5/contrib/binary-amd64/Packages -O /dev/null
--2018-06-15 20:18:28--  http://archive.cloudera.com/cdh5/ubuntu/xenial/amd64/cdh/dists/xenial-cdh5/contrib/binary-amd64/Packages
Resolving archive.cloudera.com (archive.cloudera.com)... 151.101.188.167
Connecting to archive.cloudera.com (archive.cloudera.com)|151.101.188.167|:80... connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 200 OK
  x-amz-id-2: lPYDOGkJYXXosqdrnisQTPuvz08pi+FBt3nzPwOiIK0SjXs5X0PAlHUED/Pg/qot1njie9IvGI8=
  x-amz-request-id: CB7F7D5E8F7E949B
  Last-Modified: Thu, 14 Jun 2018 18:34:19 GMT
  ETag: "ea889102f2e467dabece99b7c57b82cb"
  Content-Type: binary/octet-stream
  Server: AmazonS3
  Via: 1.1 varnish
  Content-Length: 110106
  Accept-Ranges: bytes
  Date: Fri, 15 Jun 2018 20:18:28 GMT
  Via: 1.1 varnish
  Age: 2441
  Connection: keep-alive
  X-Served-By: cache-sjc3137-SJC, cache-pao17448-PAO
  X-Cache: HIT, HIT
  X-Cache-Hits: 1, 1
  X-Timer: S1529093909.998104,VS0,VE0
Length: 110106 (108K) [binary/octet-stream]
Saving to: '/dev/null'

It would appear that the repository has been updated yesterday but some of the repository's metadata/management files have not.

New Contributor
Posts: 4
Registered: ‎06-15-2018

Re: Ubuntu xenial apt repository file size mismatch

[ Edited ]

(Title edited to reflect expansion of scope with time.)

New Contributor
Posts: 4
Registered: ‎06-15-2018

Re: File size/checksum mismatches in Ubuntu xenial apt repo, CentOS 7 yum repo

The CentOS 7 yum repository seems to have a similar problem:

 

# wget -O /etc/yum.repos.d/cloudera-cdh5.repo https://archive.cloudera.com/cdh5/redhat/7/x86_64/cdh/cloudera-cdh5.repo

# yum update
Loaded plugins: fastestmirror, ovl
Loading mirror speeds from cached hostfile
 * base: mirror.scalabledns.com
 * extras: mirror.hostduplex.com
 * updates: mirrors.usc.edu
cloudera-cdh5                                                                                                                  |  951 B  00:00:00     
cloudera-cdh5/primary          FAILED                                          
https://archive.cloudera.com/cdh5/redhat/7/x86_64/cdh/5/repodata/primary.xml.gz: [Errno -1] Metadata file does not match checksum   0 B  00:00:00     
Trying other mirror.
cloudera-cdh5/primary                                                                                                          |  50 kB  00:00:00     
https://archive.cloudera.com/cdh5/redhat/7/x86_64/cdh/5/repodata/primary.xml.gz: [Errno -1] Metadata file does not match checksum
Trying other mirror.


 One of the configured repositories failed (Cloudera's Distribution for Hadoop, Version 5),
 and yum doesn't have enough cached data to continue. At this point the only
 safe thing yum can do is fail. There are a few ways to work "fix" this:

     1. Contact the upstream for the repository and get them to fix the problem.

     2. Reconfigure the baseurl/etc. for the repository, to point to a working
        upstream. This is most often useful if you are using a newer
        distribution release than is supported by the repository (and the
        packages for the previous distribution release still work).

     3. Run the command with the repository temporarily disabled
            yum --disablerepo=cloudera-cdh5 ...

     4. Disable the repository permanently, so yum won't use it by default. Yum
        will then just ignore the repository until you permanently enable it
        again or use --enablerepo for temporary usage:

            yum-config-manager --disable cloudera-cdh5
        or
            subscription-manager repos --disable=cloudera-cdh5

     5. Configure the failing repository to be skipped, if it is unavailable.
        Note that yum will try to contact the repo. when it runs most commands,
        so will have to try and fail each time (and thus. yum will be be much
        slower). If it is a very temporary problem though, this is often a nice
        compromise:

            yum-config-manager --save --setopt=cloudera-cdh5.skip_if_unavailable=true

failure: repodata/primary.xml.gz from cloudera-cdh5: [Errno 256] No more mirrors to try.
https://archive.cloudera.com/cdh5/redhat/7/x86_64/cdh/5/repodata/primary.xml.gz: [Errno -1] Metadata file does not match checksum

https://archive.cloudera.com/cdh5/redhat/7/x86_64/cdh/5/repodata/repomd.xml contains:

 

<repomd xmlns="http://linux.duke.edu/metadata/repo">
<data type="other">
<location href="repodata/other.xml.gz"/>
<checksum type="sha">0d31c66be1730808ba69a0d7ad1d78d986d20457</checksum>
<timestamp>1528814353</timestamp>
<open-checksum type="sha">de6b242c0cec39fecc201429032e9f650cdd1254</open-checksum>
</data>
<data type="filelists">
<location href="repodata/filelists.xml.gz"/>
<checksum type="sha">2d8184b6ecc345ac179e19b192affb00b306a400</checksum>
<timestamp>1528814353</timestamp>
<open-checksum type="sha">9046d5aa5152c9fa0835a0433981039d2c6ca388</open-checksum>
</data>
<data type="primary">
<location href="repodata/primary.xml.gz"/>
<checksum type="sha">141d662449a2d21cfc5c35b8a9d91e67111cd19a</checksum>
<timestamp>1528814353</timestamp>
<open-checksum type="sha">0630d37d6d8692443e4285601e352db58e5441f9</open-checksum>
</data>
</repomd>

but the current SHA1 checksum of https://archive.cloudera.com/cdh5/redhat/7/x86_64/cdh/5/repodata/primary.xml.gz is ef0ad849d8a3b0b6509964b0ab3a8b171538e60a. (All files in https://archive.cloudera.com/cdh5/redhat/7/x86_64/cdh/5/repodata/ seem to have a Last-Modified date of June 12.)

New Contributor
Posts: 4
Registered: ‎06-15-2018

Re: File size/checksum mismatches in Ubuntu xenial apt repo, CentOS 7 yum repo

It would appear that the repo metadata has been updated and/or fixed in the last couple of days. Thanks!

Announcements