Reposync (CentOS 7) is failing

[#1]

Hi.

I am trying to create a local mirror for HDP (HDP-2.3.4.0 and HDP-UTILS-1.1.0.20). But when I am using reposync it fails for some of the packages (by the way CentOS, Ambari and other Repos are synchronizing without problems on our mirror). Is there any corruption of the files on the hortonworks host?

I have this content in my /etc/yum.repos.d/hdp.repo file:

#VERSION_NUMBER=2.3.4.0-3485
[HDP-2.3.4.0]
name=HDP Version - HDP-2.3.4.0
baseurl=http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.3.4.0
gpgcheck=0
gpgkey=http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.3.4.0/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1

[HDP-UTILS-1.1.0.20]
name=HDP Utils Version - HDP-UTILS-1.1.0.20
baseurl=http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.20/repos/centos7
gpgcheck=1
gpgkey=http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.3.4.0/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1

This is an extract of the output I get on using reposync:

# reposync -r HDP-2.3.4.0
(1/17): accumulo_2_3_4_0_3485-conf-standalone-1.7.0.2.3.4. | 3.4 kB   00:01
(2/17): accumulo_2_3_4_0_3485-source-1.7.0.2.3.4.0-3485.el | 3.2 kB   00:00
(3/17): accumulo_2_3_4_0_3485-test-1.7.0.2.3.4.0-3485.el6. | 2.5 MB   00:06
...
flume_2_3_4_0_3485-1.5.2.2.3.4 FAILED
hadoop_2_3_4_0_3485-httpfs-2.7 FAILED
hadoop_2_3_4_0_3485-mapreduce- FAILED
hive_2_3_4_0_3485-jdbc-1.2.1.2 FAILED

For example, hadoop-httpfs is always failing. I have tried deleting the file and the complete repo and synchronizing them again. The same files are failing on and on. The files have been downloaded. Is there a problem with the checksums?

Thank you in advance for your help!

[#2 · Accepted solution]

I'm not exactly sure what is causing the error you are coming across, but it may be easier to complete the install from a locally hosted repo. Broad strokes, you'll create a temporary HTTP server, download the the repos, place them on the HTTP server, configure the yum clients on all the nodes, and proceed with your install. Here's a link to our documentation that covers this process. Hopefully it will get you past this point.

http://docs.hortonworks.com/HDPDocuments/Ambari-2.2.0.0/bk_Installing_HDP_AMB/content/_using_a_local...

Use the "No Internet Access" instructions.

[#3]

@Henry Sowell Thank you for your help. I still don't what causes the problem but I found a solution which is similar to yours. I downloaded the RPM tarball (using wget) into the HTTP Server (that I already had for the other repositories which were synced with reposync). Then I created a local repository using the createrepo command and adapted the repo files on all hosts in the cluster. Afterwards the yum clients accepted the repo and downloaded the necessary packages.

http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_HDP_RelNotes/content/download-links-230.h...