Ambari HDP install fails during Confirm Hosts with duplicate sources.list entry warning

[#1]

@Rohit Sureka @Predrag Minovic

I'm trying to install a 5 node hadoop cluster with HDP 2.3.4.7 on Ubuntu 14.04 using Ambari 2.2.0.0. In the "Ambari - Cluster Install Wizard" I left the default settings for the repository options during "Select Stack" unchanged (HDP-2.3: http://public-repo-1.hortonworks.com/HDP/ubuntu14/2.x/updates/2.3.4.7 and HDP-UTILS-1.1.0.20: http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.20/repos/ubuntu12).

But then I get the following warning during the "Confirm Hosts" Step:

"W: Duplicate sources.list entry http://public-repo-1.hortonworks.com/ambari/ubuntu14/2.x/updates/2.2.0.0/ Ambari/main amd64 Packages (/var/lib/apt/lists/public-repo-1.hortonworks.com_ambari_ubuntu14_2.x_updates_2.2.0.0_dists_Ambari_main_binary-amd64_Packages) W: You may want to run apt-get update to correct these problems"

• Running apt-get update makes no difference.
• I can't see the duplicates in "sources.list"

This results in a failure of the "Confirm Hosts" step for 4 out of the 5 nodes. The only node that doesn't fail during this step is the node with the Ambari Server itself.

Any help would be really appreciated.

_________________________

Here is the full Registration log:

========================== Creating target directory... ==========================

Command start time 2016-05-03 10:55:04

Connection to sgscaiu0376.in623.corpintra.net closed. SSH command execution finished host=sgscaiu0376.in623.corpintra.net, exitcode=0 Command end time 2016-05-03 10:55:04

========================== Copying common functions script... ==========================

Command start time 2016-05-03 10:55:04

scp /usr/lib/python2.6/site-packages/ambari_commons host=sgscaiu0376.in623.corpintra.net, exitcode=0 Command end time 2016-05-03 10:55:05

========================== Copying OS type check script... ==========================

Command start time 2016-05-03 10:55:05

scp /usr/lib/python2.6/site-packages/ambari_server/os_check_type.py host=sgscaiu0376.in623.corpintra.net, exitcode=0 Command end time 2016-05-03 10:55:05

========================== Running OS type check... ==========================

Command start time 2016-05-03 10:55:05 Cluster primary/cluster OS family is ubuntu14 and local/current OS family is ubuntu14

Connection to sgscaiu0376.in623.corpintra.net closed. SSH command execution finished host=sgscaiu0376.in623.corpintra.net, exitcode=0 Command end time 2016-05-03 10:55:06

========================== Checking 'sudo' package on remote host... ==========================

Command start time 2016-05-03 10:55:06 sudo install

Connection to sgscaiu0376.in623.corpintra.net closed. SSH command execution finished host=sgscaiu0376.in623.corpintra.net, exitcode=0 Command end time 2016-05-03 10:55:07

========================== Copying repo file to 'tmp' folder... ==========================

Command start time 2016-05-03 10:55:07

scp /etc/apt/sources.list.d/ambari.list host=sgscaiu0376.in623.corpintra.net, exitcode=0 Command end time 2016-05-03 10:55:07

========================== Moving file to repo dir... ==========================

Command start time 2016-05-03 10:55:07

Connection to sgscaiu0376.in623.corpintra.net closed. SSH command execution finished host=sgscaiu0376.in623.corpintra.net, exitcode=0 Command end time 2016-05-03 10:55:08

========================== Changing permissions for ambari.repo... ==========================

Command start time 2016-05-03 10:55:08

Connection to sgscaiu0376.in623.corpintra.net closed. SSH command execution finished host=sgscaiu0376.in623.corpintra.net, exitcode=0 Command end time 2016-05-03 10:55:08

========================== Update apt cache of repository... ==========================

Command start time 2016-05-03 10:55:08

0% [Working] 0% [Waiting for headers] Hit http://public-repo-1.hortonworks.com Ambari InRelease

0% [Working] 0% [InRelease gpgv 3,187 B] 33% [Working] Hit http://public-repo-1.hortonworks.com Ambari/main amd64 Packages

33% [Working] 100% [Waiting for headers] 100% [Waiting for headers] 100% [Waiting for headers] 100% [Waiting for headers] Ign http://public-repo-1.hortonworks.com Ambari/main Translation-en_IN

100% [Working] Ign http://public-repo-1.hortonworks.com Ambari/main Translation-en

100% [Working]

Reading package lists... 0%

Reading package lists... 0%

Reading package lists... 0%

Reading package lists... 0%

Reading package lists... 14%

Reading package lists... Done

W: Duplicate sources.list entry http://public-repo-1.hortonworks.com/ambari/ubuntu14/2.x/updates/2.2.0.0/ Ambari/main amd64 Packages (/var/lib/apt/lists/public-repo-1.hortonworks.com_ambari_ubuntu14_2.x_updates_2.2.0.0_dists_Ambari_main_binary-amd64_Packages) W: You may want to run apt-get update to correct these problems

Connection to sgscaiu0376.in623.corpintra.net closed. SSH command execution finished host=sgscaiu0376.in623.corpintra.net, exitcode=0 Command end time 2016-05-03 10:55:11

========================== Copying setup script file... ==========================

Command start time 2016-05-03 10:55:11

scp /usr/lib/python2.6/site-packages/ambari_server/setupAgent.py host=sgscaiu0376.in623.corpintra.net, exitcode=0 Command end time 2016-05-03 10:55:12

========================== Running setup agent script... ==========================

Command start time 2016-05-03 10:55:12 E: There are problems and -y was used without --force-yes E: There are problems and -y was used without --force-yes E: There are problems and -y was used without --force-yes ('Reading package lists... Building dependency tree... Reading state information... The following packages will be DOWNGRADED: ambari-agent 0 upgraded, 0 newly installed, 1 downgraded, 0 to remove and 49 not upgraded. Need to get 21.2 MB of archives. After this operation, 367 kB disk space will be freed. ', None)

Connection to sgscaiu0376.in623.corpintra.net closed. SSH command execution finished host=sgscaiu0376.in623.corpintra.net, exitcode=100 Command end time 2016-05-03 10:55:37

ERROR: Bootstrap of host sgscaiu0376.in623.corpintra.net fails because previous action finished with non-zero exit code (100) ERROR MESSAGE: Connection to sgscaiu0376.in623.corpintra.net closed.

STDOUT: E: There are problems and -y was used without --force-yes E: There are problems and -y was used without --force-yes E: There are problems and -y was used without --force-yes ('Reading package lists... Building dependency tree... Reading state information... The following packages will be DOWNGRADED: ambari-agent 0 upgraded, 0 newly installed, 1 downgraded, 0 to remove and 49 not upgraded. Need to get 21.2 MB of archives. After this operation, 367 kB disk space will be freed. ', None)

Connection to sgscaiu0376.in623.corpintra.net closed.

[#2]

As a workaround, you can try manual registration of ambari agents since this is a 5 node cluster. Steps are here

[#3]

The ambari agents are already installed on all nodes.

[#4]

Hi Rohit, can you inspect /etc/apt/ directory on those 4 nodes. It looks like they contain *.list files listing HDP-2.2.0 repo. If so, move them out of /etc/apt and retry.

[#5]

I moved /etc/apt/sources.list.d/ambari.list out of /etc/apt/ and retried it for the failed nodes but no difference. ambari.list got recreated in the same directory during the failed attempt.

[#6]

@Felix Karabanov

Please check your repositories. Try moving the existing HDP* repositories and get added the host again.

Hope that helps.

[#7]

Just to clarify your answer, do you mean the same as Predrag with "moving the existing HDP* repositories" or what exactly should I move from where to where? Thanks, for the clarification.

[#8 · Accepted solution]

I found the problem.

Earlier I had problems with the SSH connection and so I manually installed the Ambari agents on the nodes. After I resolved the SSH problems, I forgot to remove the Ambari agents. That was causing the problem during the "Confirm Hosts" step. So after I removed them from the failing nodes, it worked.

Thank you for your help.