Amabari multi-node cluster installation failing

[#1]

Hi all,

I've set up a local repository using httpd, that hosts Amabari 2.5.1, HDP 2.6.1 and HDP_UTILS 1.1.0.21. I have 5 nodes using CentOS 6.5/RHEL 6, and the installation fails during "Install, Start and Test" step. Below is an output from one of the hosts installing DataNode.

stderr: /var/lib/ambari-agent/data/errors-94.txt

2017-07-14 16:47:10,887 - Could not find stack selector for stack: HDP
Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/datanode.py", line 161, in <module>
    DataNode().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 329, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/datanode.py", line 55, in install
    self.install_packages(env)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 693, in install_packages
    retry_count=agent_stack_retry_count)
  File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, in __init__
    self.env.run()
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run
    self.run_action(resource, action)
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action
    provider_action()
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/__init__.py", line 54, in action_install
    self.install_package(package_name, self.resource.use_repos, self.resource.skip_repos)
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/yumrpm.py", line 51, in install_package
    self.checked_call_with_retries(cmd, sudo=True, logoutput=self.get_logoutput())
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/__init__.py", line 86, in checked_call_with_retries
    return self._call_with_retries(cmd, is_checked=True, **kwargs)
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/__init__.py", line 98, in _call_with_retries
    code, out = func(cmd, **kwargs)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 72, in inner
    result = function(command, **kwargs)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 102, in checked_call
    tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 150, in _call_wrapper
    result = _call(command, **kwargs_copy)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 303, in _call
    raise ExecutionFailed(err_msg, code, out, err)
resource_management.core.exceptions.ExecutionFailed: Execution of '/usr/bin/yum -d 0 -e 0 -y install hadoop_2_3_2_0_2950' returned 1. Error: Cannot find a valid baseurl for repo: base
Could not retrieve mirrorlist http://mirrorlist.centos.org/?release=6&arch=x86_64&repo=os error was
14: PYCURL ERROR 7 - "Failed to connect to 2001:4178:5:200::10: Network is unreachable"

stdout: /var/lib/ambari-agent/data/output-94.txt

2017-07-14 16:46:09,584 - Stack Feature Version Info: stack_version=2.6, version=None, current_cluster_version=None -> 2.6
2017-07-14 16:46:09,586 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
User Group mapping (user_group) is missing in the hostLevelParams
2017-07-14 16:46:09,587 - Group['hadoop'] {}
2017-07-14 16:46:09,588 - Group['users'] {}
2017-07-14 16:46:09,589 - User['zookeeper'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2017-07-14 16:46:09,589 - User['ams'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2017-07-14 16:46:09,590 - User['ambari-qa'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['users']}
2017-07-14 16:46:09,590 - User['hdfs'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2017-07-14 16:46:09,591 - User['yarn'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2017-07-14 16:46:09,592 - User['mapred'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2017-07-14 16:46:09,592 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2017-07-14 16:46:09,594 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'}
2017-07-14 16:46:09,600 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] due to not_if
2017-07-14 16:46:09,600 - Group['hdfs'] {}
2017-07-14 16:46:09,601 - User['hdfs'] {'fetch_nonlocal_groups': True, 'groups': ['hadoop', 'hdfs']}
2017-07-14 16:46:09,601 - FS Type: 
2017-07-14 16:46:09,601 - Directory['/etc/hadoop'] {'mode': 0755}
2017-07-14 16:46:09,602 - Directory['/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 01777}
2017-07-14 16:46:09,627 - Initializing 2 repositories
2017-07-14 16:46:09,627 - Repository['HDP-2.6'] {'base_url': 'http://10.161.0.15/HDP/centos6/', 'action': ['create'], 'components': ['HDP', 'main'], 'repo_template': '[{{repo_id}}]\nname={{repo_id}}\n{% if mirror_list %}mirrorlist={{mirror_list}}{% else %}baseurl={{base_url}}{% endif %}\n\npath=/\nenabled=1\ngpgcheck=0', 'repo_file_name': 'HDP', 'mirror_list': None}
2017-07-14 16:46:09,637 - File['/etc/yum.repos.d/HDP.repo'] {'content': '[HDP-2.6]\nname=HDP-2.6\nbaseurl=http://10.161.0.15/HDP/centos6/\n\npath=/\nenabled=1\ngpgcheck=0'}
2017-07-14 16:46:09,638 - Repository['HDP-UTILS-1.1.0.21'] {'base_url': 'http://10.161.0.15/HDP-UTILS/', 'action': ['create'], 'components': ['HDP-UTILS', 'main'], 'repo_template': '[{{repo_id}}]\nname={{repo_id}}\n{% if mirror_list %}mirrorlist={{mirror_list}}{% else %}baseurl={{base_url}}{% endif %}\n\npath=/\nenabled=1\ngpgcheck=0', 'repo_file_name': 'HDP-UTILS', 'mirror_list': None}
2017-07-14 16:46:09,643 - File['/etc/yum.repos.d/HDP-UTILS.repo'] {'content': '[HDP-UTILS-1.1.0.21]\nname=HDP-UTILS-1.1.0.21\nbaseurl=http://10.161.0.15/HDP-UTILS/\n\npath=/\nenabled=1\ngpgcheck=0'}
2017-07-14 16:46:09,644 - Package['unzip'] {'retry_on_repo_unavailability': False, 'retry_count': 5}
2017-07-14 16:46:09,732 - Skipping installation of existing package unzip
2017-07-14 16:46:09,736 - Package['curl'] {'retry_on_repo_unavailability': False, 'retry_count': 5}
2017-07-14 16:46:09,749 - Skipping installation of existing package curl
2017-07-14 16:46:09,749 - Package['hdp-select'] {'retry_on_repo_unavailability': False, 'retry_count': 5}
2017-07-14 16:46:09,762 - Skipping installation of existing package hdp-select
2017-07-14 16:46:09,975 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2017-07-14 16:46:09,983 - Stack Feature Version Info: stack_version=2.6, version=None, current_cluster_version=None -> 2.6
2017-07-14 16:46:09,986 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2017-07-14 16:46:09,992 - checked_call['rpm -q --queryformat '%{version}-%{release}' hdp-select | sed -e 's/\.el[0-9]//g''] {'stderr': -1}
2017-07-14 16:46:10,033 - checked_call returned (0, '2.3.2.0-2950', '')
2017-07-14 16:46:10,039 - Package['hadoop_2_3_2_0_2950'] {'retry_on_repo_unavailability': False, 'retry_count': 5}
2017-07-14 16:46:10,124 - Installing package hadoop_2_3_2_0_2950 ('/usr/bin/yum -d 0 -e 0 -y install hadoop_2_3_2_0_2950')
2017-07-14 16:46:25,425 - Execution of '/usr/bin/yum -d 0 -e 0 -y install hadoop_2_3_2_0_2950' returned 1. Error: Cannot find a valid baseurl for repo: base
Could not retrieve mirrorlist http://mirrorlist.centos.org/?release=6&arch=x86_64&repo=os error was
14: PYCURL ERROR 7 - "Failed to connect to 2604:1580:fe02:2::10: Network is unreachable"
2017-07-14 16:46:25,426 - Failed to install package hadoop_2_3_2_0_2950. Executing '/usr/bin/yum clean metadata'
2017-07-14 16:46:25,633 - Retrying to install package hadoop_2_3_2_0_2950 after 30 seconds
2017-07-14 16:47:10,887 - Could not find stack selector for stack: HDP

Command failed after 1 tries

Looks like the issue is caused by the following:

Could not retrieve mirrorlist http://mirrorlist.centos.org/?release=6&arch=x86_64&repo=os error was
14: PYCURL ERROR 7 - "Failed to connect to 2604:1580:fe02:2::10: Network is unreachable"
2017-07-14 16:46:25,426 - Failed to install package hadoop_2_3_2_0_2950. Executing '/usr/bin/yum clean metadata'

I suspect it to be a proxy issue. I have configured the proxy (http_proxy, and https_proxy) in .bash_profile, and disabled proxy for the local repository, using no_proxy.

Any suggestions on how to resolve this issue?

Thanks,

[#2 · Accepted solution]

@Hovo Khachikyan

Following seems to be the culprit of the issue, Looks like a N/W issue or a proxy setup issue.

2017-07-14 16:46:25,425 - Execution of '/usr/bin/yum -d 0 -e 0 -y install hadoop_2_3_2_0_2950' returned 1. Error: Cannot find a valid baseurl for repo: base Could not retrieve mirrorlist http://mirrorlist.centos.org/?release=6&arch=x86_64&repo=os error was
.
.
PYCURL ERROR 7 - "Failed to connect to 2604:1580:fe02:2::10: Network is unreachable"

Can you please check if you can "wget" the mentioned URL from the problematic host? (To verify if the proxy settings are working or not? Else the proxy settings need to be defined inside the "~/.profile" or at the ENV level)

# wget http://mirrorlist.centos.org/?release=6&arch=x86_64&repo=os

Also please check the "/etc/yum.conf" file to see if the "proxy" setting is mentioned there or not?

# grep 'proxy' /etc/yum.conf 

.

[#3]

I was able to resolve the proxy issues, however now I seem to be having versioning issue. Looking at the output log, its trying to install hadoop_2_3_2_0_2950, whereas i have HDP 2.6.1. I suspect hdp-select to be the issue. What are your thoughts?

[#4]

@Jay SenSharma thanks for the prompt reply.

I've tried to wget, and here's the content of the file is:

Invalid Arch

I am on the other hand able to wget google without any issues.

I've also checked the yum.config file and the proxy details are there

[main]
cachedir=/var/cache/yum/$basearch/$releasever
keepcache=0
debuglevel=2
logfile=/var/log/yum.log
exactarch=1
obsoletes=1
gpgcheck=1
plugins=1
installonly_limit=5
bugtracker_url=http://bugs.centos.org/set_project.php?project_id=16&ref=http://b$
distroverpkg=centos-release

proxy=http://<ip>:<port>
proxy_username=<username>
proxy_password=<password>

[#5]

@Hovo Khachikyan

Looks like some N/W configuration issue at your end which might be blocking access to few specific sites Or the mirrorlist is not connecting from your side.

I can access the same repo without any issue locally:

Example:

[root@standalone1 tmp]# wget "http://mirrorlist.centos.org/?release=6&arch=x86_64&repo=os"
--2017-07-17 13:34:17--  http://mirrorlist.centos.org/?release=6&arch=x86_64&repo=os
Resolving mirrorlist.centos.org (mirrorlist.centos.org)... 85.236.43.108, 212.69.166.138, 216.176.179.218, ...
Connecting to mirrorlist.centos.org (mirrorlist.centos.org)|85.236.43.108|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 493 [text/plain]
Saving to: ‘index.html?release=6&arch=x86_64&repo=os’

100%[==================================================================================================================================================================================================>] 493         --.-K/s   in 0s      

2017-07-17 13:34:17 (54.8 MB/s) - ‘index.html?release=6&arch=x86_64&repo=os’ saved [493/493]

.
.

[root@standalone1 tmp]# cat index.html\?release\=6\&arch\=x86_64\&repo\=os 
http://mirror.oss.ou.edu/centos/6.9/os/x86_64/
http://centos.firehosted.com/6.9/os/x86_64/
http://mirrors.ocf.berkeley.edu/centos/6.9/os/x86_64/
http://mirror.ancl.hawaii.edu/linux/centos/6.9/os/x86_64/
http://lug.mtu.edu/centos/6.9/os/x86_64/
http://centos.host-engine.com/6.9/os/x86_64/
http://mirror.millry.co/CentOS/6.9/os/x86_64/
http://mirror.teklinks.com/centos/6.9/os/x86_64/
http://centos.mirrors.tds.net/pub/linux/centos/6.9/os/x86_64/
http://mirrors.cmich.edu/centos/6.9/os/x86_64/

.