HDP upgrade to 2.6.4 fails on installing packages

[#1]

Hi, dear experts!

I'm trying to upgrade my HDP from 2.6.0.3 to 2.6.4 using Ambari. Ambari upgrade was completed successfuly, but while registering and installing of target version process fails with the following error:

2018-02-14 22:57:51,664 - Package Manager failed to install packages. Error: Execution of '/usr/bin/yum -d 0 -e 0 -y install ''' returned 1. Error: Nothing to do
Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/custom_actions/scripts/install_packages.py", line 409, in install_packages
    retry_count=agent_stack_retry_count
  File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 166, in __init__
    self.env.run()
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run
    self.run_action(resource, action)
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action
    provider_action()
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/__init__.py", line 57, in action_upgrade
    self.upgrade_package(package_name, self.resource.use_repos, self.resource.skip_repos)
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/yumrpm.py", line 269, in upgrade_package
    return self.install_package(name, use_repos, skip_repos, is_upgrade)
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/yumrpm.py", line 264, in install_package
    self.checked_call_with_retries(cmd, sudo=True, logoutput=self.get_logoutput())
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/__init__.py", line 266, in checked_call_with_retries
    return self._call_with_retries(cmd, is_checked=True, **kwargs)
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/__init__.py", line 283, in _call_with_retries
    code, out = func(cmd, **kwargs)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 72, in inner
    result = function(command, **kwargs)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 102, in checked_call
    tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 150, in _call_wrapper
    result = _call(command, **kwargs_copy)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 303, in _call
    raise ExecutionFailed(err_msg, code, out, err)
ExecutionFailed: Execution of '/usr/bin/yum -d 0 -e 0 -y install ''' returned 1. Error: Nothing to do
Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/custom_actions/scripts/install_packages.py", line 473, in <module>
    InstallPackages().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 375, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/custom_actions/scripts/install_packages.py", line 153, in actionexecute
    raise Fail("Failed to distribute repositories/install packages")
resource_management.core.exceptions.Fail: Failed to distribute repositories/install packages

stdout:   /var/lib/ambari-agent/data/output-8438.txt
2018-02-14 22:56:17,861 - Installing package tez_hive2_2_6_4_0_91 ('/usr/bin/yum -d 0 -e 0 -y install tez_hive2_2_6_4_0_91')
2018-02-14 22:56:33,825 - No package found for slider_${stack_version}(slider_(\d|_)+$)
2018-02-14 22:56:33,826 - Package[None] {'retry_on_repo_unavailability': False, 'retry_count': 5, 'action': ['upgrade']}
2018-02-14 22:56:33,826 - Installing package None ('/usr/bin/yum -d 0 -e 0 -y install ''')
2018-02-14 22:56:49,179 - Execution of '/usr/bin/yum -d 0 -e 0 -y install ''' returned 1. Error: Nothing to do
2018-02-14 22:56:49,179 - Failed to install package None. Executing '/usr/bin/yum clean metadata'
2018-02-14 22:56:49,422 - Retrying to install package None after 30 seconds
2018-02-14 22:57:51,664 - Package Manager failed to install packages. Error: Execution of '/usr/bin/yum -d 0 -e 0 -y install ''' returned 1. Error: Nothing to do
Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/custom_actions/scripts/install_packages.py", line 409, in install_packages
    retry_count=agent_stack_retry_count
  File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 166, in __init__
    self.env.run()
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run
    self.run_action(resource, action)
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action
    provider_action()
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/__init__.py", line 57, in action_upgrade
    self.upgrade_package(package_name, self.resource.use_repos, self.resource.skip_repos)
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/yumrpm.py", line 269, in upgrade_package
    return self.install_package(name, use_repos, skip_repos, is_upgrade)
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/yumrpm.py", line 264, in install_package
    self.checked_call_with_retries(cmd, sudo=True, logoutput=self.get_logoutput())
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/__init__.py", line 266, in checked_call_with_retries
    return self._call_with_retries(cmd, is_checked=True, **kwargs)
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/__init__.py", line 283, in _call_with_retries
    code, out = func(cmd, **kwargs)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 72, in inner
    result = function(command, **kwargs)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 102, in checked_call
    tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 150, in _call_wrapper
    result = _call(command, **kwargs_copy)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 303, in _call
    raise ExecutionFailed(err_msg, code, out, err)
ExecutionFailed: Execution of '/usr/bin/yum -d 0 -e 0 -y install ''' returned 1. Error: Nothing to do
2018-02-14 22:57:53,898 - Installation of packages failed. Checking if installation was partially complete
2018-02-14 22:57:53,899 - Old versions: ['2.6.0.3-8', '2.6.4.0-91']
2018-02-14 22:57:53,899 - call[('ambari-python-wrap', u'/usr/bin/hdp-select', 'versions')] {}
2018-02-14 22:57:53,925 - call returned (0, '2.6.0.3-8\n2.6.4.0-91')
2018-02-14 22:57:53,925 - New versions: ['2.6.0.3-8', '2.6.4.0-91']
2018-02-14 22:57:53,925 - Deltas: set([])

Command failed after 1 tries

[#2 · Accepted solution]

@Gulshan Agivetova

As we see that the "Yum installation " package name is missing in the follownig output which is casusing the issue:

2018-02-14 22:56:33,826 - Installing package None ('/usr/bin/yum -d 0 -e 0 -y install ''')
2018-02-14 22:56:49,179 - Execution of '/usr/bin/yum -d 0 -e 0 -y install ''' 
returned 1. Error: Nothing to do

So can you please try using Ambari 2.6.1 (you are using Ambari 2.6.0.3) and see if it works. Please see the reference JIRA: which looks similar. https://issues.apache.org/jira/browse/AMBARI-22563

.

[#3]

@Jay Kumar SenSharma

Hi, Jay Kumar!

But I'm using Apache Ambari Version 2.6.1.0. Before starting upgrade of HDP, I 've upgraded Ambari to 2.6.1.0.

From the link you gave I couldn't find the workaround.

[#4]

@Gulshan Agivetova

Sometimes it can happen when we some env issues like download speeds or some slowness while accessing the repositories directly (OR) if due to slowness the "agent.package.install.task.timeout" (default 1800 seconds) is not sufficient hence it may lead to some repo inconsistency state.

So please check your "/var/log/yum.log" on the node where the installation failed.
Or please tgry installing the failed package manually on the host and then continue from ambari.

Example:

# yum clean all
# yum install slider-0.92.0.2.6.4.0-91

.

[#5]

Hi, @Jay Kumar SenSharma!

Thank you, for the link you have provided.

My problem was also because of "quasi-installed state where its repo is listed as 'installed' even though it is not"

I could resolve the problem by removing the package that was installed(partially or somewhat oddly) and relaunching the installation of packages from ambari again

Thanks!!

[#6]

@Jay Kumar SenSharma

I am also facing the same issue..however in my case i am seeing that all packagaes are installed and yum.log is clean means no errors..

ambari=> select * from host_version; id | repo_version_id | host_id | state ----+-----------------+---------+---------------- 8 | 2 | 1 | CURRENT 9 | 2 | 5 | CURRENT 13 | 2 | 3 | CURRENT 12 | 2 | 2 | CURRENT 14 | 2 | 4 | CURRENT 11 | 2 | 7 | CURRENT 10 | 2 | 6 | CURRENT 62 | 52 | 2 | INSTALL_FAILED 63 | 52 | 3 | INSTALL_FAILED 58 | 52 | 1 | INSTALL_FAILED 64 | 52 | 4 | INSTALL_FAILED 59 | 52 | 5 | INSTALL_FAILED 61 | 52 | 7 | INSTALL_FAILED 60 | 52 | 6 | INSTALL_FAILED (14 rows)

The new target version is showing failed..which pakacges are installed on all nodes and i cannot get to upgrade prompt.