Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Upgrade form HDP 2.3.4.7 to HDP 2.3.4.17-5

avatar
Explorer

After preparing the system for upgrade and applying it, upgrade failed in one of my 4 nodes.

After the failed operation, seems like I'm not able to perform a retry or resume it using Ambari. No action is available for version 2.3.4.17-5.

Need some help to make it happen for all the 4 nodes. Here's Ambari output:

2016-06-06 12:59:57,738 - Caught signal 15, will handle it gracefully. Compute the actual version if possible before exiting.
2016-06-06 12:59:57,767 - Package Manager failed to install packages. Error: (4, 'Interrupted system call')
Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/custom_actions/scripts/install_packages.py", line 307, in install_packages
    skip_repos=[self.REPO_FILE_NAME_PREFIX + "*"] if OSCheck.is_redhat_family() else [])
  File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 154, in __init__
    self.env.run()
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 152, in run
    self.run_action(resource, action)
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 118, in run_action
    provider_action()
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/__init__.py", line 45, in action_install
    self.install_package(package_name, self.resource.use_repos, self.resource.skip_repos)
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/yumrpm.py", line 49, in install_package
    shell.checked_call(cmd, sudo=True, logoutput=self.get_logoutput())
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in inner
    result = function(command, **kwargs)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in checked_call
    tries=tries, try_sleep=try_sleep)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in _call_wrapper
    result = _call(command, **kwargs_copy)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 240, in _call
    ready, _, _ = select.select(read_set, [], [], 1)
error: (4, 'Interrupted system call')
Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/custom_actions/scripts/install_packages.py", line 389, in <module>
    InstallPackages().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 216, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/custom_actions/scripts/install_packages.py", line 152, in actionexecute
    raise Fail("Failed to distribute repositories/install packages")
resource_management.core.exceptions.Fail: Failed to distribute repositories/install packages

 Python script has been killed due to timeout after waiting 1800 secs
stdout: /var/lib/ambari-agent/data/output-1216.txt
2016-06-06 12:29:57,885 - Will install packages for repository version 2.3.4.17-5
2016-06-06 12:29:57,885 - Repository['HDP-2.3.4.17-5'] {'append_to_file': False, 'base_url': 'http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.3.2.0', 'action': ['create'], 'components': ['HDP', 'main'], 'repo_template': '[{{repo_id}}]\nname={{repo_id}}\n{% if mirror_list %}mirrorlist={{mirror_list}}{% else %}baseurl={{base_url}}{% endif %}\n\npath=/\nenabled=1\ngpgcheck=0', 'repo_file_name': 'HDP-2.3.4.17-5', 'mirror_list': None}
2016-06-06 12:29:57,895 - File['/etc/yum.repos.d/HDP-2.3.4.17-5.repo'] {'content': InlineTemplate(...)}
2016-06-06 12:29:57,896 - Writing File['/etc/yum.repos.d/HDP-2.3.4.17-5.repo'] because it doesn't exist
2016-06-06 12:29:57,896 - Repository['HDP-UTILS-2.3.4.17-5'] {'append_to_file': True, 'base_url': 'http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.20/repos/centos6', 'action': ['create'], 'components': ['HDP-UTILS', 'main'], 'repo_template': '[{{repo_id}}]\nname={{repo_id}}\n{% if mirror_list %}mirrorlist={{mirror_list}}{% else %}baseurl={{base_url}}{% endif %}\n\npath=/\nenabled=1\ngpgcheck=0', 'repo_file_name': 'HDP-2.3.4.17-5', 'mirror_list': None}
2016-06-06 12:29:57,901 - File['/etc/yum.repos.d/HDP-2.3.4.17-5.repo'] {'content': '[HDP-2.3.4.17-5]\nname=HDP-2.3.4.17-5\nbaseurl=http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.3.2.0\n\npath=/\nenabled=1\ngpgcheck=0\n[HDP-UTILS-2.3.4.17-5]\nname=HDP-UTILS-2.3.4.17-5\nbaseurl=http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.20/repos/centos6\n\npath=/\nenabled=1\ngpgcheck=0'}
2016-06-06 12:29:57,901 - Writing File['/etc/yum.repos.d/HDP-2.3.4.17-5.repo'] because contents don't match
2016-06-06 12:29:59,870 - Package['sqoop_2_3_*'] {'use_repos': ['HDP-2.3.4.17-5', 'HDP-UTILS-2.3.4.17-5'], 'skip_repos': ['HDP-*']}
2016-06-06 12:29:59,871 - Installing package sqoop_2_3_* ('/usr/bin/yum -d 0 -e 0 -y install '--disablerepo=HDP-*' --enablerepo=HDP-2.3.4.17-5,HDP-UTILS-2.3.4.17-5 'sqoop_2_3_*'')
2016-06-06 12:33:54,354 - Package['rpcbind'] {'use_repos': ['HDP-2.3.4.17-5', 'HDP-UTILS-2.3.4.17-5'], 'skip_repos': ['HDP-*']}
2016-06-06 12:33:54,355 - Installing package rpcbind ('/usr/bin/yum -d 0 -e 0 -y install '--disablerepo=HDP-*' --enablerepo=HDP-2.3.4.17-5,HDP-UTILS-2.3.4.17-5 rpcbind')
2016-06-06 12:33:56,314 - Package['hadoop_2_3_*'] {'use_repos': ['HDP-2.3.4.17-5', 'HDP-UTILS-2.3.4.17-5'], 'skip_repos': ['HDP-*']}
2016-06-06 12:33:56,315 - Installing package hadoop_2_3_* ('/usr/bin/yum -d 0 -e 0 -y install '--disablerepo=HDP-*' --enablerepo=HDP-2.3.4.17-5,HDP-UTILS-2.3.4.17-5 'hadoop_2_3_*'')
2016-06-06 12:36:00,585 - Package['snappy'] {'use_repos': ['HDP-2.3.4.17-5', 'HDP-UTILS-2.3.4.17-5'], 'skip_repos': ['HDP-*']}
2016-06-06 12:36:00,586 - Installing package snappy ('/usr/bin/yum -d 0 -e 0 -y install '--disablerepo=HDP-*' --enablerepo=HDP-2.3.4.17-5,HDP-UTILS-2.3.4.17-5 snappy')
2016-06-06 12:36:02,541 - Package['snappy-devel'] {'use_repos': ['HDP-2.3.4.17-5', 'HDP-UTILS-2.3.4.17-5'], 'skip_repos': ['HDP-*']}
2016-06-06 12:36:02,541 - Installing package snappy-devel ('/usr/bin/yum -d 0 -e 0 -y install '--disablerepo=HDP-*' --enablerepo=HDP-2.3.4.17-5,HDP-UTILS-2.3.4.17-5 snappy-devel')
2016-06-06 12:36:04,630 - Package['hadooplzo_2_3_*'] {'use_repos': ['HDP-2.3.4.17-5', 'HDP-UTILS-2.3.4.17-5'], 'skip_repos': ['HDP-*']}
2016-06-06 12:36:04,631 - Installing package hadooplzo_2_3_* ('/usr/bin/yum -d 0 -e 0 -y install '--disablerepo=HDP-*' --enablerepo=HDP-2.3.4.17-5,HDP-UTILS-2.3.4.17-5 'hadooplzo_2_3_*'')
2016-06-06 12:36:11,613 - Package['hadoop_2_3_*-libhdfs'] {'use_repos': ['HDP-2.3.4.17-5', 'HDP-UTILS-2.3.4.17-5'], 'skip_repos': ['HDP-*']}
2016-06-06 12:36:11,613 - Installing package hadoop_2_3_*-libhdfs ('/usr/bin/yum -d 0 -e 0 -y install '--disablerepo=HDP-*' --enablerepo=HDP-2.3.4.17-5,HDP-UTILS-2.3.4.17-5 'hadoop_2_3_*-libhdfs'')
2016-06-06 12:36:13,832 - Package['hadoop_2_3_*-mapreduce'] {'use_repos': ['HDP-2.3.4.17-5', 'HDP-UTILS-2.3.4.17-5'], 'skip_repos': ['HDP-*']}
2016-06-06 12:36:13,832 - Installing package hadoop_2_3_*-mapreduce ('/usr/bin/yum -d 0 -e 0 -y install '--disablerepo=HDP-*' --enablerepo=HDP-2.3.4.17-5,HDP-UTILS-2.3.4.17-5 'hadoop_2_3_*-mapreduce'')
2016-06-06 12:36:16,025 - Package['zip'] {'use_repos': ['HDP-2.3.4.17-5', 'HDP-UTILS-2.3.4.17-5'], 'skip_repos': ['HDP-*']}
2016-06-06 12:36:16,026 - Installing package zip ('/usr/bin/yum -d 0 -e 0 -y install '--disablerepo=HDP-*' --enablerepo=HDP-2.3.4.17-5,HDP-UTILS-2.3.4.17-5 zip')
2016-06-06 12:36:17,997 - Package['extjs'] {'use_repos': ['HDP-2.3.4.17-5', 'HDP-UTILS-2.3.4.17-5'], 'skip_repos': ['HDP-*']}
2016-06-06 12:36:17,998 - Installing package extjs ('/usr/bin/yum -d 0 -e 0 -y install '--disablerepo=HDP-*' --enablerepo=HDP-2.3.4.17-5,HDP-UTILS-2.3.4.17-5 extjs')
2016-06-06 12:36:19,919 - Package['oozie_2_3_*'] {'use_repos': ['HDP-2.3.4.17-5', 'HDP-UTILS-2.3.4.17-5'], 'skip_repos': ['HDP-*']}
2016-06-06 12:36:19,919 - Installing package oozie_2_3_* ('/usr/bin/yum -d 0 -e 0 -y install '--disablerepo=HDP-*' --enablerepo=HDP-2.3.4.17-5,HDP-UTILS-2.3.4.17-5 'oozie_2_3_*'')
2016-06-06 12:50:56,015 - Package['falcon_2_3_*'] {'use_repos': ['HDP-2.3.4.17-5', 'HDP-UTILS-2.3.4.17-5'], 'skip_repos': ['HDP-*']}
2016-06-06 12:50:56,015 - Installing package falcon_2_3_* ('/usr/bin/yum -d 0 -e 0 -y install '--disablerepo=HDP-*' --enablerepo=HDP-2.3.4.17-5,HDP-UTILS-2.3.4.17-5 'falcon_2_3_*'')
2016-06-06 12:51:55,952 - Package['tez_2_3_*'] {'use_repos': ['HDP-2.3.4.17-5', 'HDP-UTILS-2.3.4.17-5'], 'skip_repos': ['HDP-*']}
2016-06-06 12:51:55,953 - Installing package tez_2_3_* ('/usr/bin/yum -d 0 -e 0 -y install '--disablerepo=HDP-*' --enablerepo=HDP-2.3.4.17-5,HDP-UTILS-2.3.4.17-5 'tez_2_3_*'')
2016-06-06 12:52:15,840 - Package['spark_2_3_*'] {'use_repos': ['HDP-2.3.4.17-5', 'HDP-UTILS-2.3.4.17-5'], 'skip_repos': ['HDP-*']}
2016-06-06 12:52:15,841 - Installing package spark_2_3_* ('/usr/bin/yum -d 0 -e 0 -y install '--disablerepo=HDP-*' --enablerepo=HDP-2.3.4.17-5,HDP-UTILS-2.3.4.17-5 'spark_2_3_*'')
2016-06-06 12:53:48,223 - Package['spark_2_3_*-python'] {'use_repos': ['HDP-2.3.4.17-5', 'HDP-UTILS-2.3.4.17-5'], 'skip_repos': ['HDP-*']}
2016-06-06 12:53:48,224 - Installing package spark_2_3_*-python ('/usr/bin/yum -d 0 -e 0 -y install '--disablerepo=HDP-*' --enablerepo=HDP-2.3.4.17-5,HDP-UTILS-2.3.4.17-5 'spark_2_3_*-python'')
2016-06-06 12:53:50,416 - Package['hbase_2_3_*'] {'use_repos': ['HDP-2.3.4.17-5', 'HDP-UTILS-2.3.4.17-5'], 'skip_repos': ['HDP-*']}
2016-06-06 12:53:50,416 - Installing package hbase_2_3_* ('/usr/bin/yum -d 0 -e 0 -y install '--disablerepo=HDP-*' --enablerepo=HDP-2.3.4.17-5,HDP-UTILS-2.3.4.17-5 'hbase_2_3_*'')
2016-06-06 12:56:24,271 - Package['phoenix_2_3_*'] {'use_repos': ['HDP-2.3.4.17-5', 'HDP-UTILS-2.3.4.17-5'], 'skip_repos': ['HDP-*']}
2016-06-06 12:56:24,272 - Installing package phoenix_2_3_* ('/usr/bin/yum -d 0 -e 0 -y install '--disablerepo=HDP-*' --enablerepo=HDP-2.3.4.17-5,HDP-UTILS-2.3.4.17-5 'phoenix_2_3_*'')
2016-06-06 12:57:23,968 - Package['zookeeper_2_3_*'] {'use_repos': ['HDP-2.3.4.17-5', 'HDP-UTILS-2.3.4.17-5'], 'skip_repos': ['HDP-*']}
2016-06-06 12:57:23,968 - Installing package zookeeper_2_3_* ('/usr/bin/yum -d 0 -e 0 -y install '--disablerepo=HDP-*' --enablerepo=HDP-2.3.4.17-5,HDP-UTILS-2.3.4.17-5 'zookeeper_2_3_*'')
2016-06-06 12:57:27,686 - Package['hadoop_2_3_*-yarn'] {'use_repos': ['HDP-2.3.4.17-5', 'HDP-UTILS-2.3.4.17-5'], 'skip_repos': ['HDP-*']}
2016-06-06 12:57:27,687 - Installing package hadoop_2_3_*-yarn ('/usr/bin/yum -d 0 -e 0 -y install '--disablerepo=HDP-*' --enablerepo=HDP-2.3.4.17-5,HDP-UTILS-2.3.4.17-5 'hadoop_2_3_*-yarn'')
2016-06-06 12:57:29,890 - Package['hadoop_2_3_*-mapreduce'] {'use_repos': ['HDP-2.3.4.17-5', 'HDP-UTILS-2.3.4.17-5'], 'skip_repos': ['HDP-*']}
2016-06-06 12:57:29,891 - Installing package hadoop_2_3_*-mapreduce ('/usr/bin/yum -d 0 -e 0 -y install '--disablerepo=HDP-*' --enablerepo=HDP-2.3.4.17-5,HDP-UTILS-2.3.4.17-5 'hadoop_2_3_*-mapreduce'')
2016-06-06 12:57:32,094 - Package['hadoop_2_3_*-hdfs'] {'use_repos': ['HDP-2.3.4.17-5', 'HDP-UTILS-2.3.4.17-5'], 'skip_repos': ['HDP-*']}
2016-06-06 12:57:32,095 - Installing package hadoop_2_3_*-hdfs ('/usr/bin/yum -d 0 -e 0 -y install '--disablerepo=HDP-*' --enablerepo=HDP-2.3.4.17-5,HDP-UTILS-2.3.4.17-5 'hadoop_2_3_*-hdfs'')
2016-06-06 12:57:34,288 - Package['pig_2_3_*'] {'use_repos': ['HDP-2.3.4.17-5', 'HDP-UTILS-2.3.4.17-5'], 'skip_repos': ['HDP-*']}
2016-06-06 12:57:34,288 - Installing package pig_2_3_* ('/usr/bin/yum -d 0 -e 0 -y install '--disablerepo=HDP-*' --enablerepo=HDP-2.3.4.17-5,HDP-UTILS-2.3.4.17-5 'pig_2_3_*'')
2016-06-06 12:58:52,298 - Package['datafu_2_3_*'] {'use_repos': ['HDP-2.3.4.17-5', 'HDP-UTILS-2.3.4.17-5'], 'skip_repos': ['HDP-*']}
2016-06-06 12:58:52,299 - Installing package datafu_2_3_* ('/usr/bin/yum -d 0 -e 0 -y install '--disablerepo=HDP-*' --enablerepo=HDP-2.3.4.17-5,HDP-UTILS-2.3.4.17-5 'datafu_2_3_*'')
2016-06-06 12:58:56,266 - Package['hive_2_3_*'] {'use_repos': ['HDP-2.3.4.17-5', 'HDP-UTILS-2.3.4.17-5'], 'skip_repos': ['HDP-*']}
2016-06-06 12:58:56,267 - Installing package hive_2_3_* ('/usr/bin/yum -d 0 -e 0 -y install '--disablerepo=HDP-*' --enablerepo=HDP-2.3.4.17-5,HDP-UTILS-2.3.4.17-5 'hive_2_3_*'')
2016-06-06 12:59:57,687 - Package['hive_2_3_*-hcatalog'] {'use_repos': ['HDP-2.3.4.17-5', 'HDP-UTILS-2.3.4.17-5'], 'skip_repos': ['HDP-*']}
2016-06-06 12:59:57,687 - Installing package hive_2_3_*-hcatalog ('/usr/bin/yum -d 0 -e 0 -y install '--disablerepo=HDP-*' --enablerepo=HDP-2.3.4.17-5,HDP-UTILS-2.3.4.17-5 'hive_2_3_*-hcatalog'')
2016-06-06 12:59:57,738 - Caught signal 15, will handle it gracefully. Compute the actual version if possible before exiting.
2016-06-06 12:59:57,738 - Installation of packages failed. Checking if installation was partially complete
2016-06-06 12:59:57,738 - Old versions: ['2.3.4.7-4']
2016-06-06 12:59:57,766 - New versions: ['2.3.2.0-2950', '2.3.4.7-4']
2016-06-06 12:59:57,767 - Deltas: set(['2.3.2.0-2950'])
2016-06-06 12:59:57,767 - Appended value "2.3.4.17,2.3.2.0-2950" to file /var/lib/ambari-agent/data/repo_version_history.csv to track this as a new version.
2016-06-06 12:59:57,767 - Version 2.3.2.0-2950 was partially installed. 
2016-06-06 12:59:57,767 - Package Manager failed to install packages. Error: (4, 'Interrupted system call')
Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/custom_actions/scripts/install_packages.py", line 307, in install_packages
    skip_repos=[self.REPO_FILE_NAME_PREFIX + "*"] if OSCheck.is_redhat_family() else [])
  File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 154, in __init__
    self.env.run()
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 152, in run
    self.run_action(resource, action)
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 118, in run_action
    provider_action()
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/__init__.py", line 45, in action_install
    self.install_package(package_name, self.resource.use_repos, self.resource.skip_repos)
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/yumrpm.py", line 49, in install_package
    shell.checked_call(cmd, sudo=True, logoutput=self.get_logoutput())
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in inner
    result = function(command, **kwargs)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in checked_call
    tries=tries, try_sleep=try_sleep)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in _call_wrapper
    result = _call(command, **kwargs_copy)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 240, in _call
    ready, _, _ = select.select(read_set, [], [], 1)
error: (4, 'Interrupted system call')
2016-06-06 12:59:58,542 - Installation of packages failed. Checking if installation was partially complete
2016-06-06 12:59:58,543 - Old versions: ['2.3.4.7-4']
2016-06-06 12:59:58,571 - New versions: ['2.3.2.0-2950', '2.3.4.7-4']
2016-06-06 12:59:58,572 - Deltas: set(['2.3.2.0-2950'])
2016-06-06 12:59:58,572 - Appended value "2.3.4.17,2.3.2.0-2950" to file /var/lib/ambari-agent/data/repo_version_history.csv to track this as a new version.
2016-06-06 12:59:58,572 - Version 2.3.2.0-2950 was partially installed.
1 ACCEPTED SOLUTION

avatar
Super Guru

I think this is something related to agent package install timeout, You can try to increase this value in ambari using below parameter if you have low bandwith internet.

vi /etc/ambari-server/conf/ambari.properties

agent.package.install.task.timeout=5000

View solution in original post

7 REPLIES 7

avatar
Super Guru

It looks like it's attempting to do some work using HDP version 2.3.2.0-2950. Check the rpm repo configuration files on the node with the problems. Does it look the same as the other working nodes? What versions are actually installed/available in your cluster?

avatar
Explorer

Dear Michael,

I'm not aware of an installation of version 2.3.2.0-2950. Is it some kind of alias for version 2.3.4.17-5?

Current installed version is 2.3.4.7.

avatar
Super Guru

I think this is something related to agent package install timeout, You can try to increase this value in ambari using below parameter if you have low bandwith internet.

vi /etc/ambari-server/conf/ambari.properties

agent.package.install.task.timeout=5000

avatar
Super Guru

@Thomaz Moreira can you share the screenshot please where exactly it got stuck?

avatar
Explorer

Sure. Today I won't be able to access our environment anymore, but tomorrow morning, surely.

I'll try to explain here, maybe you'll get it. After registering version 2.3.4.17-5 in Ambari's version tab, it was validated.

A drop down list was available to "apply to". So I selected my cluster.

So the new version got available in a blue box and the old version was showed in a green box. In the blue box for the new version, there was this "Install" button. So I clucked on it and things started to run. 3 out of 4 of my servers were updated, this one showed this error. But Now the blue box is not shown anymore, even though seems like the version is already registered.

there's no reinstall button. There's no action available in Ambari. I want to re-run it now. Try to install it all again. How can I do it?

Tomorrow I'll send you the screenshots.

Thank you.

avatar
New Contributor

Thank you so much @Jitendra Yadav, It's worked for me.

avatar
Explorer

Well, it's simply strange cause I was really trying to update to version 2.3.4.17-5, nothing to do with version 2.3.2-2950, apparently.

Jitendra, I increased this value, but how can I try this installation again? Ambari isn't showing me any possibility to retry it. Is there a command line tool available for it?

Thank you