Created 06-06-2016 05:56 PM
After preparing the system for upgrade and applying it, upgrade failed in one of my 4 nodes.
After the failed operation, seems like I'm not able to perform a retry or resume it using Ambari. No action is available for version 2.3.4.17-5.
Need some help to make it happen for all the 4 nodes. Here's Ambari output:
2016-06-06 12:59:57,738 - Caught signal 15, will handle it gracefully. Compute the actual version if possible before exiting. 2016-06-06 12:59:57,767 - Package Manager failed to install packages. Error: (4, 'Interrupted system call') Traceback (most recent call last): File "/var/lib/ambari-agent/cache/custom_actions/scripts/install_packages.py", line 307, in install_packages skip_repos=[self.REPO_FILE_NAME_PREFIX + "*"] if OSCheck.is_redhat_family() else []) File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 154, in __init__ self.env.run() File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 152, in run self.run_action(resource, action) File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 118, in run_action provider_action() File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/__init__.py", line 45, in action_install self.install_package(package_name, self.resource.use_repos, self.resource.skip_repos) File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/yumrpm.py", line 49, in install_package shell.checked_call(cmd, sudo=True, logoutput=self.get_logoutput()) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in inner result = function(command, **kwargs) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in checked_call tries=tries, try_sleep=try_sleep) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in _call_wrapper result = _call(command, **kwargs_copy) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 240, in _call ready, _, _ = select.select(read_set, [], [], 1) error: (4, 'Interrupted system call') Traceback (most recent call last): File "/var/lib/ambari-agent/cache/custom_actions/scripts/install_packages.py", line 389, in <module> InstallPackages().execute() File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 216, in execute method(env) File "/var/lib/ambari-agent/cache/custom_actions/scripts/install_packages.py", line 152, in actionexecute raise Fail("Failed to distribute repositories/install packages") resource_management.core.exceptions.Fail: Failed to distribute repositories/install packages Python script has been killed due to timeout after waiting 1800 secsstdout: /var/lib/ambari-agent/data/output-1216.txt
2016-06-06 12:29:57,885 - Will install packages for repository version 2.3.4.17-5 2016-06-06 12:29:57,885 - Repository['HDP-2.3.4.17-5'] {'append_to_file': False, 'base_url': 'http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.3.2.0', 'action': ['create'], 'components': ['HDP', 'main'], 'repo_template': '[{{repo_id}}]\nname={{repo_id}}\n{% if mirror_list %}mirrorlist={{mirror_list}}{% else %}baseurl={{base_url}}{% endif %}\n\npath=/\nenabled=1\ngpgcheck=0', 'repo_file_name': 'HDP-2.3.4.17-5', 'mirror_list': None} 2016-06-06 12:29:57,895 - File['/etc/yum.repos.d/HDP-2.3.4.17-5.repo'] {'content': InlineTemplate(...)} 2016-06-06 12:29:57,896 - Writing File['/etc/yum.repos.d/HDP-2.3.4.17-5.repo'] because it doesn't exist 2016-06-06 12:29:57,896 - Repository['HDP-UTILS-2.3.4.17-5'] {'append_to_file': True, 'base_url': 'http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.20/repos/centos6', 'action': ['create'], 'components': ['HDP-UTILS', 'main'], 'repo_template': '[{{repo_id}}]\nname={{repo_id}}\n{% if mirror_list %}mirrorlist={{mirror_list}}{% else %}baseurl={{base_url}}{% endif %}\n\npath=/\nenabled=1\ngpgcheck=0', 'repo_file_name': 'HDP-2.3.4.17-5', 'mirror_list': None} 2016-06-06 12:29:57,901 - File['/etc/yum.repos.d/HDP-2.3.4.17-5.repo'] {'content': '[HDP-2.3.4.17-5]\nname=HDP-2.3.4.17-5\nbaseurl=http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.3.2.0\n\npath=/\nenabled=1\ngpgcheck=0\n[HDP-UTILS-2.3.4.17-5]\nname=HDP-UTILS-2.3.4.17-5\nbaseurl=http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.20/repos/centos6\n\npath=/\nenabled=1\ngpgcheck=0'} 2016-06-06 12:29:57,901 - Writing File['/etc/yum.repos.d/HDP-2.3.4.17-5.repo'] because contents don't match 2016-06-06 12:29:59,870 - Package['sqoop_2_3_*'] {'use_repos': ['HDP-2.3.4.17-5', 'HDP-UTILS-2.3.4.17-5'], 'skip_repos': ['HDP-*']} 2016-06-06 12:29:59,871 - Installing package sqoop_2_3_* ('/usr/bin/yum -d 0 -e 0 -y install '--disablerepo=HDP-*' --enablerepo=HDP-2.3.4.17-5,HDP-UTILS-2.3.4.17-5 'sqoop_2_3_*'') 2016-06-06 12:33:54,354 - Package['rpcbind'] {'use_repos': ['HDP-2.3.4.17-5', 'HDP-UTILS-2.3.4.17-5'], 'skip_repos': ['HDP-*']} 2016-06-06 12:33:54,355 - Installing package rpcbind ('/usr/bin/yum -d 0 -e 0 -y install '--disablerepo=HDP-*' --enablerepo=HDP-2.3.4.17-5,HDP-UTILS-2.3.4.17-5 rpcbind') 2016-06-06 12:33:56,314 - Package['hadoop_2_3_*'] {'use_repos': ['HDP-2.3.4.17-5', 'HDP-UTILS-2.3.4.17-5'], 'skip_repos': ['HDP-*']} 2016-06-06 12:33:56,315 - Installing package hadoop_2_3_* ('/usr/bin/yum -d 0 -e 0 -y install '--disablerepo=HDP-*' --enablerepo=HDP-2.3.4.17-5,HDP-UTILS-2.3.4.17-5 'hadoop_2_3_*'') 2016-06-06 12:36:00,585 - Package['snappy'] {'use_repos': ['HDP-2.3.4.17-5', 'HDP-UTILS-2.3.4.17-5'], 'skip_repos': ['HDP-*']} 2016-06-06 12:36:00,586 - Installing package snappy ('/usr/bin/yum -d 0 -e 0 -y install '--disablerepo=HDP-*' --enablerepo=HDP-2.3.4.17-5,HDP-UTILS-2.3.4.17-5 snappy') 2016-06-06 12:36:02,541 - Package['snappy-devel'] {'use_repos': ['HDP-2.3.4.17-5', 'HDP-UTILS-2.3.4.17-5'], 'skip_repos': ['HDP-*']} 2016-06-06 12:36:02,541 - Installing package snappy-devel ('/usr/bin/yum -d 0 -e 0 -y install '--disablerepo=HDP-*' --enablerepo=HDP-2.3.4.17-5,HDP-UTILS-2.3.4.17-5 snappy-devel') 2016-06-06 12:36:04,630 - Package['hadooplzo_2_3_*'] {'use_repos': ['HDP-2.3.4.17-5', 'HDP-UTILS-2.3.4.17-5'], 'skip_repos': ['HDP-*']} 2016-06-06 12:36:04,631 - Installing package hadooplzo_2_3_* ('/usr/bin/yum -d 0 -e 0 -y install '--disablerepo=HDP-*' --enablerepo=HDP-2.3.4.17-5,HDP-UTILS-2.3.4.17-5 'hadooplzo_2_3_*'') 2016-06-06 12:36:11,613 - Package['hadoop_2_3_*-libhdfs'] {'use_repos': ['HDP-2.3.4.17-5', 'HDP-UTILS-2.3.4.17-5'], 'skip_repos': ['HDP-*']} 2016-06-06 12:36:11,613 - Installing package hadoop_2_3_*-libhdfs ('/usr/bin/yum -d 0 -e 0 -y install '--disablerepo=HDP-*' --enablerepo=HDP-2.3.4.17-5,HDP-UTILS-2.3.4.17-5 'hadoop_2_3_*-libhdfs'') 2016-06-06 12:36:13,832 - Package['hadoop_2_3_*-mapreduce'] {'use_repos': ['HDP-2.3.4.17-5', 'HDP-UTILS-2.3.4.17-5'], 'skip_repos': ['HDP-*']} 2016-06-06 12:36:13,832 - Installing package hadoop_2_3_*-mapreduce ('/usr/bin/yum -d 0 -e 0 -y install '--disablerepo=HDP-*' --enablerepo=HDP-2.3.4.17-5,HDP-UTILS-2.3.4.17-5 'hadoop_2_3_*-mapreduce'') 2016-06-06 12:36:16,025 - Package['zip'] {'use_repos': ['HDP-2.3.4.17-5', 'HDP-UTILS-2.3.4.17-5'], 'skip_repos': ['HDP-*']} 2016-06-06 12:36:16,026 - Installing package zip ('/usr/bin/yum -d 0 -e 0 -y install '--disablerepo=HDP-*' --enablerepo=HDP-2.3.4.17-5,HDP-UTILS-2.3.4.17-5 zip') 2016-06-06 12:36:17,997 - Package['extjs'] {'use_repos': ['HDP-2.3.4.17-5', 'HDP-UTILS-2.3.4.17-5'], 'skip_repos': ['HDP-*']} 2016-06-06 12:36:17,998 - Installing package extjs ('/usr/bin/yum -d 0 -e 0 -y install '--disablerepo=HDP-*' --enablerepo=HDP-2.3.4.17-5,HDP-UTILS-2.3.4.17-5 extjs') 2016-06-06 12:36:19,919 - Package['oozie_2_3_*'] {'use_repos': ['HDP-2.3.4.17-5', 'HDP-UTILS-2.3.4.17-5'], 'skip_repos': ['HDP-*']} 2016-06-06 12:36:19,919 - Installing package oozie_2_3_* ('/usr/bin/yum -d 0 -e 0 -y install '--disablerepo=HDP-*' --enablerepo=HDP-2.3.4.17-5,HDP-UTILS-2.3.4.17-5 'oozie_2_3_*'') 2016-06-06 12:50:56,015 - Package['falcon_2_3_*'] {'use_repos': ['HDP-2.3.4.17-5', 'HDP-UTILS-2.3.4.17-5'], 'skip_repos': ['HDP-*']} 2016-06-06 12:50:56,015 - Installing package falcon_2_3_* ('/usr/bin/yum -d 0 -e 0 -y install '--disablerepo=HDP-*' --enablerepo=HDP-2.3.4.17-5,HDP-UTILS-2.3.4.17-5 'falcon_2_3_*'') 2016-06-06 12:51:55,952 - Package['tez_2_3_*'] {'use_repos': ['HDP-2.3.4.17-5', 'HDP-UTILS-2.3.4.17-5'], 'skip_repos': ['HDP-*']} 2016-06-06 12:51:55,953 - Installing package tez_2_3_* ('/usr/bin/yum -d 0 -e 0 -y install '--disablerepo=HDP-*' --enablerepo=HDP-2.3.4.17-5,HDP-UTILS-2.3.4.17-5 'tez_2_3_*'') 2016-06-06 12:52:15,840 - Package['spark_2_3_*'] {'use_repos': ['HDP-2.3.4.17-5', 'HDP-UTILS-2.3.4.17-5'], 'skip_repos': ['HDP-*']} 2016-06-06 12:52:15,841 - Installing package spark_2_3_* ('/usr/bin/yum -d 0 -e 0 -y install '--disablerepo=HDP-*' --enablerepo=HDP-2.3.4.17-5,HDP-UTILS-2.3.4.17-5 'spark_2_3_*'') 2016-06-06 12:53:48,223 - Package['spark_2_3_*-python'] {'use_repos': ['HDP-2.3.4.17-5', 'HDP-UTILS-2.3.4.17-5'], 'skip_repos': ['HDP-*']} 2016-06-06 12:53:48,224 - Installing package spark_2_3_*-python ('/usr/bin/yum -d 0 -e 0 -y install '--disablerepo=HDP-*' --enablerepo=HDP-2.3.4.17-5,HDP-UTILS-2.3.4.17-5 'spark_2_3_*-python'') 2016-06-06 12:53:50,416 - Package['hbase_2_3_*'] {'use_repos': ['HDP-2.3.4.17-5', 'HDP-UTILS-2.3.4.17-5'], 'skip_repos': ['HDP-*']} 2016-06-06 12:53:50,416 - Installing package hbase_2_3_* ('/usr/bin/yum -d 0 -e 0 -y install '--disablerepo=HDP-*' --enablerepo=HDP-2.3.4.17-5,HDP-UTILS-2.3.4.17-5 'hbase_2_3_*'') 2016-06-06 12:56:24,271 - Package['phoenix_2_3_*'] {'use_repos': ['HDP-2.3.4.17-5', 'HDP-UTILS-2.3.4.17-5'], 'skip_repos': ['HDP-*']} 2016-06-06 12:56:24,272 - Installing package phoenix_2_3_* ('/usr/bin/yum -d 0 -e 0 -y install '--disablerepo=HDP-*' --enablerepo=HDP-2.3.4.17-5,HDP-UTILS-2.3.4.17-5 'phoenix_2_3_*'') 2016-06-06 12:57:23,968 - Package['zookeeper_2_3_*'] {'use_repos': ['HDP-2.3.4.17-5', 'HDP-UTILS-2.3.4.17-5'], 'skip_repos': ['HDP-*']} 2016-06-06 12:57:23,968 - Installing package zookeeper_2_3_* ('/usr/bin/yum -d 0 -e 0 -y install '--disablerepo=HDP-*' --enablerepo=HDP-2.3.4.17-5,HDP-UTILS-2.3.4.17-5 'zookeeper_2_3_*'') 2016-06-06 12:57:27,686 - Package['hadoop_2_3_*-yarn'] {'use_repos': ['HDP-2.3.4.17-5', 'HDP-UTILS-2.3.4.17-5'], 'skip_repos': ['HDP-*']} 2016-06-06 12:57:27,687 - Installing package hadoop_2_3_*-yarn ('/usr/bin/yum -d 0 -e 0 -y install '--disablerepo=HDP-*' --enablerepo=HDP-2.3.4.17-5,HDP-UTILS-2.3.4.17-5 'hadoop_2_3_*-yarn'') 2016-06-06 12:57:29,890 - Package['hadoop_2_3_*-mapreduce'] {'use_repos': ['HDP-2.3.4.17-5', 'HDP-UTILS-2.3.4.17-5'], 'skip_repos': ['HDP-*']} 2016-06-06 12:57:29,891 - Installing package hadoop_2_3_*-mapreduce ('/usr/bin/yum -d 0 -e 0 -y install '--disablerepo=HDP-*' --enablerepo=HDP-2.3.4.17-5,HDP-UTILS-2.3.4.17-5 'hadoop_2_3_*-mapreduce'') 2016-06-06 12:57:32,094 - Package['hadoop_2_3_*-hdfs'] {'use_repos': ['HDP-2.3.4.17-5', 'HDP-UTILS-2.3.4.17-5'], 'skip_repos': ['HDP-*']} 2016-06-06 12:57:32,095 - Installing package hadoop_2_3_*-hdfs ('/usr/bin/yum -d 0 -e 0 -y install '--disablerepo=HDP-*' --enablerepo=HDP-2.3.4.17-5,HDP-UTILS-2.3.4.17-5 'hadoop_2_3_*-hdfs'') 2016-06-06 12:57:34,288 - Package['pig_2_3_*'] {'use_repos': ['HDP-2.3.4.17-5', 'HDP-UTILS-2.3.4.17-5'], 'skip_repos': ['HDP-*']} 2016-06-06 12:57:34,288 - Installing package pig_2_3_* ('/usr/bin/yum -d 0 -e 0 -y install '--disablerepo=HDP-*' --enablerepo=HDP-2.3.4.17-5,HDP-UTILS-2.3.4.17-5 'pig_2_3_*'') 2016-06-06 12:58:52,298 - Package['datafu_2_3_*'] {'use_repos': ['HDP-2.3.4.17-5', 'HDP-UTILS-2.3.4.17-5'], 'skip_repos': ['HDP-*']} 2016-06-06 12:58:52,299 - Installing package datafu_2_3_* ('/usr/bin/yum -d 0 -e 0 -y install '--disablerepo=HDP-*' --enablerepo=HDP-2.3.4.17-5,HDP-UTILS-2.3.4.17-5 'datafu_2_3_*'') 2016-06-06 12:58:56,266 - Package['hive_2_3_*'] {'use_repos': ['HDP-2.3.4.17-5', 'HDP-UTILS-2.3.4.17-5'], 'skip_repos': ['HDP-*']} 2016-06-06 12:58:56,267 - Installing package hive_2_3_* ('/usr/bin/yum -d 0 -e 0 -y install '--disablerepo=HDP-*' --enablerepo=HDP-2.3.4.17-5,HDP-UTILS-2.3.4.17-5 'hive_2_3_*'') 2016-06-06 12:59:57,687 - Package['hive_2_3_*-hcatalog'] {'use_repos': ['HDP-2.3.4.17-5', 'HDP-UTILS-2.3.4.17-5'], 'skip_repos': ['HDP-*']} 2016-06-06 12:59:57,687 - Installing package hive_2_3_*-hcatalog ('/usr/bin/yum -d 0 -e 0 -y install '--disablerepo=HDP-*' --enablerepo=HDP-2.3.4.17-5,HDP-UTILS-2.3.4.17-5 'hive_2_3_*-hcatalog'') 2016-06-06 12:59:57,738 - Caught signal 15, will handle it gracefully. Compute the actual version if possible before exiting. 2016-06-06 12:59:57,738 - Installation of packages failed. Checking if installation was partially complete 2016-06-06 12:59:57,738 - Old versions: ['2.3.4.7-4'] 2016-06-06 12:59:57,766 - New versions: ['2.3.2.0-2950', '2.3.4.7-4'] 2016-06-06 12:59:57,767 - Deltas: set(['2.3.2.0-2950']) 2016-06-06 12:59:57,767 - Appended value "2.3.4.17,2.3.2.0-2950" to file /var/lib/ambari-agent/data/repo_version_history.csv to track this as a new version. 2016-06-06 12:59:57,767 - Version 2.3.2.0-2950 was partially installed. 2016-06-06 12:59:57,767 - Package Manager failed to install packages. Error: (4, 'Interrupted system call') Traceback (most recent call last): File "/var/lib/ambari-agent/cache/custom_actions/scripts/install_packages.py", line 307, in install_packages skip_repos=[self.REPO_FILE_NAME_PREFIX + "*"] if OSCheck.is_redhat_family() else []) File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 154, in __init__ self.env.run() File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 152, in run self.run_action(resource, action) File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 118, in run_action provider_action() File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/__init__.py", line 45, in action_install self.install_package(package_name, self.resource.use_repos, self.resource.skip_repos) File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/yumrpm.py", line 49, in install_package shell.checked_call(cmd, sudo=True, logoutput=self.get_logoutput()) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in inner result = function(command, **kwargs) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in checked_call tries=tries, try_sleep=try_sleep) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in _call_wrapper result = _call(command, **kwargs_copy) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 240, in _call ready, _, _ = select.select(read_set, [], [], 1) error: (4, 'Interrupted system call') 2016-06-06 12:59:58,542 - Installation of packages failed. Checking if installation was partially complete 2016-06-06 12:59:58,543 - Old versions: ['2.3.4.7-4'] 2016-06-06 12:59:58,571 - New versions: ['2.3.2.0-2950', '2.3.4.7-4'] 2016-06-06 12:59:58,572 - Deltas: set(['2.3.2.0-2950']) 2016-06-06 12:59:58,572 - Appended value "2.3.4.17,2.3.2.0-2950" to file /var/lib/ambari-agent/data/repo_version_history.csv to track this as a new version. 2016-06-06 12:59:58,572 - Version 2.3.2.0-2950 was partially installed.
Created 06-06-2016 06:14 PM
I think this is something related to agent package install timeout, You can try to increase this value in ambari using below parameter if you have low bandwith internet.
vi /etc/ambari-server/conf/ambari.properties agent.package.install.task.timeout=5000
Created 06-06-2016 06:11 PM
It looks like it's attempting to do some work using HDP version 2.3.2.0-2950. Check the rpm repo configuration files on the node with the problems. Does it look the same as the other working nodes? What versions are actually installed/available in your cluster?
Created 06-06-2016 07:51 PM
Dear Michael,
I'm not aware of an installation of version 2.3.2.0-2950. Is it some kind of alias for version 2.3.4.17-5?
Current installed version is 2.3.4.7.
Created 06-06-2016 06:14 PM
I think this is something related to agent package install timeout, You can try to increase this value in ambari using below parameter if you have low bandwith internet.
vi /etc/ambari-server/conf/ambari.properties agent.package.install.task.timeout=5000
Created 06-06-2016 07:41 PM
@Thomaz Moreira can you share the screenshot please where exactly it got stuck?
Created 06-06-2016 07:48 PM
Sure. Today I won't be able to access our environment anymore, but tomorrow morning, surely.
I'll try to explain here, maybe you'll get it. After registering version 2.3.4.17-5 in Ambari's version tab, it was validated.
A drop down list was available to "apply to". So I selected my cluster.
So the new version got available in a blue box and the old version was showed in a green box. In the blue box for the new version, there was this "Install" button. So I clucked on it and things started to run. 3 out of 4 of my servers were updated, this one showed this error. But Now the blue box is not shown anymore, even though seems like the version is already registered.
there's no reinstall button. There's no action available in Ambari. I want to re-run it now. Try to install it all again. How can I do it?
Tomorrow I'll send you the screenshots.
Thank you.
Created 11-17-2016 12:08 PM
Thank you so much @Jitendra Yadav, It's worked for me.
Created 06-06-2016 07:27 PM
Well, it's simply strange cause I was really trying to update to version 2.3.4.17-5, nothing to do with version 2.3.2-2950, apparently.
Jitendra, I increased this value, but how can I try this installation again? Ambari isn't showing me any possibility to retry it. Is there a command line tool available for it?
Thank you