Created 05-17-2017 02:03 PM
My config: HDP 126.96.36.199 on Ubuntu14
I was trying to upgrade Ambari from 2.2.2 to 188.8.131.52 and I followed instructions in
Everything worked fine until I got to the point of Upgrading Ambari Metrics:
apt-get install ambari-metrics-collector
and got error that this package couldn't be found. My /etc/apt/sources.list.d/ambari.list is as follows:
#VERSION_NUMBER=184.108.40.206-7 deb http://public-repo-1.hortonworks.com/ambari/ubuntu14/2.x/updates/220.127.116.11 Ambari main
All other ambari packages seem to be installed fine (dpkg -l ambari-*):
ii ambari-agent 18.104.22.168-7 amd64 Ambari Agent ii ambari-metrics-assembly 22.214.171.124-7 amd64 Ambari Metrics Assembly ii ambari-server 126.96.36.199-7 amd64 Ambari Server
What should I do to install ambari-metrics-collector?
Created 05-17-2017 03:12 PM
Did you try running the "apt-get install ambari-metrics-collector" again? It might be a temporary n/w issue?
Else you might want to try downloading the ambari offline repository that has the required RPMs from:
For other OS versions offline repositories you can take a look at:
Created 05-18-2017 08:00 AM
Thanks for the hints.
I did try "apt-get install ambari-metrics-collector" couple of times for a few days with the same result.
I downloaded the ambari offline repository you mentioned, put it to ambari.list file and apt gives me the same answer. If you look into the contents of ambari-188.8.131.52-ubuntu14.tar.gz:ambari\ubuntu14\dists\Ambari\main\binary-amd64\Packages file, there is no such package defined as ambari-metrics-collector.
Maybe all necessary files of ambari-metrics-collector are already installed with different package?
There is /usr/sbin/ambari-metrics-collector file in the system, but I couldn't find the way to check its version.
Created 06-02-2017 11:53 AM
I did some more research and found that for Ubuntu you do not need to do the following step.
apt-get install ambari-metrics-collector
(For Ubuntu) Just 'apt-get install ambari-metrics-assembly' should be sufficient for upgrading Ambari metrics on Ubuntu 14.
Can you confirm by checking the ambari-metrics-*.jar versions in /usr/lib/ambari-metrics-collector.
# ls /usr/lib/ambari-metrics-collector/ambari-metrics-* /usr/lib/ambari-metrics-collector/ambari-metrics-common-184.108.40.206.7.jar /usr/lib/ambari-metrics-collector/ambari-metrics-timelineservice-220.127.116.11.7.jar
Created 06-02-2017 12:03 PM
Yes, these two jars with this version are there.
As I wrote a few minutes ago, I got back to Ambari 2.2.2 and will do the upgrade in a couple of weeks. I have one server with Ubuntu 16 and Ambari 2.5.0 for testing. So, what you wrote is true for Ubuntu 16 too.
Created 06-02-2017 12:09 PM
Yes that is true. For Ubuntu.
Please see the update from Ambari Metrics Collector Engineer @Aravindan Vijayan
Created 05-17-2017 08:08 PM
Run on of these 2 commands to clear the cache
sudo apt-get clean sudo apt-get autoclean
Then rerun you command your
sudo apt-get install ambari-metrics-collector
The installation should work
Created 05-18-2017 08:16 AM
Thanks, I tried it and with no success.
As I mentioned in comment to Jay SenSharma -I couldn't find this package in the tarball version of ambari repository, so maybe all necessary files are installed with different one/ones?
Created 05-18-2017 11:36 AM
in your initial steps did you execute the following in sequence
On cluster node running a Metrics Monitor
apt-get clean all apt-get update apt-get install ambari-metrics-assembly
Execute the following command on all hosts running the Metrics Collector
apt-get clean all apt-get update apt-get install ambari-metrics-collector
On the host running the Grafana component
apt-get clean all apt-get update apt-get install ambari-metrics-grafana
Start Ambari Metrics Service.
From Ambari Web, browse to Services > Ambari Metrics select Service Actions then choose Start.
Created 05-18-2017 11:58 AM
Yes, I went strictly with instructions and when I look into bash_history I'm sure I followed this sequence. But I didn't get to the Grafana Component because of error with ambari-metrics-collector.
Start action in Ambari Web is disabled for Ambari Metrics now, which is quite obvious.
I consider preparing new virtual machine for AMS to workaround this problem.
Created 05-18-2017 01:07 PM
I prepared fresh new virtual machine with Ubuntu16 with /etc/apt/sources.list.d/ambari.list:
#VERSION_NUMBER=18.104.22.168-7 deb http://public-repo-1.hortonworks.com/ambari/ubuntu16/2.x/updates/22.214.171.124 Ambari main
After installing packages: ambari-agent and ambari-metrics-assembly I noticed that there are the following commands available:
ambari-metrics-collector ambari-metrics-grafana ambari-metrics-monitor
Does it mean that ambari-metrics-collector is already installed and I don't need to install it with separate package?
The same situation is on the server I try to upgrade. How can I verify that ambari-metrics-[collector,grafana,monitor] are correctly upgraded?
Created 05-18-2017 04:23 PM
Its good alwasy to follow the hortonworks documentation.
Can you remove the offending package through Ambari UI?
List of the applications installed add a grep to filter:
sudo apt --installed list | more
Then remove a program run the following command:
sudo apt-get remove
To completely remove run the following command:
sudo apt-get remove --purge
Then try re-installing
Created 05-19-2017 02:43 PM
The situation gets more complicated 🙂
Now I have problem with ambari-agents which can't connect to ambari-server.
Ambari-server seems to work fine except AMS. Upgraded ambari-agent starts with the following error:
INFO 2017-05-19 12:49:41,405 main.py:145 - loglevel=logging.INFO INFO 2017-05-19 12:49:41,405 main.py:145 - loglevel=logging.INFO INFO 2017-05-19 12:49:41,405 main.py:145 - loglevel=logging.INFO INFO 2017-05-19 12:49:41,406 DataCleaner.py:39 - Data cleanup thread started INFO 2017-05-19 12:49:41,408 DataCleaner.py:120 - Data cleanup started INFO 2017-05-19 12:49:41,409 DataCleaner.py:122 - Data cleanup finished INFO 2017-05-19 12:49:41,437 PingPortListener.py:50 - Ping port listener started on port: 8670 INFO 2017-05-19 12:49:41,438 main.py:436 - Connecting to Ambari server at https://hadoop-metrics.pip:8440 (192.168.2.113) INFO 2017-05-19 12:49:41,438 NetUtil.py:67 - Connecting to https://hadoop-metrics.pip:8440 INFO 2017-05-19 12:49:41,623 main.py:446 - Connected to Ambari server hadoop-metrics.pip INFO 2017-05-19 12:49:41,624 threadpool.py:58 - Started thread pool with 3 core threads and 20 maximum threads WARNING 2017-05-19 12:49:41,624 AlertSchedulerHandler.py:280 - [AlertScheduler] /var/lib/ambari-agent/cache/alerts/definitions.json not found or invalid. No alerts will be scheduled until registration occurs. INFO 2017-05-19 12:49:41,624 AlertSchedulerHandler.py:175 - [AlertScheduler] Starting <ambari_agent.apscheduler.scheduler.Scheduler object at 0x7fbf67547c90>; currently running: False ERROR 2017-05-19 12:49:41,629 Controller.py:502 - Controller thread failed with exception: Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/ambari_agent/Controller.py", line 482, in run self.actionQueue = ActionQueue(self.config, controller=self) File "/usr/lib/python2.6/site-packages/ambari_agent/ActionQueue.py", line 79, in __init__ self.statusCommandResultQueue = multiprocessing.Queue() # this queue is filled by StatuCommandsExecutor. File "/usr/lib/python2.7/multiprocessing/__init__.py", line 218, in Queue return Queue(maxsize) File "/usr/lib/python2.7/multiprocessing/queues.py", line 63, in __init__ self._rlock = Lock() File "/usr/lib/python2.7/multiprocessing/synchronize.py", line 147, in __init__ SemLock.__init__(self, SEMAPHORE, 1, 1) File "/usr/lib/python2.7/multiprocessing/synchronize.py", line 75, in __init__ sl = self._semlock = _multiprocessing.SemLock(kind, value, maxvalue) OSError: [Errno 30] Read-only file system ERROR 2017-05-19 12:49:43,628 main.py:476 - Exiting with exception: Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/ambari_agent/main.py", line 471, in <module> main(heartbeat_stop_callback) File "/usr/lib/python2.6/site-packages/ambari_agent/main.py", line 450, in main run_threads(server_hostname, heartbeat_stop_callback) File "/usr/lib/python2.6/site-packages/ambari_agent/main.py", line 340, in run_threads controller.get_status_commands_executor().kill("AGENT_STOPPED", can_relaunch=False) AttributeError: 'NoneType' object has no attribute 'kill' INFO 2017-05-19 12:49:43,630 ExitHelper.py:56 - Performing cleanup before exiting... INFO 2017-05-19 12:49:43,630 threadpool.py:120 - Shutting down thread pool INFO 2017-05-19 12:49:43,630 scheduler.py:606 - Scheduler has been shut down INFO 2017-05-19 12:49:43,631 threadpool.py:58 - Started thread pool with 3 core threads and 20 maximum threads INFO 2017-05-19 12:49:43,631 AlertSchedulerHandler.py:185 - [AlertScheduler] Stopped the alert scheduler. INFO 2017-05-19 12:49:43,632 threadpool.py:120 - Shutting down thread pool INFO 2017-05-19 12:49:43,639 Controller.py:151 - Server connection disconnected.
On the ambari-server side there is only such log at the same time:
19 maj 2017 12:49:41,621 WARN [qtp-ambari-agent-43] SecurityFilter:103 - Request https://hadoop-metrics.pip:8440/ca doesn't match any pattern. 19 maj 2017 12:49:41,621 WARN [qtp-ambari-agent-43] SecurityFilter:62 - This request is not allowed on this port: https://hadoop-metrics.pip:8440/ca
It can't be networking problem - it's on the same node. Ambari-agents on other nodes behave the same way.
I also gave a try of installing ambari-server on a completely fresh server with Ubuntu16 following instructions from https://docs.hortonworks.com/HDPDocuments/Ambari-126.96.36.199/bk_ambari-reference/content/install_the_new...
It gave me the same kind of errors. Maybe this can help in finding the cause of this problem.
Created 05-19-2017 04:21 PM
Okay now to validate the process,do the following
apt-get install ambari-agent
ambari-agent.inifile as shown in the following example:
The agents should now register with the Ambari server <your.ambari.server.hostname> and this should be the FQDN of the Ambari server
Created 06-02-2017 11:48 AM
Sorry for delay, I had a week off...
Thanks for your hints and engagement.
I decided to roll back to Ambari version 2.2.2 to take back control over systems. In a few weeks I will give it another try.