Support Questions

Find answers, ask questions, and share your expertise
Check out our newest addition to the community, the Cloudera Data Analytics (CDA) group hub.

Problem with Ambari Metrics Collector while upgrading Ambari 2.2.2 ->



My config: HDP on Ubuntu14

I was trying to upgrade Ambari from 2.2.2 to and I followed instructions in

Everything worked fine until I got to the point of Upgrading Ambari Metrics:

apt-get install ambari-metrics-collector

and got error that this package couldn't be found. My /etc/apt/sources.list.d/ambari.list is as follows:

deb Ambari main

All other ambari packages seem to be installed fine (dpkg -l ambari-*):

ii  ambari-agent           amd64                     Ambari Agent
ii  ambari-metrics-assembly      amd64                     Ambari Metrics Assembly
ii  ambari-server          amd64                     Ambari Server

What should I do to install ambari-metrics-collector?


Super Mentor

@Darek Rutkowski

Did you try running the "apt-get install ambari-metrics-collector" again? It might be a temporary n/w issue?

Else you might want to try downloading the ambari offline repository that has the required RPMs from:


For other OS versions offline repositories you can take a look at:


Thanks for the hints.

I did try "apt-get install ambari-metrics-collector" couple of times for a few days with the same result.

I downloaded the ambari offline repository you mentioned, put it to ambari.list file and apt gives me the same answer. If you look into the contents of ambari-\ubuntu14\dists\Ambari\main\binary-amd64\Packages file, there is no such package defined as ambari-metrics-collector.

Maybe all necessary files of ambari-metrics-collector are already installed with different package?

There is /usr/sbin/ambari-metrics-collector file in the system, but I couldn't find the way to check its version.

Super Mentor

@Darek Rutkowski

I did some more research and found that for Ubuntu you do not need to do the following step.

apt-get install ambari-metrics-collector


(For Ubuntu) Just 'apt-get install ambari-metrics-assembly' should be sufficient for upgrading Ambari metrics on Ubuntu 14.

Can you confirm by checking the ambari-metrics-*.jar versions in /usr/lib/ambari-metrics-collector.

# ls /usr/lib/ambari-metrics-collector/ambari-metrics-* 



Yes, these two jars with this version are there.

As I wrote a few minutes ago, I got back to Ambari 2.2.2 and will do the upgrade in a couple of weeks. I have one server with Ubuntu 16 and Ambari 2.5.0 for testing. So, what you wrote is true for Ubuntu 16 too.

Super Mentor


@Darek Rutkowski

Run on of these 2 commands to clear the cache

sudo apt-get clean 
sudo apt-get autoclean

Then rerun you command your

sudo apt-get install ambari-metrics-collector 

The installation should work


Thanks, I tried it and with no success.

As I mentioned in comment to Jay SenSharma -I couldn't find this package in the tarball version of ambari repository, so maybe all necessary files are installed with different one/ones?


@Darek Rutkowski

in your initial steps did you execute the following in sequence

On cluster node running a Metrics Monitor

apt-get clean all
apt-get update
apt-get install ambari-metrics-assembly 

Execute the following command on all hosts running the Metrics Collector

apt-get clean all
apt-get update
apt-get install ambari-metrics-collector 

On the host running the Grafana component

apt-get clean all
apt-get update
apt-get install ambari-metrics-grafana 

Start Ambari Metrics Service.

From Ambari Web, browse to Services > Ambari Metrics select Service Actions then choose Start.

Please revert


Yes, I went strictly with instructions and when I look into bash_history I'm sure I followed this sequence. But I didn't get to the Grafana Component because of error with ambari-metrics-collector.

Start action in Ambari Web is disabled for Ambari Metrics now, which is quite obvious.

I consider preparing new virtual machine for AMS to workaround this problem.


I prepared fresh new virtual machine with Ubuntu16 with /etc/apt/sources.list.d/ambari.list:

deb Ambari main

After installing packages: ambari-agent and ambari-metrics-assembly I noticed that there are the following commands available:

ambari-metrics-collector ambari-metrics-grafana ambari-metrics-monitor

Does it mean that ambari-metrics-collector is already installed and I don't need to install it with separate package?

The same situation is on the server I try to upgrade. How can I verify that ambari-metrics-[collector,grafana,monitor] are correctly upgraded?


@Darek Rutkowski

Its good alwasy to follow the hortonworks documentation.

Can you remove the offending package through Ambari UI?

List of the applications installed add a grep to filter:

sudo apt --installed list | more 

Then remove a program run the following command:

sudo apt-get remove 

To completely remove run the following command:

sudo apt-get remove --purge 

Then try re-installing


The situation gets more complicated 🙂

Now I have problem with ambari-agents which can't connect to ambari-server.

Ambari-server seems to work fine except AMS. Upgraded ambari-agent starts with the following error:

INFO 2017-05-19 12:49:41,405 - loglevel=logging.INFO
INFO 2017-05-19 12:49:41,405 - loglevel=logging.INFO
INFO 2017-05-19 12:49:41,405 - loglevel=logging.INFO
INFO 2017-05-19 12:49:41,406 - Data cleanup thread started
INFO 2017-05-19 12:49:41,408 - Data cleanup started
INFO 2017-05-19 12:49:41,409 - Data cleanup finished
INFO 2017-05-19 12:49:41,437 - Ping port listener started on port: 8670
INFO 2017-05-19 12:49:41,438 - Connecting to Ambari server at https://hadoop-metrics.pip:8440 (
INFO 2017-05-19 12:49:41,438 - Connecting to https://hadoop-metrics.pip:8440
INFO 2017-05-19 12:49:41,623 - Connected to Ambari server hadoop-metrics.pip
INFO 2017-05-19 12:49:41,624 - Started thread pool with 3 core threads and 20 maximum threads
WARNING 2017-05-19 12:49:41,624 - [AlertScheduler] /var/lib/ambari-agent/cache/alerts/definitions.json not found or invalid. No alerts will be scheduled until registration occurs.
INFO 2017-05-19 12:49:41,624 - [AlertScheduler] Starting <ambari_agent.apscheduler.scheduler.Scheduler object at 0x7fbf67547c90>; currently running: False
ERROR 2017-05-19 12:49:41,629 - Controller thread failed with exception:
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/ambari_agent/", line 482, in run
    self.actionQueue = ActionQueue(self.config, controller=self)
  File "/usr/lib/python2.6/site-packages/ambari_agent/", line 79, in __init__
    self.statusCommandResultQueue = multiprocessing.Queue() # this queue is filled by StatuCommandsExecutor.
  File "/usr/lib/python2.7/multiprocessing/", line 218, in Queue
    return Queue(maxsize)
  File "/usr/lib/python2.7/multiprocessing/", line 63, in __init__
    self._rlock = Lock()
  File "/usr/lib/python2.7/multiprocessing/", line 147, in __init__
    SemLock.__init__(self, SEMAPHORE, 1, 1)
  File "/usr/lib/python2.7/multiprocessing/", line 75, in __init__
    sl = self._semlock = _multiprocessing.SemLock(kind, value, maxvalue)
OSError: [Errno 30] Read-only file system
ERROR 2017-05-19 12:49:43,628 - Exiting with exception:
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/ambari_agent/", line 471, in <module>
  File "/usr/lib/python2.6/site-packages/ambari_agent/", line 450, in main
    run_threads(server_hostname, heartbeat_stop_callback)
  File "/usr/lib/python2.6/site-packages/ambari_agent/", line 340, in run_threads
    controller.get_status_commands_executor().kill("AGENT_STOPPED", can_relaunch=False)
AttributeError: 'NoneType' object has no attribute 'kill'
INFO 2017-05-19 12:49:43,630 - Performing cleanup before exiting...
INFO 2017-05-19 12:49:43,630 - Shutting down thread pool
INFO 2017-05-19 12:49:43,630 - Scheduler has been shut down
INFO 2017-05-19 12:49:43,631 - Started thread pool with 3 core threads and 20 maximum threads
INFO 2017-05-19 12:49:43,631 - [AlertScheduler] Stopped the alert scheduler.
INFO 2017-05-19 12:49:43,632 - Shutting down thread pool
INFO 2017-05-19 12:49:43,639 - Server connection disconnected.

On the ambari-server side there is only such log at the same time:

19 maj 2017 12:49:41,621  WARN [qtp-ambari-agent-43] SecurityFilter:103 - Request https://hadoop-metrics.pip:8440/ca doesn't match any pattern.
19 maj 2017 12:49:41,621  WARN [qtp-ambari-agent-43] SecurityFilter:62 - This request is not allowed on this port: https://hadoop-metrics.pip:8440/ca

It can't be networking problem - it's on the same node. Ambari-agents on other nodes behave the same way.

I also gave a try of installing ambari-server on a completely fresh server with Ubuntu16 following instructions from

It gave me the same kind of errors. Maybe this can help in finding the cause of this problem.


@Darek Rutkowski

Okay now to validate the process,do the following


  1. Install the Ambari Agent on every host in your cluster.

    apt-get install ambari-agent

  2. Configure the Ambari Agent by editing the ambari-agent.ini file as shown in the following example:

    vi /etc/ambari-agent/conf/ambari-agent.ini





  3. Start the agents on every host in your cluster.

    ambari-agent start

    The agents should now register with the Ambari server <your.ambari.server.hostname> and this should be the FQDN of the Ambari server


Sorry for delay, I had a week off...

Thanks for your hints and engagement.

I decided to roll back to Ambari version 2.2.2 to take back control over systems. In a few weeks I will give it another try.

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.