Problem with Ambari Metrics Collector while upgrading Ambari 2.2.2 -> 2.5.0.3

[#1]

Hi,

My config: HDP 2.4.2.0 on Ubuntu14

I was trying to upgrade Ambari from 2.2.2 to 2.5.0.3 and I followed instructions in

https://docs.hortonworks.com/HDPDocuments/Ambari-2.5.0.3/bk_ambari-upgrade/content/upgrading_ambari....

Everything worked fine until I got to the point of Upgrading Ambari Metrics:

apt-get install ambari-metrics-collector

and got error that this package couldn't be found. My /etc/apt/sources.list.d/ambari.list is as follows:

#VERSION_NUMBER=2.5.0.3-7
deb http://public-repo-1.hortonworks.com/ambari/ubuntu14/2.x/updates/2.5.0.3 Ambari main

All other ambari packages seem to be installed fine (dpkg -l ambari-*):

ii  ambari-agent               2.5.0.3-7      amd64                     Ambari Agent
ii  ambari-metrics-assembly    2.5.0.3-7      amd64                     Ambari Metrics Assembly
ii  ambari-server              2.5.0.3-7      amd64                     Ambari Server

What should I do to install ambari-metrics-collector?

[#2]

@Darek Rutkowski

Did you try running the "apt-get install ambari-metrics-collector" again? It might be a temporary n/w issue?

Else you might want to try downloading the ambari offline repository that has the required RPMs from:

http://public-repo-1.hortonworks.com/ambari/ubuntu14/2.x/updates/2.5.0.3/ambari-2.5.0.3-ubuntu14.tar...

.

For other OS versions offline repositories you can take a look at:

http://docs.hortonworks.com/HDPDocuments/Ambari-2.5.0.3/bk_ambari-installation/content/ambari_reposi...

[#3]

Thanks for the hints.

I did try "apt-get install ambari-metrics-collector" couple of times for a few days with the same result.

I downloaded the ambari offline repository you mentioned, put it to ambari.list file and apt gives me the same answer. If you look into the contents of ambari-2.5.0.3-ubuntu14.tar.gz:ambari\ubuntu14\dists\Ambari\main\binary-amd64\Packages file, there is no such package defined as ambari-metrics-collector.

Maybe all necessary files of ambari-metrics-collector are already installed with different package?

There is /usr/sbin/ambari-metrics-collector file in the system, but I couldn't find the way to check its version.

[#4]

@Darek Rutkowski

I did some more research and found that for Ubuntu you do not need to do the following step.

apt-get install ambari-metrics-collector

.

(For Ubuntu) Just 'apt-get install ambari-metrics-assembly' should be sufficient for upgrading Ambari metrics on Ubuntu 14.

Can you confirm by checking the ambari-metrics-*.jar versions in /usr/lib/ambari-metrics-collector.

# ls /usr/lib/ambari-metrics-collector/ambari-metrics-* 
    /usr/lib/ambari-metrics-collector/ambari-metrics-common-2.5.0.3.7.jar 
    /usr/lib/ambari-metrics-collector/ambari-metrics-timelineservice-2.5.0.3.7.jar

.

[#5]

Yes, these two jars with this version are there.

As I wrote a few minutes ago, I got back to Ambari 2.2.2 and will do the upgrade in a couple of weeks. I have one server with Ubuntu 16 and Ambari 2.5.0 for testing. So, what you wrote is true for Ubuntu 16 too.

[#6]

@Darek Rutkowski

Yes that is true. For Ubuntu.

Please see the update from Ambari Metrics Collector Engineer @Aravindan Vijayan

https://community.hortonworks.com/questions/105281/upgrading-ambari-to-25-unable-to-locate-ambari-me...

[#7]

@Darek Rutkowski

Run on of these 2 commands to clear the cache

sudo apt-get clean 
sudo apt-get autoclean

Then rerun you command your

sudo apt-get install ambari-metrics-collector 

The installation should work

[#8]

Thanks, I tried it and with no success.

As I mentioned in comment to Jay SenSharma -I couldn't find this package in the tarball version of ambari repository, so maybe all necessary files are installed with different one/ones?

[#9]

@Darek Rutkowski

in your initial steps did you execute the following in sequence

On cluster node running a Metrics Monitor

apt-get clean all
apt-get update
apt-get install ambari-metrics-assembly 

Execute the following command on all hosts running the Metrics Collector

apt-get clean all
apt-get update
apt-get install ambari-metrics-collector 

On the host running the Grafana component

apt-get clean all
apt-get update
apt-get install ambari-metrics-grafana 

Start Ambari Metrics Service.

From Ambari Web, browse to Services > Ambari Metrics select Service Actions then choose Start.

Please revert

[#10]

Yes, I went strictly with instructions and when I look into bash_history I'm sure I followed this sequence. But I didn't get to the Grafana Component because of error with ambari-metrics-collector.

Start action in Ambari Web is disabled for Ambari Metrics now, which is quite obvious.

I consider preparing new virtual machine for AMS to workaround this problem.

[#11]

I prepared fresh new virtual machine with Ubuntu16 with /etc/apt/sources.list.d/ambari.list:

#VERSION_NUMBER=2.5.0.3-7
deb http://public-repo-1.hortonworks.com/ambari/ubuntu16/2.x/updates/2.5.0.3 Ambari main

After installing packages: ambari-agent and ambari-metrics-assembly I noticed that there are the following commands available:

ambari-metrics-collector ambari-metrics-grafana ambari-metrics-monitor

Does it mean that ambari-metrics-collector is already installed and I don't need to install it with separate package?

The same situation is on the server I try to upgrade. How can I verify that ambari-metrics-[collector,grafana,monitor] are correctly upgraded?

[#12]

@Darek Rutkowski

Its good alwasy to follow the hortonworks documentation.

Can you remove the offending package through Ambari UI?

List of the applications installed add a grep to filter:

sudo apt --installed list | more 

Then remove a program run the following command:

sudo apt-get remove 

To completely remove run the following command:

sudo apt-get remove --purge 

Then try re-installing

[#13]

The situation gets more complicated 🙂

Now I have problem with ambari-agents which can't connect to ambari-server.

Ambari-server seems to work fine except AMS. Upgraded ambari-agent starts with the following error:

INFO 2017-05-19 12:49:41,405 main.py:145 - loglevel=logging.INFO
INFO 2017-05-19 12:49:41,405 main.py:145 - loglevel=logging.INFO
INFO 2017-05-19 12:49:41,405 main.py:145 - loglevel=logging.INFO
INFO 2017-05-19 12:49:41,406 DataCleaner.py:39 - Data cleanup thread started
INFO 2017-05-19 12:49:41,408 DataCleaner.py:120 - Data cleanup started
INFO 2017-05-19 12:49:41,409 DataCleaner.py:122 - Data cleanup finished
INFO 2017-05-19 12:49:41,437 PingPortListener.py:50 - Ping port listener started on port: 8670
INFO 2017-05-19 12:49:41,438 main.py:436 - Connecting to Ambari server at https://hadoop-metrics.pip:8440 (192.168.2.113)
INFO 2017-05-19 12:49:41,438 NetUtil.py:67 - Connecting to https://hadoop-metrics.pip:8440
INFO 2017-05-19 12:49:41,623 main.py:446 - Connected to Ambari server hadoop-metrics.pip
INFO 2017-05-19 12:49:41,624 threadpool.py:58 - Started thread pool with 3 core threads and 20 maximum threads
WARNING 2017-05-19 12:49:41,624 AlertSchedulerHandler.py:280 - [AlertScheduler] /var/lib/ambari-agent/cache/alerts/definitions.json not found or invalid. No alerts will be scheduled until registration occurs.
INFO 2017-05-19 12:49:41,624 AlertSchedulerHandler.py:175 - [AlertScheduler] Starting <ambari_agent.apscheduler.scheduler.Scheduler object at 0x7fbf67547c90>; currently running: False
ERROR 2017-05-19 12:49:41,629 Controller.py:502 - Controller thread failed with exception:
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/ambari_agent/Controller.py", line 482, in run
    self.actionQueue = ActionQueue(self.config, controller=self)
  File "/usr/lib/python2.6/site-packages/ambari_agent/ActionQueue.py", line 79, in __init__
    self.statusCommandResultQueue = multiprocessing.Queue() # this queue is filled by StatuCommandsExecutor.
  File "/usr/lib/python2.7/multiprocessing/__init__.py", line 218, in Queue
    return Queue(maxsize)
  File "/usr/lib/python2.7/multiprocessing/queues.py", line 63, in __init__
    self._rlock = Lock()
  File "/usr/lib/python2.7/multiprocessing/synchronize.py", line 147, in __init__
    SemLock.__init__(self, SEMAPHORE, 1, 1)
  File "/usr/lib/python2.7/multiprocessing/synchronize.py", line 75, in __init__
    sl = self._semlock = _multiprocessing.SemLock(kind, value, maxvalue)
OSError: [Errno 30] Read-only file system
ERROR 2017-05-19 12:49:43,628 main.py:476 - Exiting with exception:
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/ambari_agent/main.py", line 471, in <module>
    main(heartbeat_stop_callback)
  File "/usr/lib/python2.6/site-packages/ambari_agent/main.py", line 450, in main
    run_threads(server_hostname, heartbeat_stop_callback)
  File "/usr/lib/python2.6/site-packages/ambari_agent/main.py", line 340, in run_threads
    controller.get_status_commands_executor().kill("AGENT_STOPPED", can_relaunch=False)
AttributeError: 'NoneType' object has no attribute 'kill'
INFO 2017-05-19 12:49:43,630 ExitHelper.py:56 - Performing cleanup before exiting...
INFO 2017-05-19 12:49:43,630 threadpool.py:120 - Shutting down thread pool
INFO 2017-05-19 12:49:43,630 scheduler.py:606 - Scheduler has been shut down
INFO 2017-05-19 12:49:43,631 threadpool.py:58 - Started thread pool with 3 core threads and 20 maximum threads
INFO 2017-05-19 12:49:43,631 AlertSchedulerHandler.py:185 - [AlertScheduler] Stopped the alert scheduler.
INFO 2017-05-19 12:49:43,632 threadpool.py:120 - Shutting down thread pool
INFO 2017-05-19 12:49:43,639 Controller.py:151 - Server connection disconnected.

On the ambari-server side there is only such log at the same time:

19 maj 2017 12:49:41,621  WARN [qtp-ambari-agent-43] SecurityFilter:103 - Request https://hadoop-metrics.pip:8440/ca doesn't match any pattern.
19 maj 2017 12:49:41,621  WARN [qtp-ambari-agent-43] SecurityFilter:62 - This request is not allowed on this port: https://hadoop-metrics.pip:8440/ca

It can't be networking problem - it's on the same node. Ambari-agents on other nodes behave the same way.

I also gave a try of installing ambari-server on a completely fresh server with Ubuntu16 following instructions from https://docs.hortonworks.com/HDPDocuments/Ambari-2.4.2.0/bk_ambari-reference/content/install_the_new...

It gave me the same kind of errors. Maybe this can help in finding the cause of this problem.

[#14]

@Darek Rutkowski

Okay now to validate the process,do the following

Ubuntu

1. Install the Ambari Agent on every host in your cluster.

   apt-get install ambari-agent

2. Configure the Ambari Agent by editing the ambari-agent.ini file as shown in the following example:

   vi /etc/ambari-agent/conf/ambari-agent.ini

   [server]

   hostname=<your.ambari.server.hostname>

   url_port=8440

   secured_url_port=8441

3. Start the agents on every host in your cluster.

   ambari-agent start

   The agents should now register with the Ambari server <your.ambari.server.hostname> and this should be the FQDN of the Ambari server

[#15]

Sorry for delay, I had a week off...

Thanks for your hints and engagement.

I decided to roll back to Ambari version 2.2.2 to take back control over systems. In a few weeks I will give it another try.