Cloudera Community

Support Questions

Find answers, ask questions, and share your expertise
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Advanced Search
Solved Go to solution

Ambari Metrics Won't Install

Labels:
avatar
author-rank rgarcia
Expert Contributor

Created ‎11-05-2015 04:47 AM

On a 6 node cluster, using Ambari 2.1.2/HDP 2.3.2.

Scenario 1:

When installing HDP and it goes to a point where it's installing the services across all nodes, it suddenly fails and it's due to Ambari Metrics/Monitors failing. Full stack trace below from Ambari UI.

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts/metrics_collector.py", line 131, in <module>
    AmsCollector().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 219, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts/metrics_collector.py", line 34, in install
    self.install_packages(env)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 395, in install_packages
    Package(name)
  File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 154, in __init__
    self.env.run()
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 152, in run
    self.run_action(resource, action)
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 118, in run_action
    provider_action()
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/__init__.py", line 45, in action_install
    self.install_package(package_name, self.resource.use_repos, self.resource.skip_repos)
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/yumrpm.py", line 49, in install_package
    shell.checked_call(cmd, sudo=True, logoutput=self.get_logoutput())
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in inner
    result = function(command, **kwargs)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in checked_call
    tries=tries, try_sleep=try_sleep)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in _call_wrapper
    result = _call(command, **kwargs_copy)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291, in _call
    raise Fail(err_msg)
resource_management.core.exceptions.Fail: Execution of '/usr/bin/yum -d 0 -e 0 -y install ambari-metrics-collector' returned 1. Error: Nothing to do

Running the same command from the last line of the error yields the same response. There were no Ambari Metrics log generated. Ambari server log didn't any info.

Scenario 2:

I reset ambari-server and cleanup all hosts. Re-run ambari wizard and install all services except for Ambari Metrics. HDP installed successfully. I now added Ambari Metrics back and I'm getting the same error. 

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts/metrics_collector.py", line 131, in <module>
    AmsCollector().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 219, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts/metrics_collector.py", line 34, in install
    self.install_packages(env)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 395, in install_packages
    Package(name)
  File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 154, in __init__
    self.env.run()
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 152, in run
    self.run_action(resource, action)
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 118, in run_action
    provider_action()
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/__init__.py", line 45, in action_install
    self.install_package(package_name, self.resource.use_repos, self.resource.skip_repos)
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/yumrpm.py", line 49, in install_package
    shell.checked_call(cmd, sudo=True, logoutput=self.get_logoutput())
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in inner
    result = function(command, **kwargs)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in checked_call
    tries=tries, try_sleep=try_sleep)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in _call_wrapper
    result = _call(command, **kwargs_copy)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291, in _call
    raise Fail(err_msg)
resource_management.core.exceptions.Fail: Execution of '/usr/bin/yum -d 0 -e 0 -y install ambari-metrics-collector' returned 1. Error: Nothing to do
13,657 Views
1 ACCEPTED SOLUTION

avatar
author-rank abajwa author-rank
Guru

Created ‎11-06-2015 02:56 AM

Just got off webex with @rgarcia@hortonworks.com

We were able to remove the failed Ranger/Metrics installation using something like the below:

su postgres
psql
\c ambari
DELETE FROM ambari.hostcomponentstate WHERE service_name IN ('RANGER');
DELETE FROM ambari.hostcomponentdesiredstate WHERE service_name IN ('RANGER');
DELETE FROM ambari.servicecomponentdesiredstate WHERE service_name IN ('RANGER');
DELETE FROM ambari.servicedesiredstate WHERE service_name IN ('RANGER');
DELETE FROM ambari.clusterservices WHERE service_name IN ('RANGER');

Then make sure to restart Ambari

service ambari-server restart

Now re-install Ranger/Ambair metrics

View solution in original post

8,685 Views
28 REPLIES 28

avatar
author-rank abajwa author-rank
Guru

Created ‎11-05-2015 04:56 AM

Can you run the yum install manually and see what happens?

yum install ambari-metrics-collector
4,936 Views

avatar
author-rank rgarcia
Expert Contributor

Created ‎11-05-2015 04:59 AM 

[root@great-wall02 ~]# yum install ambari-metrics-collector

Loaded plugins: fastestmirror

Loading mirror speeds from cached hostfile

 * base: mirror.spro.net

 * extras: mirrors.sonic.net

 * updates: centos-distro.cavecreek.net

Setting up Install Process

No package ambari-metrics-collector available.

Error: Nothing to do

[root@great-wall02 ~]# rpm -q ambari-metrics-collector

package ambari-metrics-collector is not installed
4,936 Views
0 Kudos

avatar
author-rank nsabharwal
Master Mentor

Created ‎11-05-2015 05:04 AM

@rgarcia@hortonworks.com

Can you verify the repo version?

try yum repo list in all the nodes + if you are using repo for correct OS

4,936 Views
0 Kudos

avatar
author-rank rgarcia
Expert Contributor

Created ‎11-05-2015 05:20 AM

All nodes has the same repo version. I noticed that ambari.repo is gone. Possibly got deleted during host cleanup (python script)? Downloaded the ambari.repo and reinstalling ambari metrics. Stay tuned.

4,936 Views

avatar
author-rank rgarcia
Expert Contributor

Created ‎11-05-2015 05:02 AM

Tried deleting Ambari Metrics from Ambari throug DELETE REST API and rerunning yum install ambari-metrics-collector. Same output.

[root@great-wall02 ~]# curl -u admin:admin -H "X-Requested-By: ambari" -X DELETE  http://great-wall01.cloud.hortonworks.com:8080/api/v1/clusters/smesecurity/services/AMBARI_METRICS

[root@great-wall02 ~]# yum install ambari-metrics-collector

Loaded plugins: fastestmirror

Loading mirror speeds from cached hostfile

 * base: mirror.spro.net

 * extras: mirrors.sonic.net

 * updates: mirror.hostduplex.com

Setting up Install Process

No package ambari-metrics-collector available.

Error: Nothing to do
4,936 Views
0 Kudos

avatar
author-rank rgarcia
Expert Contributor

Created ‎11-05-2015 05:06 AM

After removing Ambari Metrics, its got a side effect on Kafka. Kafka shouldn't be having problems after this process since it was running fine prior.

[2015-11-04 20:58:47,740] FATAL  (kafka.Kafka$)

java.lang.ClassNotFoundException: org.apache.hadoop.metrics2.sink.kafka.KafkaTimelineMetricsReporter

        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)

        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)

        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

        at java.lang.Class.forName0(Native Method)

        at java.lang.Class.forName(Class.java:264)

        at kafka.utils.CoreUtils$.createObject(CoreUtils.scala:231)

        at kafka.metrics.KafkaMetricsReporter$$anonfun$startReporters$1.apply(KafkaMetricsReporter.scala:59)

        at kafka.metrics.KafkaMetricsReporter$$anonfun$startReporters$1.apply(KafkaMetricsReporter.scala:58)

        at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)

        at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:34)

        at kafka.metrics.KafkaMetricsReporter$.startReporters(KafkaMetricsReporter.scala:58)

        at kafka.Kafka$.main(Kafka.scala:62)

        at kafka.Kafka.main(Kafka.scala)

[2015-11-04 21:00:51,935] FATAL  (kafka.Kafka$)

java.lang.ClassNotFoundException: org.apache.hadoop.metrics2.sink.kafka.KafkaTimelineMetricsReporter

        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)

        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)

        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

        at java.lang.Class.forName0(Native Method)

        at java.lang.Class.forName(Class.java:264)

        at kafka.utils.CoreUtils$.createObject(CoreUtils.scala:231)

        at kafka.metrics.KafkaMetricsReporter$$anonfun$startReporters$1.apply(KafkaMetricsReporter.scala:59)

        at kafka.metrics.KafkaMetricsReporter$$anonfun$startReporters$1.apply(KafkaMetricsReporter.scala:58)

        at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)

        at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:34)

        at kafka.metrics.KafkaMetricsReporter$.startReporters(KafkaMetricsReporter.scala:58)

        at kafka.Kafka$.main(Kafka.scala:62)

        at kafka.Kafka.main(Kafka.scala)
4,936 Views
0 Kudos

avatar
author-rank rgarcia
Expert Contributor

Created ‎11-05-2015 05:30 AM

Getting this amabri server error now. This came up after clicking deploy (for Ranger, Ranger KMS and Ambari Metrics)

Caused by: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "clusterservices_pkey"

	at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2161)

	at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1890)

	at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)

	at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:559)

	at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:417)

	at org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(AbstractJdbc2Statement.java:363)

	at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeDirectNoSelect(DatabaseAccessor.java:890)

	... 130 more
4,936 Views
0 Kudos

avatar
author-rank rgarcia
Expert Contributor

Created on ‎11-05-2015 05:35 AM - edited ‎08-19-2019 05:52 AM

This is what's coming up in Ambari.

396-untitled-5.png


untitled-5.png
4,936 Views
0 Kudos

avatar
author-rank nsabharwal
Master Mentor

Created ‎11-05-2015 12:06 PM

@jeff@hortonworks.com

@mahadev@hortonworks.com

Please see this

4,936 Views
0 Kudos
Announcements
What's New @ Cloudera
Cloudera DataFlow 2.9 now supports building GenAI pipelines ...
What's New @ Cloudera
Accelerate Data Scientist Productivity with Cloudera Copilot...
What's New @ Cloudera
[RELEASED] Cloudera Streaming Analytics - Kubernetes Operato...
Community Announcements
October 2024 Community Highlights
What's New @ Cloudera
Accelerate Your AI with the Cloudera AI Inference Service wi...
View More Announcements