Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Ambari Metrics Won't Install

Solved Go to solution

Ambari Metrics Won't Install

Expert Contributor

On a 6 node cluster, using Ambari 2.1.2/HDP 2.3.2.

Scenario 1:

When installing HDP and it goes to a point where it's installing the services across all nodes, it suddenly fails and it's due to Ambari Metrics/Monitors failing. Full stack trace below from Ambari UI.

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts/metrics_collector.py", line 131, in <module>
    AmsCollector().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 219, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts/metrics_collector.py", line 34, in install
    self.install_packages(env)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 395, in install_packages
    Package(name)
  File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 154, in __init__
    self.env.run()
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 152, in run
    self.run_action(resource, action)
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 118, in run_action
    provider_action()
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/__init__.py", line 45, in action_install
    self.install_package(package_name, self.resource.use_repos, self.resource.skip_repos)
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/yumrpm.py", line 49, in install_package
    shell.checked_call(cmd, sudo=True, logoutput=self.get_logoutput())
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in inner
    result = function(command, **kwargs)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in checked_call
    tries=tries, try_sleep=try_sleep)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in _call_wrapper
    result = _call(command, **kwargs_copy)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291, in _call
    raise Fail(err_msg)
resource_management.core.exceptions.Fail: Execution of '/usr/bin/yum -d 0 -e 0 -y install ambari-metrics-collector' returned 1. Error: Nothing to do

Running the same command from the last line of the error yields the same response. There were no Ambari Metrics log generated. Ambari server log didn't any info.

Scenario 2:

I reset ambari-server and cleanup all hosts. Re-run ambari wizard and install all services except for Ambari Metrics. HDP installed successfully. I now added Ambari Metrics back and I'm getting the same error.

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts/metrics_collector.py", line 131, in <module>
    AmsCollector().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 219, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts/metrics_collector.py", line 34, in install
    self.install_packages(env)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 395, in install_packages
    Package(name)
  File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 154, in __init__
    self.env.run()
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 152, in run
    self.run_action(resource, action)
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 118, in run_action
    provider_action()
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/__init__.py", line 45, in action_install
    self.install_package(package_name, self.resource.use_repos, self.resource.skip_repos)
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/yumrpm.py", line 49, in install_package
    shell.checked_call(cmd, sudo=True, logoutput=self.get_logoutput())
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in inner
    result = function(command, **kwargs)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in checked_call
    tries=tries, try_sleep=try_sleep)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in _call_wrapper
    result = _call(command, **kwargs_copy)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291, in _call
    raise Fail(err_msg)
resource_management.core.exceptions.Fail: Execution of '/usr/bin/yum -d 0 -e 0 -y install ambari-metrics-collector' returned 1. Error: Nothing to do
1 ACCEPTED SOLUTION

Accepted Solutions

Re: Ambari Metrics Won't Install

Just got off webex with @rgarcia@hortonworks.com

We were able to remove the failed Ranger/Metrics installation using something like the below:

su postgres
psql
\c ambari
DELETE FROM ambari.hostcomponentstate WHERE service_name IN ('RANGER');
DELETE FROM ambari.hostcomponentdesiredstate WHERE service_name IN ('RANGER');
DELETE FROM ambari.servicecomponentdesiredstate WHERE service_name IN ('RANGER');
DELETE FROM ambari.servicedesiredstate WHERE service_name IN ('RANGER');
DELETE FROM ambari.clusterservices WHERE service_name IN ('RANGER');

Then make sure to restart Ambari

service ambari-server restart

Now re-install Ranger/Ambair metrics

28 REPLIES 28

Re: Ambari Metrics Won't Install

Can you run the yum install manually and see what happens?

yum install ambari-metrics-collector

Re: Ambari Metrics Won't Install

Expert Contributor
[root@great-wall02 ~]# yum install ambari-metrics-collector

Loaded plugins: fastestmirror

Loading mirror speeds from cached hostfile

 * base: mirror.spro.net

 * extras: mirrors.sonic.net

 * updates: centos-distro.cavecreek.net

Setting up Install Process

No package ambari-metrics-collector available.

Error: Nothing to do

[root@great-wall02 ~]# rpm -q ambari-metrics-collector

package ambari-metrics-collector is not installed

Re: Ambari Metrics Won't Install

@rgarcia@hortonworks.com

Can you verify the repo version?

try yum repo list in all the nodes + if you are using repo for correct OS

Re: Ambari Metrics Won't Install

Expert Contributor

All nodes has the same repo version. I noticed that ambari.repo is gone. Possibly got deleted during host cleanup (python script)? Downloaded the ambari.repo and reinstalling ambari metrics. Stay tuned.

Re: Ambari Metrics Won't Install

Expert Contributor

Tried deleting Ambari Metrics from Ambari throug DELETE REST API and rerunning yum install ambari-metrics-collector. Same output.

[root@great-wall02 ~]# curl -u admin:admin -H "X-Requested-By: ambari" -X DELETE  http://great-wall01.cloud.hortonworks.com:8080/api/v1/clusters/smesecurity/services/AMBARI_METRICS

[root@great-wall02 ~]# yum install ambari-metrics-collector

Loaded plugins: fastestmirror

Loading mirror speeds from cached hostfile

 * base: mirror.spro.net

 * extras: mirrors.sonic.net

 * updates: mirror.hostduplex.com

Setting up Install Process

No package ambari-metrics-collector available.

Error: Nothing to do

Re: Ambari Metrics Won't Install

Expert Contributor

After removing Ambari Metrics, its got a side effect on Kafka. Kafka shouldn't be having problems after this process since it was running fine prior.

[2015-11-04 20:58:47,740] FATAL  (kafka.Kafka$)

java.lang.ClassNotFoundException: org.apache.hadoop.metrics2.sink.kafka.KafkaTimelineMetricsReporter

        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)

        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)

        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

        at java.lang.Class.forName0(Native Method)

        at java.lang.Class.forName(Class.java:264)

        at kafka.utils.CoreUtils$.createObject(CoreUtils.scala:231)

        at kafka.metrics.KafkaMetricsReporter$$anonfun$startReporters$1.apply(KafkaMetricsReporter.scala:59)

        at kafka.metrics.KafkaMetricsReporter$$anonfun$startReporters$1.apply(KafkaMetricsReporter.scala:58)

        at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)

        at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:34)

        at kafka.metrics.KafkaMetricsReporter$.startReporters(KafkaMetricsReporter.scala:58)

        at kafka.Kafka$.main(Kafka.scala:62)

        at kafka.Kafka.main(Kafka.scala)

[2015-11-04 21:00:51,935] FATAL  (kafka.Kafka$)

java.lang.ClassNotFoundException: org.apache.hadoop.metrics2.sink.kafka.KafkaTimelineMetricsReporter

        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)

        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)

        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

        at java.lang.Class.forName0(Native Method)

        at java.lang.Class.forName(Class.java:264)

        at kafka.utils.CoreUtils$.createObject(CoreUtils.scala:231)

        at kafka.metrics.KafkaMetricsReporter$$anonfun$startReporters$1.apply(KafkaMetricsReporter.scala:59)

        at kafka.metrics.KafkaMetricsReporter$$anonfun$startReporters$1.apply(KafkaMetricsReporter.scala:58)

        at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)

        at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:34)

        at kafka.metrics.KafkaMetricsReporter$.startReporters(KafkaMetricsReporter.scala:58)

        at kafka.Kafka$.main(Kafka.scala:62)

        at kafka.Kafka.main(Kafka.scala)



Re: Ambari Metrics Won't Install

Expert Contributor

Getting this amabri server error now. This came up after clicking deploy (for Ranger, Ranger KMS and Ambari Metrics)

Caused by: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "clusterservices_pkey"

	at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2161)

	at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1890)

	at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)

	at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:559)

	at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:417)

	at org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(AbstractJdbc2Statement.java:363)

	at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeDirectNoSelect(DatabaseAccessor.java:890)

	... 130 more

Re: Ambari Metrics Won't Install

Expert Contributor

This is what's coming up in Ambari.

396-untitled-5.png


untitled-5.png

Re: Ambari Metrics Won't Install

Don't have an account?
Coming from Hortonworks? Activate your account here