Support Questions

rgarcia · ‎11-05-2015

On a 6 node cluster, using Ambari 2.1.2/HDP 2.3.2.

Scenario 1:

When installing HDP and it goes to a point where it's installing the services across all nodes, it suddenly fails and it's due to Ambari Metrics/Monitors failing. Full stack trace below from Ambari UI.

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts/metrics_collector.py", line 131, in <module>
    AmsCollector().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 219, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts/metrics_collector.py", line 34, in install
    self.install_packages(env)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 395, in install_packages
    Package(name)
  File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 154, in __init__
    self.env.run()
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 152, in run
    self.run_action(resource, action)
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 118, in run_action
    provider_action()
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/__init__.py", line 45, in action_install
    self.install_package(package_name, self.resource.use_repos, self.resource.skip_repos)
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/yumrpm.py", line 49, in install_package
    shell.checked_call(cmd, sudo=True, logoutput=self.get_logoutput())
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in inner
    result = function(command, **kwargs)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in checked_call
    tries=tries, try_sleep=try_sleep)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in _call_wrapper
    result = _call(command, **kwargs_copy)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291, in _call
    raise Fail(err_msg)
resource_management.core.exceptions.Fail: Execution of '/usr/bin/yum -d 0 -e 0 -y install ambari-metrics-collector' returned 1. Error: Nothing to do

Running the same command from the last line of the error yields the same response. There were no Ambari Metrics log generated. Ambari server log didn't any info.

Scenario 2:

I reset ambari-server and cleanup all hosts. Re-run ambari wizard and install all services except for Ambari Metrics. HDP installed successfully. I now added Ambari Metrics back and I'm getting the same error.

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts/metrics_collector.py", line 131, in <module>
    AmsCollector().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 219, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts/metrics_collector.py", line 34, in install
    self.install_packages(env)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 395, in install_packages
    Package(name)
  File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 154, in __init__
    self.env.run()
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 152, in run
    self.run_action(resource, action)
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 118, in run_action
    provider_action()
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/__init__.py", line 45, in action_install
    self.install_package(package_name, self.resource.use_repos, self.resource.skip_repos)
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/yumrpm.py", line 49, in install_package
    shell.checked_call(cmd, sudo=True, logoutput=self.get_logoutput())
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in inner
    result = function(command, **kwargs)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in checked_call
    tries=tries, try_sleep=try_sleep)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in _call_wrapper
    result = _call(command, **kwargs_copy)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291, in _call
    raise Fail(err_msg)
resource_management.core.exceptions.Fail: Execution of '/usr/bin/yum -d 0 -e 0 -y install ambari-metrics-collector' returned 1. Error: Nothing to do

abajwa · ‎11-06-2015

Just got off webex with @rgarcia@hortonworks.com

We were able to remove the failed Ranger/Metrics installation using something like the below:

su postgres
psql
\c ambari
DELETE FROM ambari.hostcomponentstate WHERE service_name IN ('RANGER');
DELETE FROM ambari.hostcomponentdesiredstate WHERE service_name IN ('RANGER');
DELETE FROM ambari.servicecomponentdesiredstate WHERE service_name IN ('RANGER');
DELETE FROM ambari.servicedesiredstate WHERE service_name IN ('RANGER');
DELETE FROM ambari.clusterservices WHERE service_name IN ('RANGER');

Then make sure to restart Ambari

service ambari-server restart

Now re-install Ranger/Ambair metrics

View solution in original post

abajwa · ‎11-05-2015

Can you run the yum install manually and see what happens?

yum install ambari-metrics-collector

rgarcia · ‎11-05-2015

[root@great-wall02 ~]# yum install ambari-metrics-collector

Loaded plugins: fastestmirror

Loading mirror speeds from cached hostfile

 * base: mirror.spro.net

 * extras: mirrors.sonic.net

 * updates: centos-distro.cavecreek.net

Setting up Install Process

No package ambari-metrics-collector available.

Error: Nothing to do

[root@great-wall02 ~]# rpm -q ambari-metrics-collector

package ambari-metrics-collector is not installed

nsabharwal · ‎11-05-2015

@rgarcia@hortonworks.com

Can you verify the repo version?

try yum repo list in all the nodes + if you are using repo for correct OS

rgarcia · ‎11-05-2015

All nodes has the same repo version. I noticed that ambari.repo is gone. Possibly got deleted during host cleanup (python script)? Downloaded the ambari.repo and reinstalling ambari metrics. Stay tuned.

rgarcia · ‎11-05-2015

Tried deleting Ambari Metrics from Ambari throug DELETE REST API and rerunning yum install ambari-metrics-collector. Same output.

[root@great-wall02 ~]# curl -u admin:admin -H "X-Requested-By: ambari" -X DELETE  http://great-wall01.cloud.hortonworks.com:8080/api/v1/clusters/smesecurity/services/AMBARI_METRICS

[root@great-wall02 ~]# yum install ambari-metrics-collector

Loaded plugins: fastestmirror

Loading mirror speeds from cached hostfile

 * base: mirror.spro.net

 * extras: mirrors.sonic.net

 * updates: mirror.hostduplex.com

Setting up Install Process

No package ambari-metrics-collector available.

Error: Nothing to do

rgarcia · ‎11-05-2015

After removing Ambari Metrics, its got a side effect on Kafka. Kafka shouldn't be having problems after this process since it was running fine prior.

[2015-11-04 20:58:47,740] FATAL  (kafka.Kafka$)

java.lang.ClassNotFoundException: org.apache.hadoop.metrics2.sink.kafka.KafkaTimelineMetricsReporter

        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)

        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)

        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

        at java.lang.Class.forName0(Native Method)

        at java.lang.Class.forName(Class.java:264)

        at kafka.utils.CoreUtils$.createObject(CoreUtils.scala:231)

        at kafka.metrics.KafkaMetricsReporter$$anonfun$startReporters$1.apply(KafkaMetricsReporter.scala:59)

        at kafka.metrics.KafkaMetricsReporter$$anonfun$startReporters$1.apply(KafkaMetricsReporter.scala:58)

        at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)

        at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:34)

        at kafka.metrics.KafkaMetricsReporter$.startReporters(KafkaMetricsReporter.scala:58)

        at kafka.Kafka$.main(Kafka.scala:62)

        at kafka.Kafka.main(Kafka.scala)

[2015-11-04 21:00:51,935] FATAL  (kafka.Kafka$)

java.lang.ClassNotFoundException: org.apache.hadoop.metrics2.sink.kafka.KafkaTimelineMetricsReporter

        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)

        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)

        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

        at java.lang.Class.forName0(Native Method)

        at java.lang.Class.forName(Class.java:264)

        at kafka.utils.CoreUtils$.createObject(CoreUtils.scala:231)

        at kafka.metrics.KafkaMetricsReporter$$anonfun$startReporters$1.apply(KafkaMetricsReporter.scala:59)

        at kafka.metrics.KafkaMetricsReporter$$anonfun$startReporters$1.apply(KafkaMetricsReporter.scala:58)

        at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)

        at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:34)

        at kafka.metrics.KafkaMetricsReporter$.startReporters(KafkaMetricsReporter.scala:58)

        at kafka.Kafka$.main(Kafka.scala:62)

        at kafka.Kafka.main(Kafka.scala)

rgarcia · ‎11-05-2015

Getting this amabri server error now. This came up after clicking deploy (for Ranger, Ranger KMS and Ambari Metrics)

Caused by: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "clusterservices_pkey"

	at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2161)

	at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1890)

	at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)

	at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:559)

	at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:417)

	at org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(AbstractJdbc2Statement.java:363)

	at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeDirectNoSelect(DatabaseAccessor.java:890)

	... 130 more

rgarcia · ‎11-05-2015

This is what's coming up in Ambari.

nsabharwal · ‎11-05-2015

@jeff@hortonworks.com

@mahadev@hortonworks.com

Please see this

Cloudera Community

Support Questions

Ambari Metrics Won't Install