question Re: Ambari Metrics Won't Install in Support Questions

Ambari Metrics Won't Install

rgarcia — Thu, 05 Nov 2015 12:47:10 GMT

On a 6 node cluster, using Ambari 2.1.2/HDP 2.3.2.

Scenario 1:

When installing HDP and it goes to a point where it's installing the services across all nodes, it suddenly fails and it's due to Ambari Metrics/Monitors failing. Full stack trace below from Ambari UI.

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts/metrics_collector.py", line 131, in <module>
    AmsCollector().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 219, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts/metrics_collector.py", line 34, in install
    self.install_packages(env)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 395, in install_packages
    Package(name)
  File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 154, in __init__
    self.env.run()
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 152, in run
    self.run_action(resource, action)
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 118, in run_action
    provider_action()
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/__init__.py", line 45, in action_install
    self.install_package(package_name, self.resource.use_repos, self.resource.skip_repos)
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/yumrpm.py", line 49, in install_package
    shell.checked_call(cmd, sudo=True, logoutput=self.get_logoutput())
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in inner
    result = function(command, **kwargs)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in checked_call
    tries=tries, try_sleep=try_sleep)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in _call_wrapper
    result = _call(command, **kwargs_copy)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291, in _call
    raise Fail(err_msg)
resource_management.core.exceptions.Fail: Execution of '/usr/bin/yum -d 0 -e 0 -y install ambari-metrics-collector' returned 1. Error: Nothing to do

Running the same command from the last line of the error yields the same response. There were no Ambari Metrics log generated. Ambari server log didn't any info.

Scenario 2:

I reset ambari-server and cleanup all hosts. Re-run ambari wizard and install all services except for Ambari Metrics. HDP installed successfully. I now added Ambari Metrics back and I'm getting the same error.

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts/metrics_collector.py", line 131, in <module>
    AmsCollector().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 219, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts/metrics_collector.py", line 34, in install
    self.install_packages(env)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 395, in install_packages
    Package(name)
  File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 154, in __init__
    self.env.run()
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 152, in run
    self.run_action(resource, action)
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 118, in run_action
    provider_action()
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/__init__.py", line 45, in action_install
    self.install_package(package_name, self.resource.use_repos, self.resource.skip_repos)
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/yumrpm.py", line 49, in install_package
    shell.checked_call(cmd, sudo=True, logoutput=self.get_logoutput())
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in inner
    result = function(command, **kwargs)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in checked_call
    tries=tries, try_sleep=try_sleep)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in _call_wrapper
    result = _call(command, **kwargs_copy)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291, in _call
    raise Fail(err_msg)
resource_management.core.exceptions.Fail: Execution of '/usr/bin/yum -d 0 -e 0 -y install ambari-metrics-collector' returned 1. Error: Nothing to do

Re: Ambari Metrics Won't Install

abajwa — Thu, 05 Nov 2015 12:56:03 GMT

Can you run the yum install manually and see what happens?

yum install ambari-metrics-collector

Re: Ambari Metrics Won't Install

rgarcia — Thu, 05 Nov 2015 12:59:34 GMT

[root@great-wall02 ~]# yum install ambari-metrics-collector

Loaded plugins: fastestmirror

Loading mirror speeds from cached hostfile

 * base: mirror.spro.net

 * extras: mirrors.sonic.net

 * updates: centos-distro.cavecreek.net

Setting up Install Process

No package ambari-metrics-collector available.

Error: Nothing to do

[root@great-wall02 ~]# rpm -q ambari-metrics-collector

package ambari-metrics-collector is not installed

Re: Ambari Metrics Won't Install

rgarcia — Thu, 05 Nov 2015 13:02:47 GMT

Tried deleting Ambari Metrics from Ambari throug DELETE REST API and rerunning yum install ambari-metrics-collector. Same output.

[root@great-wall02 ~]# curl -u admin:admin -H "X-Requested-By: ambari" -X DELETE  http://great-wall01.cloud.hortonworks.com:8080/api/v1/clusters/smesecurity/services/AMBARI_METRICS

[root@great-wall02 ~]# yum install ambari-metrics-collector

Loaded plugins: fastestmirror

Loading mirror speeds from cached hostfile

 * base: mirror.spro.net

 * extras: mirrors.sonic.net

 * updates: mirror.hostduplex.com

Setting up Install Process

No package ambari-metrics-collector available.

Error: Nothing to do

Re: Ambari Metrics Won't Install

nsabharwal — Thu, 05 Nov 2015 13:04:20 GMT

@rgarcia@hortonworks.com

Can you verify the repo version?

try yum repo list in all the nodes + if you are using repo for correct OS

Re: Ambari Metrics Won't Install

rgarcia — Thu, 05 Nov 2015 13:06:50 GMT

After removing Ambari Metrics, its got a side effect on Kafka. Kafka shouldn't be having problems after this process since it was running fine prior.

[2015-11-04 20:58:47,740] FATAL  (kafka.Kafka$)

java.lang.ClassNotFoundException: org.apache.hadoop.metrics2.sink.kafka.KafkaTimelineMetricsReporter

        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)

        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)

        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

        at java.lang.Class.forName0(Native Method)

        at java.lang.Class.forName(Class.java:264)

        at kafka.utils.CoreUtils$.createObject(CoreUtils.scala:231)

        at kafka.metrics.KafkaMetricsReporter$$anonfun$startReporters$1.apply(KafkaMetricsReporter.scala:59)

        at kafka.metrics.KafkaMetricsReporter$$anonfun$startReporters$1.apply(KafkaMetricsReporter.scala:58)

        at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)

        at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:34)

        at kafka.metrics.KafkaMetricsReporter$.startReporters(KafkaMetricsReporter.scala:58)

        at kafka.Kafka$.main(Kafka.scala:62)

        at kafka.Kafka.main(Kafka.scala)

[2015-11-04 21:00:51,935] FATAL  (kafka.Kafka$)

java.lang.ClassNotFoundException: org.apache.hadoop.metrics2.sink.kafka.KafkaTimelineMetricsReporter

        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)

        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)

        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

        at java.lang.Class.forName0(Native Method)

        at java.lang.Class.forName(Class.java:264)

        at kafka.utils.CoreUtils$.createObject(CoreUtils.scala:231)

        at kafka.metrics.KafkaMetricsReporter$$anonfun$startReporters$1.apply(KafkaMetricsReporter.scala:59)

        at kafka.metrics.KafkaMetricsReporter$$anonfun$startReporters$1.apply(KafkaMetricsReporter.scala:58)

        at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)

        at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:34)

        at kafka.metrics.KafkaMetricsReporter$.startReporters(KafkaMetricsReporter.scala:58)

        at kafka.Kafka$.main(Kafka.scala:62)

        at kafka.Kafka.main(Kafka.scala)

Re: Ambari Metrics Won't Install

rgarcia — Thu, 05 Nov 2015 13:20:20 GMT

All nodes has the same repo version. I noticed that ambari.repo is gone. Possibly got deleted during host cleanup (python script)? Downloaded the ambari.repo and reinstalling ambari metrics. Stay tuned.

Re: Ambari Metrics Won't Install

rgarcia — Thu, 05 Nov 2015 13:30:22 GMT

Getting this amabri server error now. This came up after clicking deploy (for Ranger, Ranger KMS and Ambari Metrics)

Caused by: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "clusterservices_pkey"

	at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2161)

	at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1890)

	at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)

	at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:559)

	at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:417)

	at org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(AbstractJdbc2Statement.java:363)

	at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeDirectNoSelect(DatabaseAccessor.java:890)

	... 130 more

Re: Ambari Metrics Won't Install

rgarcia — Mon, 19 Aug 2019 12:52:12 GMT

This is what's coming up in Ambari.

Re: Ambari Metrics Won't Install

nsabharwal — Thu, 05 Nov 2015 20:06:40 GMT

@jeff@hortonworks.com

@mahadev@hortonworks.com

Please see this

Re: Ambari Metrics Won't Install

nsabharwal — Thu, 05 Nov 2015 20:23:26 GMT

@rgarcia@hortonworks.com Please see this

Also, I have added Jeff & Mahadev in this loop.

Re: Ambari Metrics Won't Install

rgarcia — Fri, 06 Nov 2015 02:57:49 GMT

Followed the recommended AH link from @Neeraj. See details below.

Logged in to postgresql:

ambari=> select * from hostcomponentstate where component_name LIKE '%RANGER%'; id  | cluster_id |  component_name   |   version  | current_stack_id | current_state  | host_id | service_name | upgrade_state | security_state-----+------------+-------------------+--------------+------------------+----------------+---------+--------------+---------------+---------------- 163 |  2 | RANGER_KMS_SERVER | 2.3.2.0-2950 |  4 | INSTALLED  |   5 | RANGER_KMS   | NONE  | UNKNOWN 164 |  2 | RANGER_USERSYNC   | UNKNOWN  |  4 | INSTALL_FAILED |   4 | RANGER   | NONE  | UNKNOWN 165 |  2 | RANGER_ADMIN  | UNKNOWN  |  4 | INSTALL_FAILED |   4 | RANGER   | NONE  | UNKNOWN(3 rows)ambari=> select * from hostcomponentdesiredstate where component_name LIKE '%RANGER%'; cluster_id | component_name | desired_stack_id | desired_state | host_id | service_name | admin_state | maintenance_state | security_state | restart_required------------+----------------+------------------+---------------+---------+--------------+-------------+-------------------+----------------+------------------(0 rows)ambari=> select * from servicecomponentdesiredstate where component_name LIKE '%RANGER%';  component_name   | cluster_id | desired_stack_id | desired_state | service_name-------------------+------------+------------------+---------------+-------------- RANGER_KMS_SERVER |  2 |  4 | INSTALLED   | RANGER_KMS RANGER_ADMIN  |  2 |  4 | INSTALLED   | RANGER RANGER_USERSYNC   |  2 |  4 | INSTALLED   | RANGER
(3 rows)

Delete in tables:

ambari=> delete from hostcomponentstate where component_name LIKE '%RANGER%';DELETE 3ambari=> delete from servicecomponentdesiredstate where component_name LIKE '%RANGER%';
DELETE 3

Then delete Services:

[root@great-wall02 ~]# curl -u admin:admin -H "X-Requested-By: ambari" -X DELETE  http://great-wall01.cloud.hortonworks.com:8080/api/v1/clusters/smesecurity/services/RANGER
[root@great-wall02 ~]# curl -u admin:admin -H "X-Requested-By: ambari" -X DELETE  http://great-wall01.cloud.hortonworks.com:8080/api/v1/clusters/smesecurity/services/RANGER_KMS

Deleted the databases for ranger: 'ranger', 'ranger_kms', 'ranger_audit'.

Tried reinstalling Ranger only but now getting the error below. Looks like there are some metadata in Ambari DB still that needs to be cleaned up. What ambari tables should I clean up?

05 Nov 2015 10:51:25,540 ERROR [qtp-client-26] ClusterImpl:2016 - Config inconsistency exists: unknown configType=ams-hbase-security-site
05 Nov 2015 10:51:25,541 ERROR [qtp-client-26] ClusterImpl:2016 - Config inconsistency exists: unknown configType=ams-hbase-site
05 Nov 2015 10:51:25,541 ERROR [qtp-client-26] ClusterImpl:2016 - Config inconsistency exists: unknown configType=ams-log4j
05 Nov 2015 10:51:25,542 ERROR [qtp-client-26] ClusterImpl:2016 - Config inconsistency exists: unknown configType=ams-site
05 Nov 2015 10:51:25,542 ERROR [qtp-client-26] ClusterImpl:2016 - Config inconsistency exists: unknown configType=ams-hbase-policy
05 Nov 2015 10:51:25,543 ERROR [qtp-client-26] ClusterImpl:2016 - Config inconsistency exists: unknown configType=ams-hbase-log4j
05 Nov 2015 10:51:25,543 ERROR [qtp-client-26] ClusterImpl:2016 - Config inconsistency exists: unknown configType=ams-env
05 Nov 2015 10:51:25,544 ERROR [qtp-client-26] ClusterImpl:2016 - Config inconsistency exists: unknown configType=ams-hbase-env
05 Nov 2015 10:51:25,544 ERROR [qtp-client-26] ClusterImpl:2016 - Config inconsistency exists: unknown configType=kms-properties
05 Nov 2015 10:51:25,545 ERROR [qtp-client-26] ClusterImpl:2016 - Config inconsistency exists: unknown configType=ranger-kms-policymgr-ssl
05 Nov 2015 10:51:25,545 ERROR [qtp-client-26] ClusterImpl:2016 - Config inconsistency exists: unknown configType=kms-log4j
05 Nov 2015 10:51:25,545 ERROR [qtp-client-26] ClusterImpl:2016 - Config inconsistency exists: unknown configType=ranger-kms-security
05 Nov 2015 10:51:25,546 ERROR [qtp-client-26] ClusterImpl:2016 - Config inconsistency exists: unknown configType=ranger-kms-audit
05 Nov 2015 10:51:25,546 ERROR [qtp-client-26] ClusterImpl:2016 - Config inconsistency exists: unknown configType=dbks-site
05 Nov 2015 10:51:25,546 ERROR [qtp-client-26] ClusterImpl:2016 - Config inconsistency exists: unknown configType=kms-env
05 Nov 2015 10:51:25,547 ERROR [qtp-client-26] ClusterImpl:2016 - Config inconsistency exists: unknown configType=kms-site
05 Nov 2015 10:51:25,547 ERROR [qtp-client-26] ClusterImpl:2016 - Config inconsistency exists: unknown configType=ranger-kms-site

Re: Ambari Metrics Won't Install

nsabharwal — Fri, 06 Nov 2015 03:00:46 GMT

@sneethiraj@hortonworks.com @vperiasamy@hortonworks.com

Re: Ambari Metrics Won't Install

aervits — Fri, 06 Nov 2015 03:50:36 GMT

I recommend you select from each existing table in ambari, usually there are stale alerts, host configs also exist, you need to purge those as well. Don't just delete hostcomponentstate, hostcomponentdesiredstate, etc, go through all tables in ambari database and look for specific service. Sometimes you're lucky and only need to touch those desiredstate and componentstate tables but sometimes you need to clean up alerts, hostconfigs, etc.

Re: Ambari Metrics Won't Install

abajwa — Fri, 06 Nov 2015 10:56:50 GMT

Just got off webex with @rgarcia@hortonworks.com

We were able to remove the failed Ranger/Metrics installation using something like the below:

su postgres
psql
\c ambari
DELETE FROM ambari.hostcomponentstate WHERE service_name IN ('RANGER');
DELETE FROM ambari.hostcomponentdesiredstate WHERE service_name IN ('RANGER');
DELETE FROM ambari.servicecomponentdesiredstate WHERE service_name IN ('RANGER');
DELETE FROM ambari.servicedesiredstate WHERE service_name IN ('RANGER');
DELETE FROM ambari.clusterservices WHERE service_name IN ('RANGER');

Then make sure to restart Ambari

service ambari-server restart

Now re-install Ranger/Ambair metrics

Re: Ambari Metrics Won't Install

nsabharwal — Fri, 06 Nov 2015 19:33:13 GMT

@jeff@hortonworks.com @mahadev@hortonworks.com @bganesan@hortonworks.com @bdurai@hortonworks.com @sneethiraj@hortonworks.com

Please see this thread.

Re: Ambari Metrics Won't Install

rxu — Wed, 10 Feb 2016 00:05:19 GMT

What is the solution to this problem? I also encounter it, able to delete and re-install, but fail with same error.

Re: Ambari Metrics Won't Install

nsabharwal — Wed, 10 Feb 2016 00:58:35 GMT

@rxu There are couple of things to check..make sure repo files are correct.

Whats the error?

Re: Ambari Metrics Won't Install

rxu — Wed, 10 Feb 2016 01:09:03 GMT

Same error. Repo checked, all looks good.

Re: Ambari Metrics Won't Install

Shelton — Wed, 10 Feb 2016 02:00:55 GMT

@rgarcia why can't you just all together

#drop the database and then retry if it fails

Then just yum remove postgresql* and reinstall yum install postgresql*

That could be much faster though I know the urge to know why is unavoidable