not able to start my metrics collector

Anuj — Fri, 25 Oct 2019 16:18:05 GMT

stderr: 
Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/AMBARI_METRICS/package/scripts/metrics_collector.py", line 90, in 
    AmsCollector().execute()
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/AMBARI_METRICS/package/scripts/metrics_collector.py", line 50, in start
    ams_service('collector', action = 'stop')
  File "/usr/lib/ambari-agent/lib/ambari_commons/os_family_impl.py", line 89, in thunk
    return fn(*args, **kwargs)
  File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/AMBARI_METRICS/package/scripts/ams_service.py", line 61, in ams_service
    hbase_service('regionserver', action=action)
  File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/AMBARI_METRICS/package/scripts/hbase_service.py", line 49, in hbase_service
    on_timeout = format("{no_op_test} && {sudo} -H -E kill -9 `{sudo} cat {pid_file}`")
  File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__
    self.env.run()
  File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 160, in run
    self.run_action(resource, action)
  File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 124, in run_action
    provider_action()
  File "/usr/lib/ambari-agent/lib/resource_management/core/providers/system.py", line 263, in action_run
    returns=self.resource.returns)
  File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 72, in inner
    result = function(command, **kwargs)
  File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 102, in checked_call
    tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy, returns=returns)
  File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 150, in _call_wrapper
    result = _call(command, **kwargs_copy)
  File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 314, in _call
    raise ExecutionFailed(err_msg, code, out, err)
resource_management.core.exceptions.ExecutionFailed: Execution of '/usr/lib/ams-hbase/bin/hbase-daemon.sh --config /etc/ams-hbase/conf stop regionserver' returned 127. -bash: /usr/lib/ams-hbase/bin/hbase-daemon.sh: No such file or directory
 stdout:
2019-10-25 07:59:02,481 - Stack Feature Version Info: Cluster Stack=3.1, Command Stack=None, Command Version=3.1.4.0-315 -> 3.1.4.0-315
2019-10-25 07:59:02,558 - Using hadoop conf dir: /usr/hdp/3.1.4.0-315/hadoop/conf
2019-10-25 07:59:03,312 - Stack Feature Version Info: Cluster Stack=3.1, Command Stack=None, Command Version=3.1.4.0-315 -> 3.1.4.0-315
2019-10-25 07:59:03,323 - Using hadoop conf dir: /usr/hdp/3.1.4.0-315/hadoop/conf
2019-10-25 07:59:03,326 - Group['kms'] {}
2019-10-25 07:59:03,328 - Group['ranger'] {}
2019-10-25 07:59:03,328 - Group['hdfs'] {}
2019-10-25 07:59:03,328 - Group['hadoop'] {}
2019-10-25 07:59:03,328 - Group['users'] {}
2019-10-25 07:59:03,329 - User['yarn-ats'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2019-10-25 07:59:03,331 - User['hive'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2019-10-25 07:59:03,333 - User['infra-solr'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2019-10-25 07:59:03,334 - User['zookeeper'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2019-10-25 07:59:03,336 - User['oozie'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop', 'users'], 'uid': None}
2019-10-25 07:59:03,338 - User['atlas'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2019-10-25 07:59:03,340 - User['ams'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2019-10-25 07:59:03,341 - User['ranger'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['ranger', 'hadoop'], 'uid': None}
2019-10-25 07:59:03,343 - User['tez'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop', 'users'], 'uid': None}
2019-10-25 07:59:03,344 - User['kms'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['kms', 'hadoop'], 'uid': None}
2019-10-25 07:59:03,346 - User['ambari-qa'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop', 'users'], 'uid': None}
2019-10-25 07:59:03,400 - User['kafka'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2019-10-25 07:59:03,402 - User['hdfs'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hdfs', 'hadoop'], 'uid': None}
2019-10-25 07:59:03,404 - User['sqoop'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2019-10-25 07:59:03,406 - User['yarn'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2019-10-25 07:59:03,407 - User['mapred'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2019-10-25 07:59:03,409 - User['hbase'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2019-10-25 07:59:03,410 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2019-10-25 07:59:03,437 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'}
2019-10-25 07:59:03,504 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] due to not_if
2019-10-25 07:59:03,505 - Directory['/tmp/hbase-hbase'] {'owner': 'hbase', 'create_parents': True, 'mode': 0775, 'cd_access': 'a'}
2019-10-25 07:59:03,507 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2019-10-25 07:59:03,509 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2019-10-25 07:59:03,510 - call['/var/lib/ambari-agent/tmp/changeUid.sh hbase'] {}
2019-10-25 07:59:03,572 - call returned (0, '1015')
2019-10-25 07:59:03,573 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase 1015'] {'not_if': '(test $(id -u hbase) -gt 1000) || (false)'}
2019-10-25 07:59:03,679 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase 1015'] due to not_if
2019-10-25 07:59:03,680 - Group['hdfs'] {}
2019-10-25 07:59:03,681 - User['hdfs'] {'fetch_nonlocal_groups': True, 'groups': ['hdfs', 'hadoop', u'hdfs']}
2019-10-25 07:59:03,682 - FS Type: HDFS
2019-10-25 07:59:03,682 - Directory['/etc/hadoop'] {'mode': 0755}
2019-10-25 07:59:03,714 - File['/usr/hdp/3.1.4.0-315/hadoop/conf/hadoop-env.sh'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop'}
2019-10-25 07:59:03,715 - Directory['/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 01777}
2019-10-25 07:59:03,751 - Execute[('setenforce', '0')] {'not_if': '(! which getenforce ) || (which getenforce && getenforce | grep -q Disabled)', 'sudo': True, 'only_if': 'test -f /selinux/enforce'}
2019-10-25 07:59:03,856 - Skipping Execute[('setenforce', '0')] due to not_if
2019-10-25 07:59:03,857 - Directory['/var/log/hadoop'] {'owner': 'root', 'create_parents': True, 'group': 'hadoop', 'mode': 0775, 'cd_access': 'a'}
2019-10-25 07:59:03,859 - Directory['/var/run/hadoop'] {'owner': 'root', 'create_parents': True, 'group': 'root', 'cd_access': 'a'}
2019-10-25 07:59:03,860 - Directory['/var/run/hadoop/hdfs'] {'owner': 'hdfs', 'cd_access': 'a'}
2019-10-25 07:59:03,861 - Directory['/tmp/hadoop-hdfs'] {'owner': 'hdfs', 'create_parents': True, 'cd_access': 'a'}
2019-10-25 07:59:03,867 - File['/usr/hdp/3.1.4.0-315/hadoop/conf/commons-logging.properties'] {'content': Template('commons-logging.properties.j2'), 'owner': 'hdfs'}
2019-10-25 07:59:03,870 - File['/usr/hdp/3.1.4.0-315/hadoop/conf/health_check'] {'content': Template('health_check.j2'), 'owner': 'hdfs'}
2019-10-25 07:59:03,882 - File['/usr/hdp/3.1.4.0-315/hadoop/conf/log4j.properties'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop', 'mode': 0644}
2019-10-25 07:59:03,933 - File['/usr/hdp/3.1.4.0-315/hadoop/conf/hadoop-metrics2.properties'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop'}
2019-10-25 07:59:03,934 - File['/usr/hdp/3.1.4.0-315/hadoop/conf/task-log4j.properties'] {'content': StaticFile('task-log4j.properties'), 'mode': 0755}
2019-10-25 07:59:03,935 - File['/usr/hdp/3.1.4.0-315/hadoop/conf/configuration.xsl'] {'owner': 'hdfs', 'group': 'hadoop'}
2019-10-25 07:59:03,966 - File['/etc/hadoop/conf/topology_mappings.data'] {'owner': 'hdfs', 'content': Template('topology_mappings.data.j2'), 'only_if': 'test -d /etc/hadoop/conf', 'group': 'hadoop', 'mode': 0644}
2019-10-25 07:59:04,007 - File['/etc/hadoop/conf/topology_script.py'] {'content': StaticFile('topology_script.py'), 'only_if': 'test -d /etc/hadoop/conf', 'mode': 0755}
2019-10-25 07:59:04,044 - Skipping unlimited key JCE policy check and setup since it is not required
2019-10-25 07:59:04,068 - Skipping stack-select on AMBARI_METRICS because it does not exist in the stack-select package structure.
2019-10-25 07:59:05,170 - Using hadoop conf dir: /usr/hdp/3.1.4.0-315/hadoop/conf
2019-10-25 07:59:05,193 - checked_call['hostid'] {}
2019-10-25 07:59:05,218 - checked_call returned (0, '800a1300')
2019-10-25 07:59:05,222 - Directory['/etc/ams-hbase/conf'] {'owner': 'ams', 'group': 'hadoop', 'create_parents': True, 'recursive_ownership': True}
2019-10-25 07:59:05,224 - Directory['/var/lib/ambari-metrics-collector/hbase-tmp'] {'owner': 'ams', 'create_parents': True, 'recursive_ownership': True, 'cd_access': 'a'}
2019-10-25 07:59:05,253 - Directory['/var/lib/ambari-metrics-collector/hbase-tmp/local/jars'] {'owner': 'ams', 'group': 'hadoop', 'create_parents': True, 'mode': 0775, 'cd_access': 'a'}
2019-10-25 07:59:05,254 - File['/etc/ams-hbase/conf/core-site.xml'] {'owner': 'ams', 'action': ['delete']}
2019-10-25 07:59:05,255 - File['/etc/ams-hbase/conf/hdfs-site.xml'] {'owner': 'ams', 'action': ['delete']}
2019-10-25 07:59:05,255 - XmlConfig['hbase-site.xml'] {'owner': 'ams', 'group': 'hadoop', 'conf_dir': '/etc/ams-hbase/conf', 'configuration_attributes': {u'final': {u'hbase.zookeeper.quorum': u'true'}}, 'configurations': ...}
2019-10-25 07:59:05,294 - Generating config: /etc/ams-hbase/conf/hbase-site.xml
2019-10-25 07:59:05,295 - File['/etc/ams-hbase/conf/hbase-site.xml'] {'owner': 'ams', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2019-10-25 07:59:05,368 - XmlConfig['hbase-policy.xml'] {'owner': 'ams', 'group': 'hadoop', 'conf_dir': '/etc/ams-hbase/conf', 'configuration_attributes': {}, 'configurations': {u'security.admin.protocol.acl': u'*', u'security.masterregion.protocol.acl': u'*', u'security.client.protocol.acl': u'*'}}
2019-10-25 07:59:05,389 - Generating config: /etc/ams-hbase/conf/hbase-policy.xml
2019-10-25 07:59:05,390 - File['/etc/ams-hbase/conf/hbase-policy.xml'] {'owner': 'ams', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2019-10-25 07:59:05,431 - File['/etc/ams-hbase/conf/hbase-env.sh'] {'content': InlineTemplate(...), 'owner': 'ams'}
2019-10-25 07:59:05,455 - File['/etc/ams-hbase/conf/hadoop-metrics2-hbase.properties'] {'content': Template('hadoop-metrics2-hbase.properties.j2'), 'owner': 'ams', 'group': 'hadoop'}
2019-10-25 07:59:05,457 - TemplateConfig['/etc/ams-hbase/conf/regionservers'] {'owner': 'ams', 'template_tag': None}
2019-10-25 07:59:05,460 - File['/etc/ams-hbase/conf/regionservers'] {'content': Template('regionservers.j2'), 'owner': 'ams', 'group': None, 'mode': None}
2019-10-25 07:59:05,461 - Directory['/var/run/ambari-metrics-collector/'] {'owner': 'ams', 'create_parents': True, 'mode': 0755, 'cd_access': 'a'}
2019-10-25 07:59:05,462 - Directory['/var/log/ambari-metrics-collector'] {'owner': 'ams', 'create_parents': True, 'mode': 0755, 'cd_access': 'a'}
2019-10-25 07:59:05,474 - Directory['/var/lib/ambari-metrics-collector/hbase'] {'owner': 'ams', 'create_parents': True, 'recursive_ownership': True, 'cd_access': 'a'}
2019-10-25 07:59:05,522 - File['/var/run/ambari-metrics-collector//distributed_mode'] {'owner': 'ams', 'action': ['delete']}
2019-10-25 07:59:05,527 - File['/etc/ams-hbase/conf/log4j.properties'] {'content': InlineTemplate(...), 'owner': 'ams', 'group': 'hadoop', 'mode': 0644}
2019-10-25 07:59:05,547 - Directory['/usr/lib/ambari-logsearch-logfeeder/conf'] {'create_parents': True, 'mode': 0755, 'cd_access': 'a'}
2019-10-25 07:59:05,547 - Generate Log Feeder config file: /usr/lib/ambari-logsearch-logfeeder/conf/input.config-ambari-metrics.json
2019-10-25 07:59:05,548 - File['/usr/lib/ambari-logsearch-logfeeder/conf/input.config-ambari-metrics.json'] {'content': Template('input.config-ambari-metrics.json.j2'), 'mode': 0644}
2019-10-25 07:59:05,549 - Directory['/etc/ams-hbase/conf'] {'owner': 'ams', 'group': 'hadoop', 'create_parents': True, 'recursive_ownership': True}
2019-10-25 07:59:05,561 - Directory['/var/lib/ambari-metrics-collector/hbase-tmp'] {'owner': 'ams', 'create_parents': True, 'recursive_ownership': True, 'cd_access': 'a'}
2019-10-25 07:59:05,572 - Directory['/var/lib/ambari-metrics-collector/hbase-tmp/local/jars'] {'owner': 'ams', 'group': 'hadoop', 'create_parents': True, 'mode': 0775, 'cd_access': 'a'}
2019-10-25 07:59:05,584 - File['/etc/ams-hbase/conf/core-site.xml'] {'owner': 'ams', 'action': ['delete']}
2019-10-25 07:59:05,585 - File['/etc/ams-hbase/conf/hdfs-site.xml'] {'owner': 'ams', 'action': ['delete']}
2019-10-25 07:59:05,585 - XmlConfig['hbase-site.xml'] {'owner': 'ams', 'group': 'hadoop', 'conf_dir': '/etc/ams-hbase/conf', 'configuration_attributes': {u'final': {u'hbase.zookeeper.quorum': u'true'}}, 'configurations': ...}
2019-10-25 07:59:05,610 - Generating config: /etc/ams-hbase/conf/hbase-site.xml
2019-10-25 07:59:05,611 - File['/etc/ams-hbase/conf/hbase-site.xml'] {'owner': 'ams', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2019-10-25 07:59:05,734 - XmlConfig['hbase-policy.xml'] {'owner': 'ams', 'group': 'hadoop', 'conf_dir': '/etc/ams-hbase/conf', 'configuration_attributes': {}, 'configurations': {u'security.admin.protocol.acl': u'*', u'security.masterregion.protocol.acl': u'*', u'security.client.protocol.acl': u'*'}}
2019-10-25 07:59:05,759 - Generating config: /etc/ams-hbase/conf/hbase-policy.xml
2019-10-25 07:59:05,759 - File['/etc/ams-hbase/conf/hbase-policy.xml'] {'owner': 'ams', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2019-10-25 07:59:05,787 - File['/etc/ams-hbase/conf/hbase-env.sh'] {'content': InlineTemplate(...), 'owner': 'ams'}
2019-10-25 07:59:05,796 - File['/etc/ams-hbase/conf/hadoop-metrics2-hbase.properties'] {'content': Template('hadoop-metrics2-hbase.properties.j2'), 'owner': 'ams', 'group': 'hadoop'}
2019-10-25 07:59:05,809 - TemplateConfig['/etc/ams-hbase/conf/regionservers'] {'owner': 'ams', 'template_tag': None}
2019-10-25 07:59:05,812 - File['/etc/ams-hbase/conf/regionservers'] {'content': Template('regionservers.j2'), 'owner': 'ams', 'group': None, 'mode': None}
2019-10-25 07:59:05,813 - Directory['/var/run/ambari-metrics-collector/'] {'owner': 'ams', 'create_parents': True, 'mode': 0755, 'cd_access': 'a'}
2019-10-25 07:59:05,814 - Directory['/var/log/ambari-metrics-collector'] {'owner': 'ams', 'create_parents': True, 'mode': 0755, 'cd_access': 'a'}
2019-10-25 07:59:05,819 - File['/etc/ams-hbase/conf/log4j.properties'] {'content': InlineTemplate(...), 'owner': 'ams', 'group': 'hadoop', 'mode': 0644}
2019-10-25 07:59:05,828 - Directory['/usr/lib/ambari-logsearch-logfeeder/conf'] {'create_parents': True, 'mode': 0755, 'cd_access': 'a'}
2019-10-25 07:59:05,829 - Generate Log Feeder config file: /usr/lib/ambari-logsearch-logfeeder/conf/input.config-ambari-metrics.json
2019-10-25 07:59:05,829 - File['/usr/lib/ambari-logsearch-logfeeder/conf/input.config-ambari-metrics.json'] {'content': Template('input.config-ambari-metrics.json.j2'), 'mode': 0644}
2019-10-25 07:59:05,830 - Directory['/etc/ambari-metrics-collector/conf'] {'owner': 'ams', 'group': 'hadoop', 'create_parents': True, 'recursive_ownership': True}
2019-10-25 07:59:05,831 - Directory['/var/lib/ambari-metrics-collector/checkpoint'] {'owner': 'ams', 'group': 'hadoop', 'create_parents': True, 'recursive_ownership': True, 'cd_access': 'a'}
2019-10-25 07:59:05,832 - XmlConfig['ams-site.xml'] {'owner': 'ams', 'group': 'hadoop', 'conf_dir': '/etc/ambari-metrics-collector/conf', 'configuration_attributes': {}, 'configurations': ...}
2019-10-25 07:59:05,847 - Generating config: /etc/ambari-metrics-collector/conf/ams-site.xml
2019-10-25 07:59:05,847 - File['/etc/ambari-metrics-collector/conf/ams-site.xml'] {'owner': 'ams', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2019-10-25 07:59:06,009 - XmlConfig['ssl-server.xml'] {'owner': 'ams', 'group': 'hadoop', 'conf_dir': '/etc/ambari-metrics-collector/conf', 'configuration_attributes': {}, 'configurations': ...}
2019-10-25 07:59:06,028 - Generating config: /etc/ambari-metrics-collector/conf/ssl-server.xml
2019-10-25 07:59:06,028 - File['/etc/ambari-metrics-collector/conf/ssl-server.xml'] {'owner': 'ams', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2019-10-25 07:59:06,047 - XmlConfig['hbase-site.xml'] {'owner': 'ams', 'group': 'hadoop', 'conf_dir': '/etc/ambari-metrics-collector/conf', 'configuration_attributes': {u'final': {u'hbase.zookeeper.quorum': u'true'}}, 'configurations': ...}
2019-10-25 07:59:06,084 - Generating config: /etc/ambari-metrics-collector/conf/hbase-site.xml
2019-10-25 07:59:06,084 - File['/etc/ambari-metrics-collector/conf/hbase-site.xml'] {'owner': 'ams', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2019-10-25 07:59:06,212 - File['/etc/ambari-metrics-collector/conf/log4j.properties'] {'content': InlineTemplate(...), 'owner': 'ams', 'group': 'hadoop', 'mode': 0644}
2019-10-25 07:59:06,221 - File['/etc/ambari-metrics-collector/conf/ams-env.sh'] {'content': InlineTemplate(...), 'owner': 'ams'}
2019-10-25 07:59:06,234 - Directory['/var/log/ambari-metrics-collector'] {'owner': 'ams', 'group': 'hadoop', 'create_parents': True, 'mode': 0755, 'cd_access': 'a'}
2019-10-25 07:59:06,235 - Directory['/var/run/ambari-metrics-collector'] {'owner': 'ams', 'group': 'hadoop', 'create_parents': True, 'mode': 0755, 'cd_access': 'a'}
2019-10-25 07:59:06,236 - File['/usr/lib/ams-hbase/bin/hadoop'] {'owner': 'ams', 'mode': 0755}
2019-10-25 07:59:06,237 - Directory['/etc/security/limits.d'] {'owner': 'root', 'create_parents': True, 'group': 'root'}
2019-10-25 07:59:06,240 - File['/etc/security/limits.d/ams.conf'] {'content': Template('ams.conf.j2'), 'owner': 'root', 'group': 'root', 'mode': 0644}
2019-10-25 07:59:06,256 - Directory['/usr/lib/ambari-logsearch-logfeeder/conf'] {'create_parents': True, 'mode': 0755, 'cd_access': 'a'}
2019-10-25 07:59:06,257 - Generate Log Feeder config file: /usr/lib/ambari-logsearch-logfeeder/conf/input.config-ambari-metrics.json
2019-10-25 07:59:06,257 - File['/usr/lib/ambari-logsearch-logfeeder/conf/input.config-ambari-metrics.json'] {'content': Template('input.config-ambari-metrics.json.j2'), 'mode': 0644}
2019-10-25 07:59:06,261 - Execute['/usr/lib/ams-hbase/bin/hbase-daemon.sh --config /etc/ams-hbase/conf stop regionserver'] {'on_timeout': 'ls /var/run/ambari-metrics-collector//hbase-ams-regionserver.pid >/dev/null 2>&1 && ps `cat /var/run/ambari-metrics-collector//hbase-ams-regionserver.pid` >/dev/null 2>&1 && ambari-sudo.sh -H -E kill -9 `ambari-sudo.sh cat /var/run/ambari-metrics-collector//hbase-ams-regionserver.pid`', 'timeout': 30, 'user': 'ams'}
2019-10-25 07:59:06,600 - Skipping stack-select on AMBARI_METRICS because it does not exist in the stack-select package structure.

Command failed after 1 tries

Re: not able to start my metrics collector

Shelton — Fri, 25 Oct 2019 09:57:04 GMT

@Anuj

Is this the first time you are starting the service? If not what happened in between was there a change in your configuration?

Please revert

Re: not able to start my metrics collector

Anuj — Fri, 25 Oct 2019 14:14:06 GMT

Actually i have decommission the node then i have deleted that host then
after ward i was facing this issue.

Re: not able to start my metrics collector

Shelton — Fri, 25 Oct 2019 21:03:44 GMT

@Anuj

Here is the official steps from the Ambari.org

read through and follow the steps look at my steps for checking the zookeeper entries

Step-by-step guide

Using Ambari
1. Set AMS to maintenance
2. Stop AMS from Ambari
3. Identify the following from the AMS Configs screen
  1. 'Metrics Service operation mode' (embedded or distributed)
  2. hbase.rootdir
  3. hbase.zookeeper.property.dataDir
AMS data would be stored in 'hbase.rootdir' identified above. Backup and remove the AMS data.
1. If the Metrics Service operation mode
  1. is 'embedded', then the data is stored in OS files. Use regular OS commands to backup and remove the files in hbase.rootdir
  2. is 'distributed', then the data is stored in HDFS. Use 'hdfs dfs' commands to backup and remove the files in hbase.rootdir
Remove the AMS zookeeper data by backing up and removing the contents of 'hbase.tmp.dir'/zookeeper
Remove any Phoenix spool files from 'hbase.tmp.dir'/phoenix-spool folder
Restart AMS using Ambari

I take the above a step further by locating the zookeeper executable usually in /usr/hdp/{hdp_version}/zookeeper/bin/

Log into zookeeper

[zookeeper@osaka bin]$ ./zkCli.sh

List the root leaf structure you should see ambari-metrics-cluster should look like below
[zk: localhost:2181(CONNECTED) 0] ls /
[cluster, registry, controller, brokers, storm, zookeeper, infra-solr, hbase-unsecure, admin, isr_change_notification, log_dir_event_notificat ion, controller_epoch, hiveserver2, hiveserver2-leader, rmstore, atsv2-hbase-unsecure, consumers, ambari-metrics-cluster, latest_producer_id_b lock, config]

Now check the entries under ambari-metrics-cluster, you should find something like below

ls /ambari-metrics-cluster/INSTANCES/

FQDN_12001

Delete the entry that corresponds to your cluster

[zk: localhost:2181(CONNECTED) 25] rmr /ambari-metrics-cluster/INSTANCES/FQDN_12001

Restart the AM this should recreate a new entry in zookeeper

question Re: not able to start my metrics collector in Support Questions

not able to start my metrics collector

Re: not able to start my metrics collector

Re: not able to start my metrics collector

Re: not able to start my metrics collector

Step-by-step guide