Support Questions
Find answers, ask questions, and share your expertise

Point Ambari Metrics Collector from HDF to HBase in HDP

I have two clusters, one HDP with HBase, and one HDF. Both have Ambari Metrics installed, but are both using the embedded Hbase instance. I was hoping to change both collectors to distributed, using the HDP HBase instance.

I've made the following changes on both instances:

	timeline.metrics.service.operation.mode=distributed
	hbase.cluster.distributed=true
	hbase.rootdir=hdfs://hdp-master.local:8020/user/ams/hbase

In HDF I can't start the service (at this step):

2016-12-12 12:53:11,055 - Generating config: /etc/ambari-metrics-collector/conf/hdfs-site.xml
2016-12-12 12:53:11,055 - File['/etc/ambari-metrics-collector/conf/hdfs-site.xml'] {'owner': 'ams', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, 'encoding': 'UTF-8'}

(I get this error):

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts/metrics_collector.py", line 148, in <module>
    AmsCollector().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 280, in execute
    method(env)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 690, in restart
    self.stop(env)
  File "/var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts/metrics_collector.py", line 55, in stop
    self.configure(env, action = 'stop') # for security
  File "/var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts/metrics_collector.py", line 43, in configure
    ams(name='collector')
  File "/usr/lib/python2.6/site-packages/ambari_commons/os_family_impl.py", line 89, in thunk
    return fn(*args, **kwargs)
  File "/var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts/ams.py", line 306, in ams
    mode=0644
  File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, in __init__
    self.env.run()
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run
    self.run_action(resource, action)
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action
    provider_action()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/xml_config.py", line 66, in action_create
    encoding = self.resource.encoding
  File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, in __init__
    self.env.run()
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run
    self.run_action(resource, action)
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action
    provider_action()
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 123, in action_create
    content = self._get_content()
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 160, in _get_content
    return content()
  File "/usr/lib/python2.6/site-packages/resource_management/core/source.py", line 51, in __call__
    return self.get_content()
  File "/usr/lib/python2.6/site-packages/resource_management/core/source.py", line 142, in get_content
    rendered = self.template.render(self.context)
  File "/usr/lib/python2.6/site-packages/ambari_jinja2/environment.py", line 891, in render
    return self.environment.handle_exception(exc_info, True)
  File "<template>", line 2, in top-level template code
  File "/usr/lib/python2.6/site-packages/ambari_jinja2/filters.py", line 176, in do_dictsort
    return sorted(value.items(), key=sort_func)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/config_dictionary.py", line 73, in __getattr__
    raise Fail("Configuration parameter '" + self.name + "' was not found in configurations dictionary!")
resource_management.core.exceptions.Fail: Configuration parameter 'hdfs-site' was not found in configurations dictionary!

Any ideas?

4 REPLIES 4

Super Guru
@Daniel Skates

did you run the following steps since you initially had Embedded mode?

Steps to migrate existing metric data to HDFS and starting AMS in distributed mode:

  • Stop AMS Metric Collector
  • Create hdfs directory for ams user, Example:
    su - hdfs -c 'hdfs dfs -mkdir /user/ams'
    su - hdfs -c 'hdfs dfs -chown ams:hadoop /user/ams'
  • Copy the metric data from the AMS local directory (existing value of hbase.rootdir in ams-hbase-site) to HDFS directory. Example:
    cd /var/lib/ambari-metrics-collector/
    su - hdfs -c 'hdfs dfs -copyFromLocal hbase hdfs://<namnode-http-address>:8020/user/ams/'
    su - hdfs -c 'hdfs dfs -chown -R ams:hadoop /user/ams/hbase'
  • Start the Metric Collector after making the changes needed for distributed mode.

Please see the following link:

https://cwiki.apache.org/confluence/display/AMBARI/AMS+-+distributed+mode

Is there any config differences required in the HDP and HDF metrics collectors required to prevent one collector instance from clobbering the other in HBase, or does each metric collector do something to namespace the data (i.e. prepend cluster name)?

Expert Contributor

@Daniel Skates

Currently, AMS can operate only in 2 modes

1. embedded - Managed HBase writing to local disk

2. distributed - Managed HBase writing to HDFS

We do not yet support AMS writing to an external HBase (For example, the cluster HBase) directly. Tracked through https://issues.apache.org/jira/browse/AMBARI-14890.