We found an issue after re-installing ambari-metrics in the HDP cluster.
We first install ambari-metrics in 50 hosts in a new cluster, then install kafka in 20 hosts and storm in the other 30. Each kafka broker has its own group_config to override the broker.id, broker.rack and host.name. Same with storm where we overridden for each supervisor and nimbus the storm.local.hostname property.
Everythings were good at that point.
We faced some issue with ambari-metrics and failed to solve them (that's another story). We decide to re-install the component.
We proceed like that:
Remove znode /ams-hbase-unsecured from the embedded zookeeper (localhost:21181)
Stop all ambari-metrics instances
Delete ambari-metrics services from ambari
Remove all rpm related to ambari-metrics
Remove all folder related to ambari-metrics and hbase(/var/lib/ambari-metrics-collector, ...)
Re-install ambari-metrics service from ambari.
After re-installing with success, we discover that each group_config for kafka and storm were overridden with all the same value. All had the broker.id set to 1, broker.rack = rack1 and host.name= myhostname1.mycompany.com. Same for storm.
We test once more the procedure in another cluster with the same configuration and it's happened exactly the same.