Created on 05-25-2016 11:07 AM - edited 09-16-2022 03:21 AM
Guys,
We have various clusters and heavily configured over the period of time, and would like to replicate same cluster and another production environment.
Secondly : Most importantly, would like to replicate any changes made in one prod environment to another automatically. Like any addition of rules in Ranger or any other configuration changes.
Is there any tool out of the box?
Alredy gone through : https://community.hortonworks.com/questions/29575/comparing-clusters-configurations.html
Created 05-25-2016 01:15 PM
Hi @Smart Solutions.
Right now... there isn't really a good, quick, clean, easy way to achieve this.
You've already identified the thread that I would otherwise point you towards for some ideas. You're just too good!
There are two main approaches that I would recommend thinking about, the first is ceasing to make any changes via the web-ui, and only making changes via the API, that way you can just call both your clusters one after the other to make the configuration changes.
The second is to use some of the ideas from the thread you linked to, where you would continue maintaining the configs on your "master" cluster, but then extract the configs from your "master" cluster on a regular basis (or on a trigger of config changed?), diff them between the previous "master" cluster config version, and then push the resulting deltas to your "slave" cluster, again via API calls.
Either way, there's quite a bit of automation that would be required there.
I'd strongly suggest if you want to go down this path, doing your work out in the open, this is something I see come up now and again, so I think you may well get others who would be interested in working with you on this.
Longer term, Ambari will no doubt support multi cluster, and this functionality would be a natural extension of that, but progress on those public JIRA's has been slow, with other more important items taking priority.
Happy to hear if you have other ideas too, sorry I couldn't be more direct help, but let me know if you plan on cutting some code moving forward, I'm sure it'd be an interesting project.
Many thanks.
Created 05-25-2016 01:15 PM
Hi @Smart Solutions.
Right now... there isn't really a good, quick, clean, easy way to achieve this.
You've already identified the thread that I would otherwise point you towards for some ideas. You're just too good!
There are two main approaches that I would recommend thinking about, the first is ceasing to make any changes via the web-ui, and only making changes via the API, that way you can just call both your clusters one after the other to make the configuration changes.
The second is to use some of the ideas from the thread you linked to, where you would continue maintaining the configs on your "master" cluster, but then extract the configs from your "master" cluster on a regular basis (or on a trigger of config changed?), diff them between the previous "master" cluster config version, and then push the resulting deltas to your "slave" cluster, again via API calls.
Either way, there's quite a bit of automation that would be required there.
I'd strongly suggest if you want to go down this path, doing your work out in the open, this is something I see come up now and again, so I think you may well get others who would be interested in working with you on this.
Longer term, Ambari will no doubt support multi cluster, and this functionality would be a natural extension of that, but progress on those public JIRA's has been slow, with other more important items taking priority.
Happy to hear if you have other ideas too, sorry I couldn't be more direct help, but let me know if you plan on cutting some code moving forward, I'm sure it'd be an interesting project.
Many thanks.