question Re: Comparing Clusters' configurations in Support Questions

Comparing Clusters' configurations

techiegreenhorn — Tue, 26 Apr 2016 16:07:19 GMT

Hi,

Just wondering what is the best way to compare configurations for two clusters? Is there any feature available on Ambari UI to do the same? At present I think Ambari cannot support managing multiple clusters. So not sure if this would be possible at all.

Is exporting the cluster configurations as blueprints and then comparing them manually, the only option available? Or are there any better ways to do the same?

Thanks in advance.

Re: Comparing Clusters' configurations

pankaj_singh — Tue, 26 Apr 2016 16:12:38 GMT

one way to compare configs for 2 different cluster is to get the current configurations for both the cluster using rest api and use some customized scripts to parsers the output.

https://cwiki.apache.org/confluence/display/AMBARI/Modify+configurations

Re: Comparing Clusters' configurations

ledel — Tue, 26 Apr 2016 16:18:16 GMT

Hi, if you just want to compare HDP components configurations you can use a simple shell script to export all configurations on each cluster and vimdiff the 2 files.

I made a very simple script to achieve that, feel free to use/update

https://github.com/laurentedel/hadoop-scripts/blob/master/backup_configs.sh

Re: Comparing Clusters' configurations

aervits — Tue, 26 Apr 2016 16:36:31 GMT

You can also download client configs from UI and run diff on each configuration.

Re: Comparing Clusters' configurations

techiegreenhorn — Tue, 26 Apr 2016 17:56:32 GMT

@Artem Ervits, but would this be for one cluster itself rather than comparison across clusters? Not sure if there is a better way than to get cluster configuration from Ambari REST APIs separately for each of them and then do a manual comparison?

Thanks

Re: Comparing Clusters' configurations

aervits — Tue, 26 Apr 2016 18:29:04 GMT

You have to get configs for both clusters

Re: Comparing Clusters' configurations

TerryP — Wed, 27 Apr 2016 21:32:44 GMT

@Greenhorn Techie I used Ambari Blueprints, Node.js and deep-diff to compare the configurations for two different clusters. The blueprints were loaded into Node.js and deep-diff did a nice job of finding the true differences in the JSON objects. The HOSTGROUP definitions in the blueprints created a fair number of red herrings but the other diff objects were dead on.

Assuming you can install Node.js, the deep-diff module and extract the blueprints (I know. A big assumption) you can use the following Node.js commands to generate a JSON file with the cluster differences.

var myCallback = function(err) {
if(err) {
return console.log(err);
}
console.log("The file was saved.");
}; 
var diff = require('deep-diff').diff;
var bpa = require('/path/to/blueprints/blue-print-a.json'); 
var bpb = require('/path/to/blueprints/blue-print-b.json'); 
var difference = diff(bpa,bpb); 
fs.writeFile('cluster-difference.json', JSON.stringify(difference, null, 99), myCallback);

Re: Comparing Clusters' configurations

james_jones — Thu, 28 Apr 2016 20:31:32 GMT

I wanted to offer another comparison tool option. After downloading configs, you might consider using a graphical tool like meld. It does a very nice job of file and directory comparison and merges. It will not only point out line differences but highlight character differences. Check out the images on their home page to know what to expect: http://http://meldmerge.org

If you're doing a shell comparison using diff, try the -y option for a side-by-side diff. And there is a recursive option -r to compare directories. diff -ry <dirA> <dirB>

Re: Comparing Clusters' configurations

techiegreenhorn — Fri, 29 Apr 2016 01:42:23 GMT

@james.jones Thanks James. I shall look into meld. Seems interesting.

Re: Comparing Clusters' configurations

kpalanisamy — Sat, 20 Aug 2016 00:00:29 GMT

Another utility which it helps to compare the Hadoop ecosystem configurations for two or more clusters. It uses Curl request for retrieving the service configuration from Ambari

https://github.com/karthikhw/ambari

https://github.com/karthikhw/ambari/blob/master/bin/compare.sh

Note : Run it in ambari server

compare-sh-screenshot.jpg

Re: Comparing Clusters' configurations

Narendra_Bidari — Wed, 21 Sep 2016 05:27:13 GMT

The below script helps to compare the configs.

https://github.com/Symantec/config-compare

Re: Comparing Clusters' configurations

dale_rodel1 — Thu, 22 Sep 2016 02:52:46 GMT

This is a pretty useful tool. You can compare the entire Ambari blueprints and it generates a tab delimited report of the the differences and similarities between two or more clusters.

Re: Comparing Clusters' configurations

sturner — Sat, 28 Jan 2017 00:50:07 GMT

If you are still seeking a solution you could use Orcaconfig to compare dumped XMLs. https://www.orcaconfig.com/compare-configurations/. It's designed to automatically compare configurations in complex environments. Hope this helps you. (yep I work there)