Comparing Clusters' configurations

[#1]

Hi,

Just wondering what is the best way to compare configurations for two clusters? Is there any feature available on Ambari UI to do the same? At present I think Ambari cannot support managing multiple clusters. So not sure if this would be possible at all.

Is exporting the cluster configurations as blueprints and then comparing them manually, the only option available? Or are there any better ways to do the same?

Thanks in advance.

[#2]

one way to compare configs for 2 different cluster is to get the current configurations for both the cluster using rest api and use some customized scripts to parsers the output.

https://cwiki.apache.org/confluence/display/AMBARI/Modify+configurations

[#3 · Accepted solution]

Hi, if you just want to compare HDP components configurations you can use a simple shell script to export all configurations on each cluster and vimdiff the 2 files.

I made a very simple script to achieve that, feel free to use/update

https://github.com/laurentedel/hadoop-scripts/blob/master/backup_configs.sh

[#4]

You can also download client configs from UI and run diff on each configuration.

[#5]

@Artem Ervits, but would this be for one cluster itself rather than comparison across clusters? Not sure if there is a better way than to get cluster configuration from Ambari REST APIs separately for each of them and then do a manual comparison?

Thanks

[#6]

You have to get configs for both clusters

[#7]

@Greenhorn Techie I used Ambari Blueprints, Node.js and deep-diff to compare the configurations for two different clusters. The blueprints were loaded into Node.js and deep-diff did a nice job of finding the true differences in the JSON objects. The HOSTGROUP definitions in the blueprints created a fair number of red herrings but the other diff objects were dead on.

Assuming you can install Node.js, the deep-diff module and extract the blueprints (I know. A big assumption) you can use the following Node.js commands to generate a JSON file with the cluster differences.

var myCallback = function(err) {
if(err) {
return console.log(err);
}
console.log("The file was saved.");
}; 
var diff = require('deep-diff').diff;
var bpa = require('/path/to/blueprints/blue-print-a.json'); 
var bpb = require('/path/to/blueprints/blue-print-b.json'); 
var difference = diff(bpa,bpb); 
fs.writeFile('cluster-difference.json', JSON.stringify(difference, null, 99), myCallback);

[#8]

I wanted to offer another comparison tool option. After downloading configs, you might consider using a graphical tool like meld. It does a very nice job of file and directory comparison and merges. It will not only point out line differences but highlight character differences. Check out the images on their home page to know what to expect: http://http://meldmerge.org

If you're doing a shell comparison using diff, try the -y option for a side-by-side diff. And there is a recursive option -r to compare directories. diff -ry <dirA> <dirB>

[#9]

@james.jones Thanks James. I shall look into meld. Seems interesting.

[#10]

Another utility which it helps to compare the Hadoop ecosystem configurations for two or more clusters. It uses Curl request for retrieving the service configuration from Ambari

https://github.com/karthikhw/ambari

https://github.com/karthikhw/ambari/blob/master/bin/compare.sh

Note : Run it in ambari server

compare-sh-screenshot.jpg