Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Comparing Clusters' configurations

avatar
Rising Star

Hi,

Just wondering what is the best way to compare configurations for two clusters? Is there any feature available on Ambari UI to do the same? At present I think Ambari cannot support managing multiple clusters. So not sure if this would be possible at all.

Is exporting the cluster configurations as blueprints and then comparing them manually, the only option available? Or are there any better ways to do the same?

Thanks in advance.

1 ACCEPTED SOLUTION

avatar
Guru

Hi, if you just want to compare HDP components configurations you can use a simple shell script to export all configurations on each cluster and vimdiff the 2 files.

I made a very simple script to achieve that, feel free to use/update

https://github.com/laurentedel/hadoop-scripts/blob/master/backup_configs.sh

View solution in original post

12 REPLIES 12

avatar
Super Collaborator

one way to compare configs for 2 different cluster is to get the current configurations for both the cluster using rest api and use some customized scripts to parsers the output.

https://cwiki.apache.org/confluence/display/AMBARI/Modify+configurations

avatar
Guru

Hi, if you just want to compare HDP components configurations you can use a simple shell script to export all configurations on each cluster and vimdiff the 2 files.

I made a very simple script to achieve that, feel free to use/update

https://github.com/laurentedel/hadoop-scripts/blob/master/backup_configs.sh

avatar
Master Mentor

You can also download client configs from UI and run diff on each configuration.

avatar
Rising Star

@Artem Ervits, but would this be for one cluster itself rather than comparison across clusters? Not sure if there is a better way than to get cluster configuration from Ambari REST APIs separately for each of them and then do a manual comparison?

Thanks

avatar
Master Mentor

You have to get configs for both clusters

avatar
Super Collaborator

@Greenhorn Techie I used Ambari Blueprints, Node.js and deep-diff to compare the configurations for two different clusters. The blueprints were loaded into Node.js and deep-diff did a nice job of finding the true differences in the JSON objects. The HOSTGROUP definitions in the blueprints created a fair number of red herrings but the other diff objects were dead on.

Assuming you can install Node.js, the deep-diff module and extract the blueprints (I know. A big assumption) you can use the following Node.js commands to generate a JSON file with the cluster differences.

var myCallback = function(err) {
if(err) {
return console.log(err);
}
console.log("The file was saved.");
}; 
var diff = require('deep-diff').diff;
var bpa = require('/path/to/blueprints/blue-print-a.json'); 
var bpb = require('/path/to/blueprints/blue-print-b.json'); 
var difference = diff(bpa,bpb); 
fs.writeFile('cluster-difference.json', JSON.stringify(difference, null, 99), myCallback);

avatar
Super Collaborator

I wanted to offer another comparison tool option. After downloading configs, you might consider using a graphical tool like meld. It does a very nice job of file and directory comparison and merges. It will not only point out line differences but highlight character differences. Check out the images on their home page to know what to expect: http://http://meldmerge.org

If you're doing a shell comparison using diff, try the -y option for a side-by-side diff. And there is a recursive option -r to compare directories. diff -ry <dirA> <dirB>

avatar
Rising Star

@james.jones Thanks James. I shall look into meld. Seems interesting.

avatar
Expert Contributor

Another utility which it helps to compare the Hadoop ecosystem configurations for two or more clusters. It uses Curl request for retrieving the service configuration from Ambari

https://github.com/karthikhw/ambari

https://github.com/karthikhw/ambari/blob/master/bin/compare.sh

Note : Run it in ambari server

compare-sh-screenshot.jpg