Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Comparing Clusters' configurations

Solved Go to solution

Comparing Clusters' configurations

New Contributor

Hi,

Just wondering what is the best way to compare configurations for two clusters? Is there any feature available on Ambari UI to do the same? At present I think Ambari cannot support managing multiple clusters. So not sure if this would be possible at all.

Is exporting the cluster configurations as blueprints and then comparing them manually, the only option available? Or are there any better ways to do the same?

Thanks in advance.

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Comparing Clusters' configurations

Guru

Hi, if you just want to compare HDP components configurations you can use a simple shell script to export all configurations on each cluster and vimdiff the 2 files.

I made a very simple script to achieve that, feel free to use/update

https://github.com/laurentedel/hadoop-scripts/blob/master/backup_configs.sh

12 REPLIES 12

Re: Comparing Clusters' configurations

Expert Contributor

one way to compare configs for 2 different cluster is to get the current configurations for both the cluster using rest api and use some customized scripts to parsers the output.

https://cwiki.apache.org/confluence/display/AMBARI/Modify+configurations

Re: Comparing Clusters' configurations

Guru

Hi, if you just want to compare HDP components configurations you can use a simple shell script to export all configurations on each cluster and vimdiff the 2 files.

I made a very simple script to achieve that, feel free to use/update

https://github.com/laurentedel/hadoop-scripts/blob/master/backup_configs.sh

Re: Comparing Clusters' configurations

Mentor

You can also download client configs from UI and run diff on each configuration.

Re: Comparing Clusters' configurations

New Contributor

@Artem Ervits, but would this be for one cluster itself rather than comparison across clusters? Not sure if there is a better way than to get cluster configuration from Ambari REST APIs separately for each of them and then do a manual comparison?

Thanks

Re: Comparing Clusters' configurations

Mentor

You have to get configs for both clusters

Re: Comparing Clusters' configurations

Expert Contributor

@Greenhorn Techie I used Ambari Blueprints, Node.js and deep-diff to compare the configurations for two different clusters. The blueprints were loaded into Node.js and deep-diff did a nice job of finding the true differences in the JSON objects. The HOSTGROUP definitions in the blueprints created a fair number of red herrings but the other diff objects were dead on.

Assuming you can install Node.js, the deep-diff module and extract the blueprints (I know. A big assumption) you can use the following Node.js commands to generate a JSON file with the cluster differences.

var myCallback = function(err) {
if(err) {
return console.log(err);
}
console.log("The file was saved.");
}; 
var diff = require('deep-diff').diff;
var bpa = require('/path/to/blueprints/blue-print-a.json'); 
var bpb = require('/path/to/blueprints/blue-print-b.json'); 
var difference = diff(bpa,bpb); 
fs.writeFile('cluster-difference.json', JSON.stringify(difference, null, 99), myCallback);

Re: Comparing Clusters' configurations

Super Collaborator

I wanted to offer another comparison tool option. After downloading configs, you might consider using a graphical tool like meld. It does a very nice job of file and directory comparison and merges. It will not only point out line differences but highlight character differences. Check out the images on their home page to know what to expect: http://http://meldmerge.org

If you're doing a shell comparison using diff, try the -y option for a side-by-side diff. And there is a recursive option -r to compare directories. diff -ry <dirA> <dirB>

Highlighted

Re: Comparing Clusters' configurations

New Contributor

@james.jones Thanks James. I shall look into meld. Seems interesting.

Re: Comparing Clusters' configurations

Contributor

Another utility which it helps to compare the Hadoop ecosystem configurations for two or more clusters. It uses Curl request for retrieving the service configuration from Ambari

https://github.com/karthikhw/ambari

https://github.com/karthikhw/ambari/blob/master/bin/compare.sh

Note : Run it in ambari server

compare-sh-screenshot.jpg

Don't have an account?
Coming from Hortonworks? Activate your account here