Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

how to calculate total number of vcores in ambari cluster

Highlighted

how to calculate total number of vcores in ambari cluster

New Contributor

In cloudera manager we are calculating the total number of vcore in YARN cluster using the below relation of individual clusters:

 

roles->roleConfigGroupName->roleConfigGroup->yarn_nodemanager_resource_cpu_vcores
Is there similar option available with ambari? If not whats is the alternate option to calculate the total vcores in a YARN cluster?

 

Note: clusters may have different node configuration

2 REPLIES 2

Re: how to calculate total number of vcores in ambari cluster

Mentor

@Ram_85 

Just tried it out the link still functions, in HDP you have a python script  so grab this tar file this HDP configuration scripts are still  available handy and provide you recommended settings based on the number of cores and memory of your servers

Use tmp or whatever you could delete later

# cd /tmp

Download the script

# wget http://public-repo-1.hortonworks.com/HDP/tools/2.3.4.0/hdp_manual_install_rpm_helper_files-2.3.4.0.3...

 

Untar

# tar zxvf hdp_manual_install_rpm_helper_files-2.3.4.0.3485.tar.gz

 

Execute

# cd hdp_manual_install_rpm_helper_files-2.3.4.0.3485

 

# cd /tmp/hdp_manual_install_rpm_helper_files-2.3.4.0.3485/scripts

 

You MUST run this script as user yarn so switch user

# su - yarn

$ cd /tmp/hdp_manual_install_rpm_helper_files-2.3.4.0.3485/scripts

$ python yarn-utils.py

 

Sample Output

 

Using cores=16 memory=64GB disks=4 hbase=True
Profile: cores=16 memory=49152MB reserved=16GB usableMem=48GB disks=4
Num Container=8
Container Ram=6144MB
Used Ram=48GB
Unused Ram=16GB
yarn.scheduler.minimum-allocation-mb=6144
yarn.scheduler.maximum-allocation-mb=49152
yarn.nodemanager.resource.memory-mb=49152
mapreduce.map.memory.mb=6144
mapreduce.map.java.opts=-Xmx4915m
mapreduce.reduce.memory.mb=6144
mapreduce.reduce.java.opts=-Xmx4915m
yarn.app.mapreduce.am.resource.mb=6144
yarn.app.mapreduce.am.command-opts=-Xmx4915m
mapreduce.task.io.sort.mb=2457

 

Hope that helps

Re: how to calculate total number of vcores in ambari cluster

Expert Contributor

@Ram_85  In addition to what @shelton mentioned this script will give you generic recommendation setting for your cluster.

It might be the case that those values from "$ python yarn-utils.py" will not help your cluster wrt performance.

 

So kindly do refer the link to understand in details about the values you need to set for your cluster -

http://crazyadmins.com/tag/yarn-tuning/

 

Also If you already have support subscription opted for HDP then you can better install component name "SMARTSENSE" which will help you to analyze you cluster and give recommendations.

 

Don't have an account?
Coming from Hortonworks? Activate your account here