Overwrite yarn, mapred, tez and hive configurations after restarting services in Ambari

[#1]

Hi,

I ran the yarn calculator and it suggested me to update the parameters in mapred-site.xml, yarn-site.xml ,tez-site.xml and hive-site.xml. So i followed the following steps but after starting the services, it overwrites the configurations and i cannot see any changes in the xml files.

- Stop all the servcies through Ambari

- Updated the xml files

- Started all services via Ambari

Any advice?

Thanks,

SJ

[#2 · Accepted solution]

- Updated the xml files

For the above step, are you making the changes manually directly to the xml files or via Ambari UI in Configs tab? If you are making the changes outside Ambari, then Ambari always overwrites them when you start services via Ambari. Please make changes via Ambari Configs tab to be able to persist them across restarts.

-Gour

[#3]

Hi Gour,

Can you please let me know how should i update them in the Ambari config? I cannot find those parameters at all in the config file also i tried to add the manually as a custom yarnsite properties but when for example i tried to add "yarn.scheduler.minimum-allocation-mb", it says that it is already exist which is true ( they are exist in xml file)

SJ.

[#4]

yarn.scheduler.minimum-allocation-mb is exposed by Ambari in the "Settings" tab as a slider (shown below) and not in the Advanced tab. That's why you will not find it anywhere in the Advanced tab.

The best way to find any config is to lookup in the search/filter bar on the top/right as shown below.

When you search for "yarn.scheduler.minimum-allocation-mb" in that search box it will show you only the "Minimum Container Size (Memory)" slider as shown in the first image above.

[#5]

Hi,

I have two issues :

1- I can see this parameter but it doesn't let me to increase that much (server has 250G memory but it let me to increase it up to 3976MB. untitled1.png

2- when i run calculator, it recomends to change serveral parameters per below. I cannot find them in the GUI.

***** mapred-site.xml *****

mapreduce.map.memory.mb=32768

mapreduce.map.java.opts=-Xmx25600m

mapreduce.reduce.memory.mb=32768

mapreduce.reduce.java.opts=-Xmx25600m

mapreduce.task.io.sort.mb=1792

***** yarn-site.xml *****

yarn.scheduler.minimum-allocation-mb=32768

yarn.scheduler.maximum-allocation-mb=262144

yarn.nodemanager.resource.memory-mb=262144

yarn.app.mapreduce.am.resource.mb=32768

yarn.app.mapreduce.am.command-opts=-Xmx25600m

***** tez-site.xml *****

tez.am.resource.memory.mb=32768

tez.am.java.opts=-Xmx25600m

***** hive-site.xml *****

hive.tez.container.size=32768

hive.tez.java.opts=-Xmx25600m

hive.auto.convert.join.noconditionaltask.size=8589934000

Please advise.

[#6]

1- You can always click on the red-pen edit button and manually enter the exact value you want.

2- You can uncheck and ignore the changes recommended by Ambari or accept the changes. To view mapred, tez and hive config properties, make sure you are in the corresponding services tab.