Created 01-20-2020 02:05 AM
<Version>
HDP 3.1.0
Ambari 2.7.3
I want to change memory size of the container of ats-hbase RegionServer on YARN NodeManager.
# The internal HBase of the YARN Timeline Service uses the HBase set when installing HDP in System Service mode, instead of the external HBase.
I changed the parameter yarn_hbase_regionserver_memory (Ambari UI-> YARN-> Configs-> Advanced-> yarn_hbase_regionserver_memory) on Ambari, but ats-hbase RegionServer container memory size did not change.
Regarding that the container memory size of ats-hbase RegionServer did not change,
I confirmed the value (Ex. Resource: 4096 Memory, 1 VCores) in Resource column on the Container Overview page of the ats-hbase RegionServer container from the YARN ResourceManager WebUI.
It was also unclear where the change of yarn_hbase_regionserver_memory was reflected.
And, in order to see if the ats-hbase Master container and the RegionServer container really launched with 4GB memory size, I set yarn.scheduler.maximum-allocation-mb less than 4GB (Ex.3GB) and checked if ats-hbase worked properly.
As a result, I found that ats-hbase failed to start because the container of ats-hbase Master and RegionServer tried to launch with 4GB memory size.
$ yarn app -start ats-hbase
20/01/16 09:49:20 INFO client.AHSProxy: Connecting to Application History server at hdpm03.***.jp/172.***.***.***:10200
20/01/16 09:49:20 INFO client.AHSProxy: Connecting to Application History server at hdpm03.***.jp/172.***.***.***:10200
20/01/16 09:49:20 INFO util.log: Logging initialized @1417ms
20/01/16 09:49:20 ERROR client.ApiServiceClient: Component master: specified memory size (4096) is larger than configured max container memory size (3072)
So, I have two questions.
<QUESTION>
1. Ats-hbase Master container and RegionServer container always seem to launch with 4GB memory size, is there any way to change the memory size?
2. What does the parameter yarn_hbase_regionserver_memory reflect?
There is a description "regionserver container memory in MB" on Ambari,
but it does not seem to work exactly.
Created 01-20-2020 08:48 AM
@nsup you can tune ats-hbase memory in yarn configuration as below.
yarn_hbase_master_memory
yarn_hbase_regionserver_memory
yarn_hbase_client_memory
Created 01-21-2020 01:15 AM
Thank you for your comment.
In my environment, I changed these parameters from Ambari, but it didn't work.
- yarn_hbase_master_memory
- yarn_hbase_regionserver_memory
- yarn_hbase_client_memory
For example, I changed yarn_hbase_regionserver_memory from 4096 to 2048 MB, then restarted YARN and ats-hbase.
# The restarting method of ats-hbase is "yarn app -stop/start ats-hbase".
After that, I checked below, but I found that changing the parameter didn't reflect.
- container overview on ResourceManager Web UI
- the result of the command "yarn app -status ats-hbase"
============================================================
"resource":{"cpus":1,"memory":"4096","additional":{}},"state":"FLEXING","configuration":{"properties":{"yarn.service.container-failure.retry.max":"10","yarn.service.framework.path":"/hdp/apps/3.1.0.0-78/yarn/rm1/service-dep.tar.gz"},"env":{"HBASE_LOG_PREFIX":"hbase-$HBASE_IDENT_STRING-regionserver-$HOSTNAME","HBASE_LOGFILE":"$HBASE_LOG_PREFIX.log","HBASE_REGIONSERVER_OPTS":"-XX:CMSInitiatingOccupancyFraction=70 -XX:ReservedCodeCacheSize=256m -Xms3276m -Xmx3276m"
============================================================
- ps command details on the host running ats-hbase RegionServer container
============================================================
$ ps -ef | grep -i Region
yarn 16261 16242 13 17:56 ? 00:00:27 /usr/lib/jvm/java-1.8.0//bin/java -Dproc_regionserver -XX:OnOutOfMemoryError=kill -9 %p -XX:+UseConcMarkSweepGC -XX:ErrorFile=/var/hadoop/d02/yarn/log/application_1579594139823_0001/container_e174_1579594139823_0001_01_000003/hs_err_pid%p.log -Djava.io.tmpdir=/tmp -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:/var/hadoop/d02/yarn/log/application_1579594139823_0001/container_e174_1579594139823_0001_01_000003/gc.log-202001211756 -XX:CMSInitiatingOccupancyFraction=70 -XX:ReservedCodeCacheSize=256m -Xms3276m -Xmx3276m
...
-Dhbase.security.logger=INFO,RFAS org.apache.hadoop.hbase.regionserver.HRegionServer start
============================================================
So, could you tell me where and what changes by changing the parameters?
Created 12-10-2020 08:38 AM
After changing the aforementioned parameters, you also need to clean up the ats-hbase config files from HDFS and restart YARN so the changes take effect.
The ats-hbase cleanup steps are detailed here, don't forget to restart YARN after applying them.