Support Questions
Find answers, ask questions, and share your expertise
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for
- Cloudera Community
- :
- Support
- :
- Support Questions
- :
- How to use CGROUP in kerberized storm cluster?
Options
- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
Options
- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
How to use CGROUP in kerberized storm cluster?
Highlighted
How to use CGROUP in kerberized storm cluster?
Labels:
New Contributor
Created 08-24-2017 10:23 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content
I want to use CGROUP to limit users's use of cpu or memory in my kerberized storm cluster.
According to the storm document, I enabled CGROUP via ambari with
storm.cgroup.enable=true
and modified /etc/cgconfig.conf file as below
// on supervisor server
-bash-4.1$ id storm
uid=55696(storm) gid=10899(hadoop) groups=10899(hadoop)
-bash-4.1$ cat /etc/cgconfig.conf
#
# Copyright IBM Corporation. 2007
#
# Authors: Balbir Singh <balbir@linux.vnet.ibm.com>
# This program is free software; you can redistribute it and/or modify it
# under the terms of version 2.1 of the GNU Lesser General Public License
# as published by the Free Software Foundation.
#
# This program is distributed in the hope that it would be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
#
# See man cgconfig.conf for further details.
#
# By default, mount all controllers to /cgroup/<controller>
mount {
cpuset = /cgroup/cpuset;
cpu = /cgroup/storm_resources;
cpuacct = /cgroup/cpuacct;
memory = /cgroup/storm_resources;
devices = /cgroup/devices;
freezer = /cgroup/freezer;
net_cls = /cgroup/net_cls;
blkio = /cgroup/blkio;
}
group storm {
perm {
task {
uid = 55696;
gid = 10899;
}
admin {
uid = 55696;
gid = 10899;
}
}
cpu {
}
}
Because I also use Resource Aware Scheduler(RAS) and defined users's use of cpu and memory as
// on nimbus server
-bash-4.1$ cat /etc/storm/conf/user-resource-pools.yaml
resource.aware.scheduler.user.pools:
ap-bliu:
cpu: 10
memory: 896
and the configs for RAS are
topology.component.resources.onheap.memory.mb=64 topology.component.resources.offheap.memory.mb=0 topology.component.cpu.pcore.percent=3 topology.worker.max.heap.size.mb=768
so when user ap-bliu submits a new topology running with 2 workers(slots) and 6 executors on each worker, I expected to confirm the use of cpu of each worker should below 18 (3*6=18)
well, I got
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 30268 ap-bliu 20 0 3287m 443m 19m S 41.8 11.2 33:05.40 java 30264 ap-bliu 20 0 3297m 450m 19m S 36.2 11.4 30:19.55 java 30238 ap-bliu 20 0 2488m 80m 19m S 0.0 2.0 0:13.19 java 30240 ap-bliu 20 0 2488m 84m 19m S 0.0 2.1 0:11.44 java
and I even could not found cgexec in the command for worker starting
-bash-4.1$ ps aux | grep worker | grep -v LogWriter | grep -v grep ap-bliu 30264 29.2 11.3 3376364 461664 ? Sl 17:27 32:50 /usr/jdk64/jdk1.8.0_112/bin/java -server -Dlogging.sensitivity=S3 -Dlogfile.name=worker.log -Dstorm.home=/usr/hdp/2.6.1.0-129/storm -Dworkers.artifacts=/var/log/storm/workers-artifacts -Dstorm.id=ap-bliu-topology-02-2-1503563256 -Dworker.id=c3edab81-9e97-425a-9a1e-acffe5298d5c -Dworker.port=6700 -Dstorm.log.dir=/var/log/storm -Dlog4j.configurationFile=/usr/hdp/2.6.1.0-129/storm/log4j2/worker.xml -DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector -Dstorm.local.dir=/hadoop/storm -Xmx768m -Djava.security.auth.login.config=/usr/hdp/current/storm-supervisor/conf/storm_jaas.conf -javaagent:/usr/hdp/current/storm-client/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar=host=localhost,port=8650,wireformat31x=true,mode=multicast,config=/usr/hdp/current/storm-client/contrib/storm-jmxetric/conf/jmxetric-conf.xml,process=Worker_6700_JVM -Djava.library.path=/hadoop/storm/supervisor/stormdist/ap-bliu-topology-02-2-1503563256/resources/Linux-amd64:/hadoop/storm/supervisor/stormdist/ap-bliu-topology-02-2-1503563256/resources:/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib -Dstorm.conf.file= -Dstorm.options= -Djava.io.tmpdir=/hadoop/storm/workers/c3edab81-9e97-425a-9a1e-acffe5298d5c/tmp -cp /usr/hdp/2.6.1.0-129/storm/lib/ambari-metrics-storm-sink.jar:/usr/hdp/2.6.1.0-129/storm/lib/ring-cors-0.1.5.jar:/usr/hdp/2.6.1.0-129/storm/lib/storm-rename-hack-1.1.0.2.6.1.0-129.jar:/usr/hdp/2.6.1.0-129/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.6.1.0-129/storm/lib/asm-5.0.3.jar:/usr/hdp/2.6.1.0-129/storm/lib/log4j-api-2.8.2.jar:/usr/hdp/2.6.1.0-129/storm/lib/log4j-core-2.8.2.jar:/usr/hdp/2.6.1.0-129/storm/lib/zookeeper.jar:/usr/hdp/2.6.1.0-129/storm/lib/disruptor-3.3.2.jar:/usr/hdp/2.6.1.0-129/storm/lib/log4j-slf4j-impl-2.8.2.jar:/usr/hdp/2.6.1.0-129/storm/lib/reflectasm-1.10.1.jar:/usr/hdp/2.6.1.0-129/storm/lib/objenesis-2.1.jar:/usr/hdp/2.6.1.0-129/storm/lib/kryo-3.0.3.jar:/usr/hdp/2.6.1.0-129/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.6.1.0-129/storm/lib/clojure-1.7.0.jar:/usr/hdp/2.6.1.0-129/storm/lib/minlog-1.3.0.jar:/usr/hdp/2.6.1.0-129/storm/lib/storm-core-1.1.0.2.6.1.0-129.jar:/usr/hdp/2.6.1.0-129/storm/lib/slf4j-api-1.7.21.jar:/usr/hdp/current/storm-supervisor/conf:/hadoop/storm/supervisor/stormdist/ap-bliu-topology-02-2-1503563256/stormjar.jar org.apache.storm.daemon.worker ap-bliu-topology-02-2-1503563256 c5e355da-3b6f-4c22-84a4-b6524aa037f0 6700 c3edab81-9e97-425a-9a1e-acffe5298d5c ap-bliu 30268 32.0 11.2 3366668 454348 ? Sl 17:27 35:56 /usr/jdk64/jdk1.8.0_112/bin/java -server -Dlogging.sensitivity=S3 -Dlogfile.name=worker.log -Dstorm.home=/usr/hdp/2.6.1.0-129/storm -Dworkers.artifacts=/var/log/storm/workers-artifacts -Dstorm.id=ap-bliu-topology-02-2-1503563256 -Dworker.id=44528bbd-2eb3-43d6-9444-30e8857c72f6 -Dworker.port=6701 -Dstorm.log.dir=/var/log/storm -Dlog4j.configurationFile=/usr/hdp/2.6.1.0-129/storm/log4j2/worker.xml -DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector -Dstorm.local.dir=/hadoop/storm -Xmx768m -Djava.security.auth.login.config=/usr/hdp/current/storm-supervisor/conf/storm_jaas.conf -javaagent:/usr/hdp/current/storm-client/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar=host=localhost,port=8650,wireformat31x=true,mode=multicast,config=/usr/hdp/current/storm-client/contrib/storm-jmxetric/conf/jmxetric-conf.xml,process=Worker_6701_JVM -Djava.library.path=/hadoop/storm/supervisor/stormdist/ap-bliu-topology-02-2-1503563256/resources/Linux-amd64:/hadoop/storm/supervisor/stormdist/ap-bliu-topology-02-2-1503563256/resources:/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib -Dstorm.conf.file= -Dstorm.options= -Djava.io.tmpdir=/hadoop/storm/workers/44528bbd-2eb3-43d6-9444-30e8857c72f6/tmp -cp /usr/hdp/2.6.1.0-129/storm/lib/ambari-metrics-storm-sink.jar:/usr/hdp/2.6.1.0-129/storm/lib/ring-cors-0.1.5.jar:/usr/hdp/2.6.1.0-129/storm/lib/storm-rename-hack-1.1.0.2.6.1.0-129.jar:/usr/hdp/2.6.1.0-129/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.6.1.0-129/storm/lib/asm-5.0.3.jar:/usr/hdp/2.6.1.0-129/storm/lib/log4j-api-2.8.2.jar:/usr/hdp/2.6.1.0-129/storm/lib/log4j-core-2.8.2.jar:/usr/hdp/2.6.1.0-129/storm/lib/zookeeper.jar:/usr/hdp/2.6.1.0-129/storm/lib/disruptor-3.3.2.jar:/usr/hdp/2.6.1.0-129/storm/lib/log4j-slf4j-impl-2.8.2.jar:/usr/hdp/2.6.1.0-129/storm/lib/reflectasm-1.10.1.jar:/usr/hdp/2.6.1.0-129/storm/lib/objenesis-2.1.jar:/usr/hdp/2.6.1.0-129/storm/lib/kryo-3.0.3.jar:/usr/hdp/2.6.1.0-129/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.6.1.0-129/storm/lib/clojure-1.7.0.jar:/usr/hdp/2.6.1.0-129/storm/lib/minlog-1.3.0.jar:/usr/hdp/2.6.1.0-129/storm/lib/storm-core-1.1.0.2.6.1.0-129.jar:/usr/hdp/2.6.1.0-129/storm/lib/slf4j-api-1.7.21.jar:/usr/hdp/current/storm-supervisor/conf:/hadoop/storm/supervisor/stormdist/ap-bliu-topology-02-2-1503563256/stormjar.jar org.apache.storm.daemon.worker ap-bliu-topology-02-2-1503563256 c5e355da-3b6f-4c22-84a4-b6524aa037f0 6701 44528bbd-2eb3-43d6-9444-30e8857c72f6
Is there anyone can tell me why and how to use CGROUP with RAS on storm?
Other info
-bash-4.1$ cat /etc/redhat-release CentOS release 6.8 (Final) -bash-4.1$ hdp-select versions 2.6.1.0-129 -bash-4.1$ storm version /usr/bin/storm: line 2: /usr/hdp/2.6.1.0-129/etc/default/hadoop: No such file or directory Running: /usr/jdk64/jdk1.8.0_112/bin/java -Ddaemon.name= -Dstorm.options= -Dstorm.home=/usr/hdp/2.6.1.0-129/storm -Dstorm.log.dir=/var/log/storm -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib -Dstorm.conf.file= -cp /usr/hdp/2.6.1.0-129/storm/lib/ambari-metrics-storm-sink.jar:/usr/hdp/2.6.1.0-129/storm/lib/ring-cors-0.1.5.jar:/usr/hdp/2.6.1.0-129/storm/lib/storm-rename-hack-1.1.0.2.6.1.0-129.jar:/usr/hdp/2.6.1.0-129/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.6.1.0-129/storm/lib/asm-5.0.3.jar:/usr/hdp/2.6.1.0-129/storm/lib/log4j-api-2.8.2.jar:/usr/hdp/2.6.1.0-129/storm/lib/log4j-core-2.8.2.jar:/usr/hdp/2.6.1.0-129/storm/lib/zookeeper.jar:/usr/hdp/2.6.1.0-129/storm/lib/disruptor-3.3.2.jar:/usr/hdp/2.6.1.0-129/storm/lib/log4j-slf4j-impl-2.8.2.jar:/usr/hdp/2.6.1.0-129/storm/lib/reflectasm-1.10.1.jar:/usr/hdp/2.6.1.0-129/storm/lib/objenesis-2.1.jar:/usr/hdp/2.6.1.0-129/storm/lib/kryo-3.0.3.jar:/usr/hdp/2.6.1.0-129/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.6.1.0-129/storm/lib/clojure-1.7.0.jar:/usr/hdp/2.6.1.0-129/storm/lib/minlog-1.3.0.jar:/usr/hdp/2.6.1.0-129/storm/lib/storm-core-1.1.0.2.6.1.0-129.jar:/usr/hdp/2.6.1.0-129/storm/lib/slf4j-api-1.7.21.jar:/usr/hdp/2.6.1.0-129/storm/extlib-daemon/ojdbc6.jar:/usr/hdp/2.6.1.0-129/storm/extlib-daemon/ranger-storm-plugin-shim-0.7.0.2.6.1.0-129.jar:/usr/hdp/2.6.1.0-129/storm/extlib-daemon/ranger-plugin-classloader-0.7.0.2.6.1.0-129.jar:/usr/hdp/current/storm-supervisor/conf org.apache.storm.utils.VersionInfo Storm 1.1.0.2.6.1.0-129
