Support Questions
Find answers, ask questions, and share your expertise

How to use CGROUP in kerberized storm cluster?

Highlighted

How to use CGROUP in kerberized storm cluster?

New Contributor

I want to use CGROUP to limit users's use of cpu or memory in my kerberized storm cluster.

According to the storm document, I enabled CGROUP via ambari with

storm.cgroup.enable=true

and modified /etc/cgconfig.conf file as below

// on supervisor server
-bash-4.1$ id storm
uid=55696(storm) gid=10899(hadoop) groups=10899(hadoop)
-bash-4.1$ cat /etc/cgconfig.conf
#
#  Copyright IBM Corporation. 2007
#
#  Authors:	Balbir Singh <balbir@linux.vnet.ibm.com>
#  This program is free software; you can redistribute it and/or modify it
#  under the terms of version 2.1 of the GNU Lesser General Public License
#  as published by the Free Software Foundation.
#
#  This program is distributed in the hope that it would be useful, but
#  WITHOUT ANY WARRANTY; without even the implied warranty of
#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
#
# See man cgconfig.conf for further details.
#
# By default, mount all controllers to /cgroup/<controller>


mount {
    cpuset  = /cgroup/cpuset;
    cpu = /cgroup/storm_resources;
    cpuacct = /cgroup/cpuacct;
    memory  = /cgroup/storm_resources;
    devices = /cgroup/devices;
    freezer = /cgroup/freezer;
    net_cls = /cgroup/net_cls;
    blkio   = /cgroup/blkio;
}


group storm {
    perm {
        task {
       	    uid = 55696;
            gid = 10899;
        }
        admin {
            uid = 55696;
            gid = 10899;
        }
    }
    cpu {
    }
}

Because I also use Resource Aware Scheduler(RAS) and defined users's use of cpu and memory as

// on nimbus server
-bash-4.1$ cat /etc/storm/conf/user-resource-pools.yaml
resource.aware.scheduler.user.pools:
    ap-bliu:
        cpu: 10
        memory: 896

and the configs for RAS are

topology.component.resources.onheap.memory.mb=64
topology.component.resources.offheap.memory.mb=0
topology.component.cpu.pcore.percent=3
topology.worker.max.heap.size.mb=768

so when user ap-bliu submits a new topology running with 2 workers(slots) and 6 executors on each worker, I expected to confirm the use of cpu of each worker should below 18 (3*6=18)

well, I got

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
30268 ap-bliu   20   0 3287m 443m  19m S 41.8 11.2  33:05.40 java
30264 ap-bliu   20   0 3297m 450m  19m S 36.2 11.4  30:19.55 java
30238 ap-bliu   20   0 2488m  80m  19m S  0.0  2.0   0:13.19 java
30240 ap-bliu   20   0 2488m  84m  19m S  0.0  2.1   0:11.44 java

and I even could not found cgexec in the command for worker starting

-bash-4.1$ ps aux | grep worker | grep -v LogWriter | grep -v grep
ap-bliu  30264 29.2 11.3 3376364 461664 ?      Sl   17:27  32:50 /usr/jdk64/jdk1.8.0_112/bin/java -server -Dlogging.sensitivity=S3 -Dlogfile.name=worker.log -Dstorm.home=/usr/hdp/2.6.1.0-129/storm -Dworkers.artifacts=/var/log/storm/workers-artifacts -Dstorm.id=ap-bliu-topology-02-2-1503563256 -Dworker.id=c3edab81-9e97-425a-9a1e-acffe5298d5c -Dworker.port=6700 -Dstorm.log.dir=/var/log/storm -Dlog4j.configurationFile=/usr/hdp/2.6.1.0-129/storm/log4j2/worker.xml -DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector -Dstorm.local.dir=/hadoop/storm -Xmx768m -Djava.security.auth.login.config=/usr/hdp/current/storm-supervisor/conf/storm_jaas.conf -javaagent:/usr/hdp/current/storm-client/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar=host=localhost,port=8650,wireformat31x=true,mode=multicast,config=/usr/hdp/current/storm-client/contrib/storm-jmxetric/conf/jmxetric-conf.xml,process=Worker_6700_JVM -Djava.library.path=/hadoop/storm/supervisor/stormdist/ap-bliu-topology-02-2-1503563256/resources/Linux-amd64:/hadoop/storm/supervisor/stormdist/ap-bliu-topology-02-2-1503563256/resources:/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib -Dstorm.conf.file= -Dstorm.options= -Djava.io.tmpdir=/hadoop/storm/workers/c3edab81-9e97-425a-9a1e-acffe5298d5c/tmp -cp /usr/hdp/2.6.1.0-129/storm/lib/ambari-metrics-storm-sink.jar:/usr/hdp/2.6.1.0-129/storm/lib/ring-cors-0.1.5.jar:/usr/hdp/2.6.1.0-129/storm/lib/storm-rename-hack-1.1.0.2.6.1.0-129.jar:/usr/hdp/2.6.1.0-129/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.6.1.0-129/storm/lib/asm-5.0.3.jar:/usr/hdp/2.6.1.0-129/storm/lib/log4j-api-2.8.2.jar:/usr/hdp/2.6.1.0-129/storm/lib/log4j-core-2.8.2.jar:/usr/hdp/2.6.1.0-129/storm/lib/zookeeper.jar:/usr/hdp/2.6.1.0-129/storm/lib/disruptor-3.3.2.jar:/usr/hdp/2.6.1.0-129/storm/lib/log4j-slf4j-impl-2.8.2.jar:/usr/hdp/2.6.1.0-129/storm/lib/reflectasm-1.10.1.jar:/usr/hdp/2.6.1.0-129/storm/lib/objenesis-2.1.jar:/usr/hdp/2.6.1.0-129/storm/lib/kryo-3.0.3.jar:/usr/hdp/2.6.1.0-129/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.6.1.0-129/storm/lib/clojure-1.7.0.jar:/usr/hdp/2.6.1.0-129/storm/lib/minlog-1.3.0.jar:/usr/hdp/2.6.1.0-129/storm/lib/storm-core-1.1.0.2.6.1.0-129.jar:/usr/hdp/2.6.1.0-129/storm/lib/slf4j-api-1.7.21.jar:/usr/hdp/current/storm-supervisor/conf:/hadoop/storm/supervisor/stormdist/ap-bliu-topology-02-2-1503563256/stormjar.jar org.apache.storm.daemon.worker ap-bliu-topology-02-2-1503563256 c5e355da-3b6f-4c22-84a4-b6524aa037f0 6700 c3edab81-9e97-425a-9a1e-acffe5298d5c
ap-bliu  30268 32.0 11.2 3366668 454348 ?      Sl   17:27  35:56 /usr/jdk64/jdk1.8.0_112/bin/java -server -Dlogging.sensitivity=S3 -Dlogfile.name=worker.log -Dstorm.home=/usr/hdp/2.6.1.0-129/storm -Dworkers.artifacts=/var/log/storm/workers-artifacts -Dstorm.id=ap-bliu-topology-02-2-1503563256 -Dworker.id=44528bbd-2eb3-43d6-9444-30e8857c72f6 -Dworker.port=6701 -Dstorm.log.dir=/var/log/storm -Dlog4j.configurationFile=/usr/hdp/2.6.1.0-129/storm/log4j2/worker.xml -DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector -Dstorm.local.dir=/hadoop/storm -Xmx768m -Djava.security.auth.login.config=/usr/hdp/current/storm-supervisor/conf/storm_jaas.conf -javaagent:/usr/hdp/current/storm-client/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar=host=localhost,port=8650,wireformat31x=true,mode=multicast,config=/usr/hdp/current/storm-client/contrib/storm-jmxetric/conf/jmxetric-conf.xml,process=Worker_6701_JVM -Djava.library.path=/hadoop/storm/supervisor/stormdist/ap-bliu-topology-02-2-1503563256/resources/Linux-amd64:/hadoop/storm/supervisor/stormdist/ap-bliu-topology-02-2-1503563256/resources:/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib -Dstorm.conf.file= -Dstorm.options= -Djava.io.tmpdir=/hadoop/storm/workers/44528bbd-2eb3-43d6-9444-30e8857c72f6/tmp -cp /usr/hdp/2.6.1.0-129/storm/lib/ambari-metrics-storm-sink.jar:/usr/hdp/2.6.1.0-129/storm/lib/ring-cors-0.1.5.jar:/usr/hdp/2.6.1.0-129/storm/lib/storm-rename-hack-1.1.0.2.6.1.0-129.jar:/usr/hdp/2.6.1.0-129/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.6.1.0-129/storm/lib/asm-5.0.3.jar:/usr/hdp/2.6.1.0-129/storm/lib/log4j-api-2.8.2.jar:/usr/hdp/2.6.1.0-129/storm/lib/log4j-core-2.8.2.jar:/usr/hdp/2.6.1.0-129/storm/lib/zookeeper.jar:/usr/hdp/2.6.1.0-129/storm/lib/disruptor-3.3.2.jar:/usr/hdp/2.6.1.0-129/storm/lib/log4j-slf4j-impl-2.8.2.jar:/usr/hdp/2.6.1.0-129/storm/lib/reflectasm-1.10.1.jar:/usr/hdp/2.6.1.0-129/storm/lib/objenesis-2.1.jar:/usr/hdp/2.6.1.0-129/storm/lib/kryo-3.0.3.jar:/usr/hdp/2.6.1.0-129/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.6.1.0-129/storm/lib/clojure-1.7.0.jar:/usr/hdp/2.6.1.0-129/storm/lib/minlog-1.3.0.jar:/usr/hdp/2.6.1.0-129/storm/lib/storm-core-1.1.0.2.6.1.0-129.jar:/usr/hdp/2.6.1.0-129/storm/lib/slf4j-api-1.7.21.jar:/usr/hdp/current/storm-supervisor/conf:/hadoop/storm/supervisor/stormdist/ap-bliu-topology-02-2-1503563256/stormjar.jar org.apache.storm.daemon.worker ap-bliu-topology-02-2-1503563256 c5e355da-3b6f-4c22-84a4-b6524aa037f0 6701 44528bbd-2eb3-43d6-9444-30e8857c72f6

Is there anyone can tell me why and how to use CGROUP with RAS on storm?

Other info

-bash-4.1$ cat /etc/redhat-release
CentOS release 6.8 (Final)

-bash-4.1$ hdp-select versions
2.6.1.0-129

-bash-4.1$ storm version
/usr/bin/storm: line 2: /usr/hdp/2.6.1.0-129/etc/default/hadoop: No such file or directory
Running: /usr/jdk64/jdk1.8.0_112/bin/java -Ddaemon.name= -Dstorm.options= -Dstorm.home=/usr/hdp/2.6.1.0-129/storm -Dstorm.log.dir=/var/log/storm -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib -Dstorm.conf.file= -cp /usr/hdp/2.6.1.0-129/storm/lib/ambari-metrics-storm-sink.jar:/usr/hdp/2.6.1.0-129/storm/lib/ring-cors-0.1.5.jar:/usr/hdp/2.6.1.0-129/storm/lib/storm-rename-hack-1.1.0.2.6.1.0-129.jar:/usr/hdp/2.6.1.0-129/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.6.1.0-129/storm/lib/asm-5.0.3.jar:/usr/hdp/2.6.1.0-129/storm/lib/log4j-api-2.8.2.jar:/usr/hdp/2.6.1.0-129/storm/lib/log4j-core-2.8.2.jar:/usr/hdp/2.6.1.0-129/storm/lib/zookeeper.jar:/usr/hdp/2.6.1.0-129/storm/lib/disruptor-3.3.2.jar:/usr/hdp/2.6.1.0-129/storm/lib/log4j-slf4j-impl-2.8.2.jar:/usr/hdp/2.6.1.0-129/storm/lib/reflectasm-1.10.1.jar:/usr/hdp/2.6.1.0-129/storm/lib/objenesis-2.1.jar:/usr/hdp/2.6.1.0-129/storm/lib/kryo-3.0.3.jar:/usr/hdp/2.6.1.0-129/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.6.1.0-129/storm/lib/clojure-1.7.0.jar:/usr/hdp/2.6.1.0-129/storm/lib/minlog-1.3.0.jar:/usr/hdp/2.6.1.0-129/storm/lib/storm-core-1.1.0.2.6.1.0-129.jar:/usr/hdp/2.6.1.0-129/storm/lib/slf4j-api-1.7.21.jar:/usr/hdp/2.6.1.0-129/storm/extlib-daemon/ojdbc6.jar:/usr/hdp/2.6.1.0-129/storm/extlib-daemon/ranger-storm-plugin-shim-0.7.0.2.6.1.0-129.jar:/usr/hdp/2.6.1.0-129/storm/extlib-daemon/ranger-plugin-classloader-0.7.0.2.6.1.0-129.jar:/usr/hdp/current/storm-supervisor/conf org.apache.storm.utils.VersionInfo
Storm 1.1.0.2.6.1.0-129

Don't have an account?