Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to use CGROUP in kerberized storm cluster?

How to use CGROUP in kerberized storm cluster?

New Contributor

I want to use CGROUP to limit users's use of cpu or memory in my kerberized storm cluster.

According to the storm document, I enabled CGROUP via ambari with

storm.cgroup.enable=true

and modified /etc/cgconfig.conf file as below

// on supervisor server
-bash-4.1$ id storm
uid=55696(storm) gid=10899(hadoop) groups=10899(hadoop)
-bash-4.1$ cat /etc/cgconfig.conf
#
#  Copyright IBM Corporation. 2007
#
#  Authors:	Balbir Singh <balbir@linux.vnet.ibm.com>
#  This program is free software; you can redistribute it and/or modify it
#  under the terms of version 2.1 of the GNU Lesser General Public License
#  as published by the Free Software Foundation.
#
#  This program is distributed in the hope that it would be useful, but
#  WITHOUT ANY WARRANTY; without even the implied warranty of
#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
#
# See man cgconfig.conf for further details.
#
# By default, mount all controllers to /cgroup/<controller>


mount {
    cpuset  = /cgroup/cpuset;
    cpu = /cgroup/storm_resources;
    cpuacct = /cgroup/cpuacct;
    memory  = /cgroup/storm_resources;
    devices = /cgroup/devices;
    freezer = /cgroup/freezer;
    net_cls = /cgroup/net_cls;
    blkio   = /cgroup/blkio;
}


group storm {
    perm {
        task {
       	    uid = 55696;
            gid = 10899;
        }
        admin {
            uid = 55696;
            gid = 10899;
        }
    }
    cpu {
    }
}

Because I also use Resource Aware Scheduler(RAS) and defined users's use of cpu and memory as

// on nimbus server
-bash-4.1$ cat /etc/storm/conf/user-resource-pools.yaml
resource.aware.scheduler.user.pools:
    ap-bliu:
        cpu: 10
        memory: 896

and the configs for RAS are

topology.component.resources.onheap.memory.mb=64
topology.component.resources.offheap.memory.mb=0
topology.component.cpu.pcore.percent=3
topology.worker.max.heap.size.mb=768

so when user ap-bliu submits a new topology running with 2 workers(slots) and 6 executors on each worker, I expected to confirm the use of cpu of each worker should below 18 (3*6=18)

well, I got

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
30268 ap-bliu   20   0 3287m 443m  19m S 41.8 11.2  33:05.40 java
30264 ap-bliu   20   0 3297m 450m  19m S 36.2 11.4  30:19.55 java
30238 ap-bliu   20   0 2488m  80m  19m S  0.0  2.0   0:13.19 java
30240 ap-bliu   20   0 2488m  84m  19m S  0.0  2.1   0:11.44 java

and I even could not found cgexec in the command for worker starting

-bash-4.1$ ps aux | grep worker | grep -v LogWriter | grep -v grep
ap-bliu  30264 29.2 11.3 3376364 461664 ?      Sl   17:27  32:50 /usr/jdk64/jdk1.8.0_112/bin/java -server -Dlogging.sensitivity=S3 -Dlogfile.name=worker.log -Dstorm.home=/usr/hdp/2.6.1.0-129/storm -Dworkers.artifacts=/var/log/storm/workers-artifacts -Dstorm.id=ap-bliu-topology-02-2-1503563256 -Dworker.id=c3edab81-9e97-425a-9a1e-acffe5298d5c -Dworker.port=6700 -Dstorm.log.dir=/var/log/storm -Dlog4j.configurationFile=/usr/hdp/2.6.1.0-129/storm/log4j2/worker.xml -DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector -Dstorm.local.dir=/hadoop/storm -Xmx768m -Djava.security.auth.login.config=/usr/hdp/current/storm-supervisor/conf/storm_jaas.conf -javaagent:/usr/hdp/current/storm-client/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar=host=localhost,port=8650,wireformat31x=true,mode=multicast,config=/usr/hdp/current/storm-client/contrib/storm-jmxetric/conf/jmxetric-conf.xml,process=Worker_6700_JVM -Djava.library.path=/hadoop/storm/supervisor/stormdist/ap-bliu-topology-02-2-1503563256/resources/Linux-amd64:/hadoop/storm/supervisor/stormdist/ap-bliu-topology-02-2-1503563256/resources:/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib -Dstorm.conf.file= -Dstorm.options= -Djava.io.tmpdir=/hadoop/storm/workers/c3edab81-9e97-425a-9a1e-acffe5298d5c/tmp -cp /usr/hdp/2.6.1.0-129/storm/lib/ambari-metrics-storm-sink.jar:/usr/hdp/2.6.1.0-129/storm/lib/ring-cors-0.1.5.jar:/usr/hdp/2.6.1.0-129/storm/lib/storm-rename-hack-1.1.0.2.6.1.0-129.jar:/usr/hdp/2.6.1.0-129/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.6.1.0-129/storm/lib/asm-5.0.3.jar:/usr/hdp/2.6.1.0-129/storm/lib/log4j-api-2.8.2.jar:/usr/hdp/2.6.1.0-129/storm/lib/log4j-core-2.8.2.jar:/usr/hdp/2.6.1.0-129/storm/lib/zookeeper.jar:/usr/hdp/2.6.1.0-129/storm/lib/disruptor-3.3.2.jar:/usr/hdp/2.6.1.0-129/storm/lib/log4j-slf4j-impl-2.8.2.jar:/usr/hdp/2.6.1.0-129/storm/lib/reflectasm-1.10.1.jar:/usr/hdp/2.6.1.0-129/storm/lib/objenesis-2.1.jar:/usr/hdp/2.6.1.0-129/storm/lib/kryo-3.0.3.jar:/usr/hdp/2.6.1.0-129/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.6.1.0-129/storm/lib/clojure-1.7.0.jar:/usr/hdp/2.6.1.0-129/storm/lib/minlog-1.3.0.jar:/usr/hdp/2.6.1.0-129/storm/lib/storm-core-1.1.0.2.6.1.0-129.jar:/usr/hdp/2.6.1.0-129/storm/lib/slf4j-api-1.7.21.jar:/usr/hdp/current/storm-supervisor/conf:/hadoop/storm/supervisor/stormdist/ap-bliu-topology-02-2-1503563256/stormjar.jar org.apache.storm.daemon.worker ap-bliu-topology-02-2-1503563256 c5e355da-3b6f-4c22-84a4-b6524aa037f0 6700 c3edab81-9e97-425a-9a1e-acffe5298d5c
ap-bliu  30268 32.0 11.2 3366668 454348 ?      Sl   17:27  35:56 /usr/jdk64/jdk1.8.0_112/bin/java -server -Dlogging.sensitivity=S3 -Dlogfile.name=worker.log -Dstorm.home=/usr/hdp/2.6.1.0-129/storm -Dworkers.artifacts=/var/log/storm/workers-artifacts -Dstorm.id=ap-bliu-topology-02-2-1503563256 -Dworker.id=44528bbd-2eb3-43d6-9444-30e8857c72f6 -Dworker.port=6701 -Dstorm.log.dir=/var/log/storm -Dlog4j.configurationFile=/usr/hdp/2.6.1.0-129/storm/log4j2/worker.xml -DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector -Dstorm.local.dir=/hadoop/storm -Xmx768m -Djava.security.auth.login.config=/usr/hdp/current/storm-supervisor/conf/storm_jaas.conf -javaagent:/usr/hdp/current/storm-client/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar=host=localhost,port=8650,wireformat31x=true,mode=multicast,config=/usr/hdp/current/storm-client/contrib/storm-jmxetric/conf/jmxetric-conf.xml,process=Worker_6701_JVM -Djava.library.path=/hadoop/storm/supervisor/stormdist/ap-bliu-topology-02-2-1503563256/resources/Linux-amd64:/hadoop/storm/supervisor/stormdist/ap-bliu-topology-02-2-1503563256/resources:/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib -Dstorm.conf.file= -Dstorm.options= -Djava.io.tmpdir=/hadoop/storm/workers/44528bbd-2eb3-43d6-9444-30e8857c72f6/tmp -cp /usr/hdp/2.6.1.0-129/storm/lib/ambari-metrics-storm-sink.jar:/usr/hdp/2.6.1.0-129/storm/lib/ring-cors-0.1.5.jar:/usr/hdp/2.6.1.0-129/storm/lib/storm-rename-hack-1.1.0.2.6.1.0-129.jar:/usr/hdp/2.6.1.0-129/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.6.1.0-129/storm/lib/asm-5.0.3.jar:/usr/hdp/2.6.1.0-129/storm/lib/log4j-api-2.8.2.jar:/usr/hdp/2.6.1.0-129/storm/lib/log4j-core-2.8.2.jar:/usr/hdp/2.6.1.0-129/storm/lib/zookeeper.jar:/usr/hdp/2.6.1.0-129/storm/lib/disruptor-3.3.2.jar:/usr/hdp/2.6.1.0-129/storm/lib/log4j-slf4j-impl-2.8.2.jar:/usr/hdp/2.6.1.0-129/storm/lib/reflectasm-1.10.1.jar:/usr/hdp/2.6.1.0-129/storm/lib/objenesis-2.1.jar:/usr/hdp/2.6.1.0-129/storm/lib/kryo-3.0.3.jar:/usr/hdp/2.6.1.0-129/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.6.1.0-129/storm/lib/clojure-1.7.0.jar:/usr/hdp/2.6.1.0-129/storm/lib/minlog-1.3.0.jar:/usr/hdp/2.6.1.0-129/storm/lib/storm-core-1.1.0.2.6.1.0-129.jar:/usr/hdp/2.6.1.0-129/storm/lib/slf4j-api-1.7.21.jar:/usr/hdp/current/storm-supervisor/conf:/hadoop/storm/supervisor/stormdist/ap-bliu-topology-02-2-1503563256/stormjar.jar org.apache.storm.daemon.worker ap-bliu-topology-02-2-1503563256 c5e355da-3b6f-4c22-84a4-b6524aa037f0 6701 44528bbd-2eb3-43d6-9444-30e8857c72f6

Is there anyone can tell me why and how to use CGROUP with RAS on storm?

Other info

-bash-4.1$ cat /etc/redhat-release
CentOS release 6.8 (Final)

-bash-4.1$ hdp-select versions
2.6.1.0-129

-bash-4.1$ storm version
/usr/bin/storm: line 2: /usr/hdp/2.6.1.0-129/etc/default/hadoop: No such file or directory
Running: /usr/jdk64/jdk1.8.0_112/bin/java -Ddaemon.name= -Dstorm.options= -Dstorm.home=/usr/hdp/2.6.1.0-129/storm -Dstorm.log.dir=/var/log/storm -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib -Dstorm.conf.file= -cp /usr/hdp/2.6.1.0-129/storm/lib/ambari-metrics-storm-sink.jar:/usr/hdp/2.6.1.0-129/storm/lib/ring-cors-0.1.5.jar:/usr/hdp/2.6.1.0-129/storm/lib/storm-rename-hack-1.1.0.2.6.1.0-129.jar:/usr/hdp/2.6.1.0-129/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.6.1.0-129/storm/lib/asm-5.0.3.jar:/usr/hdp/2.6.1.0-129/storm/lib/log4j-api-2.8.2.jar:/usr/hdp/2.6.1.0-129/storm/lib/log4j-core-2.8.2.jar:/usr/hdp/2.6.1.0-129/storm/lib/zookeeper.jar:/usr/hdp/2.6.1.0-129/storm/lib/disruptor-3.3.2.jar:/usr/hdp/2.6.1.0-129/storm/lib/log4j-slf4j-impl-2.8.2.jar:/usr/hdp/2.6.1.0-129/storm/lib/reflectasm-1.10.1.jar:/usr/hdp/2.6.1.0-129/storm/lib/objenesis-2.1.jar:/usr/hdp/2.6.1.0-129/storm/lib/kryo-3.0.3.jar:/usr/hdp/2.6.1.0-129/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.6.1.0-129/storm/lib/clojure-1.7.0.jar:/usr/hdp/2.6.1.0-129/storm/lib/minlog-1.3.0.jar:/usr/hdp/2.6.1.0-129/storm/lib/storm-core-1.1.0.2.6.1.0-129.jar:/usr/hdp/2.6.1.0-129/storm/lib/slf4j-api-1.7.21.jar:/usr/hdp/2.6.1.0-129/storm/extlib-daemon/ojdbc6.jar:/usr/hdp/2.6.1.0-129/storm/extlib-daemon/ranger-storm-plugin-shim-0.7.0.2.6.1.0-129.jar:/usr/hdp/2.6.1.0-129/storm/extlib-daemon/ranger-plugin-classloader-0.7.0.2.6.1.0-129.jar:/usr/hdp/current/storm-supervisor/conf org.apache.storm.utils.VersionInfo
Storm 1.1.0.2.6.1.0-129

Don't have an account?
Coming from Hortonworks? Activate your account here