Kafka Machine not releasing memory from buffrt

[#1]

Hello,

We have cloudera 7.4.4 installed on customer network with 8 Kafka broker node. When the application get used heavily the memory of all Kafka node get exhausted(more that 98%) because of the messages get stuck.  

We have heap size of 4GB, 8 CPU, 32GB memory on all Kafka nodes.

$ ps -ef| grep kafka
kafka 508659 16233 0 May29 ? 00:00:00 /usr/bin/python2 /opt/cloudera/cm-agent/bin/cm proc_watcher 508665
kafka 508665 508659 15 May29 ? 1-09:22:48 /usr/lib/jvm/jre-openjdk/bin/java -Xmx4096M -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/opt/amdocs/external/cloudera/kafka/heap/kafka_kafka-KAFKA_BROKER-6bc6dfe85482761059e0f8085a79733f_pid508665.hprof -XX:OnOutOfMemoryError=/opt/cloudera/cm-agent/service/common/killparent.sh -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:G1HeapRegionSize=16M -XX:MinMetaspaceFreeRatio=50 -XX:MaxMetaspaceFreeRatio=80 -XX:+DisableExplicitGC -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.host=127.0.0.1 -Dcom.sun.management.jmxremote.local.only=true -Djava.rmi.server.hostname=127.0.0.1 -javaagent:/opt/amdocs/external/ms360-prometheus-jmx-exporter/jmx_prometheus_javaagent-0.16.1.jar=7083:/opt/amdocs/external/jmx-exporter/config.yaml -Xms4096M -XX:+UnlockExperimentalVMOptions -XX:G1NewSizePercent=25 -Dsun.security.krb5.disableReferrals=true -Djdk.tls.ephemeralDHKeySize=2048 -Xloggc:/run/cloudera-scm-agent/process/1546342550-kafka-KAFKA_BROKER/kafkaServer-gc.log -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=100M -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=9393 -Dkafka.logs.dir=/run/cloudera-scm-agent/process/1546342550-kafka-KAFKA_BROKER -Dlog4j.configuration=file:/var/run/cloudera-scm-agent/process/1546342550-kafka-KAFKA_BROKER/log4j.properties -cp /opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/activation-1.1.1.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/aopalliance-repackaged-2.6.1.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/argparse4j-0.7.0.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/atlas-kafka-plugin-impl:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/atlas-plugin-classloader-2.1.0.7.1.7.0-551.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/audience-annotations-0.5.0.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/caffeine-2.8.0.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/checker-qual-2.10.0.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/commons-beanutils-1.9.4.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/commons-cli-1.4.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/commons-collections-3.2.2.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/commons-lang3-3.8.1.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/connect-api-2.5.0.7.1.7.0-551.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/connect-basic-auth-extension-2.5.0.7.1.7.0-551.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/connect-file-2.5.0.7.1.7.0-551.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/connect-json-2.5.0.7.1.7.0-551.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/connect-mirror-2.5.0.7.1.7.0-551.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/connect-mirror-client-2.5.0.7.1.7.0-551.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/connect-runtime-2.5.0.7.1.7.0-551.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/connect-transforms-2.5.0.7.1.7.0-551.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/error_prone_annotations-2.3.3.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/hk2-api-2.6.1.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/hk2-locator-2.6.1.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/hk2-utils-2.6.1.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/jackson-annotations-2.10.2.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/jackson-core-2.10.2.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/jackson-databind-2.10.2.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/jackson-dataformat-csv-2.10.2.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/jackson-datatype-jdk8-2.10.2.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/jackson-jaxrs-base-2.10.2.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/jackson-jaxrs-json-provider-2.10.2.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/jackson-module-jaxb-annotations-2.10.2.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/jackson-module-paranamer-2.10.2.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/jackson-module-scala_2.12-2.10.2.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/jakarta.activation-api-1.2.1.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/jakarta.annotation-api-1.3.5.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/jakarta.inject-2.6.1.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/jakarta.validation-api-2.0.2.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/jakarta.ws.rs-api-2.1.6.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/jakarta.xml.bind-api-2.3.2.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/javassist-3.25.0-GA.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/javassist-3.26.0-GA.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/javax.servlet-api-3.1.0.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/javax.ws.rs-api-2.1.1.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/jaxb-api-2.3.0.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/jersey-client-2.32.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/jersey-common-2.32.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/jersey-container-servlet-2.32.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/jersey-container-servlet-core-2.32.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/jersey-hk2-2.32.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/jersey-media-jaxb-2.32.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/jersey-server-2.32.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/jetty-client-9.4.39.v20210325.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/jetty-continuation-9.4.39.v20210325.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/jetty-http-9.4.39.v20210325.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/jetty-io-9.4.39.v20210325.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/jetty-security-9.4.39.v20210325.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/jetty-server-9.4.39.v20210325.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/jetty-servlet-9.4.39.v20210325.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/jetty-servlets-9.4.39.v20210325.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/jetty-util-9.4.39.v20210325.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/jetty-util-ajax-9.4.39.v20210325.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/jna-4.5.2.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/jopt-simple-5.0.4.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/kafka_2.12-2.5.0.7.1.7.0-551.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/kafka_2.12-2.5.0.7.1.7.0-551-sources.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/kafka-bridge-shim-2.1.0.7.1.7.0-551.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/kafka-clients-2.5.0.7.1.7.0-551.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/kafka-cloudera-plugins-2.5.0.7.1.7.0-551.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/kafka-log4j-appender-2.5.0.7.1.7.0-551.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/kafka-streams-2.5.0.7.1.7.0-551.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/kafka-streams-examples-2.5.0.7.1.7.0-551.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/kafka-streams-scala_2.12-2.5.0.7.1.7.0-551.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/kafka-streams-test-utils-2.5.0.7.1.7.0-551.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/kafka-tools-2.5.0.7.1.7.0-551.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/libpam4j-1.11.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/log4j-1.2.17-cloudera1.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/logredactor-2.0.8.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/lz4-java-1.7.1.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/maven-artifact-3.6.3.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/metrics-core-2.2.0.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/metrics-servlet-2.2.0.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/netty-all-4.1.60.Final.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/osgi-resource-locator-1.0.3.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/paranamer-2.8.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/plexus-utils-3.2.1.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/ranger-kafka-plugin-impl:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/ranger-kafka-plugin-shim-2.1.0.7.1.7.0-551.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/ranger-plugin-classloader-2.1.0.7.1.7.0-551.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/reflections-0.9.12.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/rocksdbjni-5.18.3.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/scala-collection-compat_2.12-2.1.3.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/scala-java8-compat_2.12-0.9.0.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/scala-library-2.12.10.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/scala-logging_2.12-3.9.2.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/scala-reflect-2.12.10.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/shiro-cache-1.7.1.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/shiro-config-core-1.7.1.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/shiro-config-ogdl-1.7.1.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/shiro-core-1.7.1.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/shiro-crypto-cipher-1.7.1.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/shiro-crypto-core-1.7.1.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/shiro-crypto-hash-1.7.1.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/shiro-event-1.7.1.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/shiro-lang-1.7.1.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/simpleclient-0.9.0.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/simpleclient_common-0.9.0.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/simpleclient_hotspot-0.9.0.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/simpleclient_httpserver-0.9.0.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/simpleclient_servlet-0.9.0.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/slf4j-api-1.7.30.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/slf4j-log4j12-1.7.30.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/snappy-java-1.1.7.3.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/zookeeper-3.5.5.7.1.7.0-551.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/zookeeper-jute-3.5.5.7.1.7.0-551.jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/kafka/bin/../libs/zstd-jni-1.4.4-7.jar -Dsun.security.krb5.disableReferrals=true -Djdk.tls.ephemeralDHKeySize=2048 com.cloudera.kafka.wrap.Kafka /var/run/cloudera-scm-agent/process/1546342550-kafka-KAFKA_BROKER/kafka.properties

Does any variable available in cloudera to flush the memory?

[#2]

@AkshayAmdocs Welcome to the Cloudera Community!

To help you get the best possible solution, I have tagged our Kafka experts @rki_ @ChethanYM @paras   who may be able to assist you further.

Please keep us updated on your post, and we hope you find a satisfactory solution to your query.

[#3]

Thanks @DianaTorres . Let me know if anything needed from my side.

[#4]

Hi @rki_ , @ChethanYM , @paras ,

Hope you are doing well!

Could you please help us with above issue?

Thanks,

Akshay