Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How do you change Kafka heap memory with CDH (properly) ?

Solved Go to solution
Highlighted

How do you change Kafka heap memory with CDH (properly) ?

Explorer

 

ps aux | grep -i kafka
kafka 3376 12.3 19.4 5444664 1184292 ? Sl Jun17 4843:42 /usr/lib/jvm/java-7-oracle-cloudera/bin/java -Xmx1G -Xms1G -server -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSScavengeBeforeRemark -XX:+DisableExplicitGC -Djava.awt.headless=true -Xloggc:/run/cloudera-scm-agent/process/2769-kafka-KAFKA_BROKER/kafkaServer-gc.log -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=9393 -Dkafka.logs.dir=/run/cloudera-scm-agent/process/2769-kafka-KAFKA_BROKER -Dlog4j.configuration=file:/run/cloudera-scm-agent/process/2769-kafka-KAFKA_BROKER/log4j.properties -cp :/opt/cloudera/parcels/KAFKA-0.8.2.0-1.kafka1.3.0.p0.29/lib/kafka/bin/../core/build/dependant-libs-2.10.4*/*.jar:/opt/cloudera/parcels/KAFKA-0.8.2.0-1.kafka1.3.0.p0.29/lib/kafka/bin/../examples/build/libs//kafka-examples*.jar:/opt/cloudera/parcels/KAFKA-0.8.2.0-1.kafka1.3.0.p0.29/lib/kafka/bin/../contrib/hadoop-consumer/build/libs//kafka-hadoop-consumer*.jar:/opt/cloudera/parcels/KAFKA-0.8.2.0-1.kafka1.3.0.p0.29/lib/kafka/bin/../contrib/hadoop-producer/build/libs//kafka-hadoop-producer*.jar:/......etc

 

how do change that java heap memory for kafka in the proper way with CDH? (because i didn't see it on the cdh config) (CDH 5.4.x)

 

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: How do you change Kafka heap memory with CDH (properly) ?

Master Guru
Edit the "Kafka Broker Environment Advanced Configuration Snippet (Safety Valve)" field under CM -> Kafka -> Configuration with the below entry (For a 5g example):

KAFKA_HEAP_OPTS="-Xmx5g -Xms5g"

Save and restart.

View solution in original post

2 REPLIES 2
Highlighted

Re: How do you change Kafka heap memory with CDH (properly) ?

Master Guru
Edit the "Kafka Broker Environment Advanced Configuration Snippet (Safety Valve)" field under CM -> Kafka -> Configuration with the below entry (For a 5g example):

KAFKA_HEAP_OPTS="-Xmx5g -Xms5g"

Save and restart.

View solution in original post

Highlighted

Re: How do you change Kafka heap memory with CDH (properly)

Master Guru
A direct property to change this is now available in the most recent CM
5.4.x releases. Once you upgrade, please remember to remove the env-var
workaround so things continue to be configured consistently.
Don't have an account?
Coming from Hortonworks? Activate your account here