how to determine java heep size on kafka machines

[#1]

we have Hadoop cluster ( version 2.6.4 ) with 3 physical kafka machines

we want to know what is the values of Xmx and Xms that we need to allocate on the kafka machines

in order to setup the in the Xmx and Xms on kafka machine we need to configure the script

/usr/hdp/2.6.4 /kafka/bin/kafka-server-start

for now the default values are ( Xmx is 1G and Xms is 1G )

if [ "x$KAFKA_HEAP_OPTS" = "x" ]; then
export
KAFKA_HEAP_OPTS="-Xmx1G –Xms1G"
fi

[root@kafka01 ~]# free -g
              total        used        free      shared  buff/cache   available
Mem:            251          17          11           0         222         233
Swap:            15           6           9

on each kafka we have 256G

and we found this link - https://stackoverflow.com/questions/4667483/how-is-the-default-java-heap-size-determined

according to the link in stackoverflow we can use this formula:

[root@kafka01 ~]# java -XX:+PrintFlagsFinal -version | grep HeapSize

    uintx ErgoHeapSizeLimit                         = 0                                                                                                                {product}

    uintx HeapSizePerGCThread                       = 87241520                                                                                                         {product}

    uintx InitialHeapSize                          := 2147483648                                                                                                       {product}

    uintx LargePageHeapSizeThreshold                = 134217728                                                                                                        {product}

    uintx MaxHeapSize                              := 32210157568                                                                                                      {product}

openjdk version "1.8.0_65"

OpenJDK Runtime Environment (build 1.8.0_65-b17)

OpenJDK 64-Bit Server VM (build 25.65-b01, mixed mode)

[root@kafka01 ~]# java -XX:+PrintFlagsFinal -version | grep HeapSize

    uintx ErgoHeapSizeLimit                         = 0                                   {product}

    uintx HeapSizePerGCThread                       = 87241520                            {product}

    uintx InitialHeapSize                          := 2147483648                          {product}

    uintx LargePageHeapSizeThreshold                = 134217728                           {product}

    uintx MaxHeapSize                              := 32210157568                         {product}

openjdk version "1.8.0_65"

OpenJDK Runtime Environment (build 1.8.0_65-b17)

OpenJDK 64-Bit Server VM (build 25.65-b01, mixed mode)

So the values comes with bytes and we can calculate them to Giga

So

Xmx = 32G

Xms = 2G

What is the recommendation of hortonworks about Xmx and Xms ?

Dose hortonworks except the formula - java -XX:+PrintFlagsFinal -version | grep HeapSize

In order to get the right values of Xmx and Xms ?

[#2]

@Michael Bronson

please check your environment variables if somewhere you are setting "" variable is defined twice of the script is being invoked 3 times.

But practically it will not harm. Because duplicate JVM flags are allowed .. only the last occurrance of the duplicate value takes precedence.

However please try this to fix this: remove the "$KAFKA_HEAP_OPTS" from the previously mentioned line. thenr estart broker.

# Set KAFKA specific environment variables here.
export KAFKA_HEAP_OPTS="-Xms3g -Xmx3g"

.

[#3]

@jay

according to the article ( https://community.hortonworks.com/articles/80813/kafka-best-practices-2.html ) , I see

Kafka Broker:

Java Version

We recommend latest java 1.8 with G1 collector ( which is default in new version). If you are using Java 1.7 and G1 collector make sure you are on u51 or higher.

A recommended setting for JVM looks like following

-Xmx8g -Xms8g -XX:MetaspaceSize=96m -XX:+UseG1GC-XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:G1HeapRegionSize=16M-XX:MinMetaspaceFreeRatio=50 -XX:MaxMetaspaceFreeRatio=80

lets say that I update the new "-Xmx8g -Xms8g"

in the script - /usr/hdp/2.6.4 /kafka/bin/kafka-server-start on each kafka

then where I need to add/change other values - "-XX:MetaspaceSize=96m -XX:+UseG1GC-XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:G1HeapRegionSize=16M-XX:MinMetaspaceFreeRatio=50 -XX:MaxMetaspaceFreeRatio=80"  ?

[#4]

@Jay

now I get the right values from ps -ef | grep kafka

kafka     74086      1 99 12:53 ?        00:00:15 /usr/jdk64/jdk1.8.0_112/bin/java -Xms3g -Xmx3g -server

now is it possible to set the value

 export KAFKA_HEAP_OPTS="-Xms3g -Xmx3g" 

by API ?

* we want to automate this process by script bash