@Aditya thank you so much for the great answer , I am really appropriate this , about the "-XX:MetaspaceSize=96m -XX:+UseG1GC-XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:G1HeapRegionSize=16M-XX:MinMetaspaceFreeRatio=50 -XX:MaxMetaspaceFreeRatio=80" , where we can put thiose values , in which variables? ... View more

Did you check the repo path on ambari GUI? , if not created by API and you not see any repo PATH on ambari GUI , you can create the repo manual on ambari GUI , go to admin --> stack and version --> versions --> manage version and click on the version link then you get repo window ... View more

@Jay now I get the right values from ps -ef | grep kafka kafka 74086 1 99 12:53 ? 00:00:15 /usr/jdk64/jdk1.8.0_112/bin/java -Xms3g -Xmx3g -server now is it possible to set the value export KAFKA_HEAP_OPTS="-Xms3g -Xmx3g" by API ? * we want to automate this process by script bash ... View more

@jay according to the article ( https://community.hortonworks.com/articles/80813/kafka-best-practices-2.html ) , I see Kafka Broker: Java Version We recommend latest java 1.8 with G1 collector ( which is default in new version). If you are using Java 1.7 and G1 collector make sure you are on u51 or higher. A recommended setting for JVM looks like following -Xmx8g -Xms8g -XX:MetaspaceSize=96m -XX:+UseG1GC-XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:G1HeapRegionSize=16M-XX:MinMetaspaceFreeRatio=50 -XX:MaxMetaspaceFreeRatio=80 lets say that I update the new "-Xmx8g -Xms8g" in the script - /usr/hdp/2.6.4 /kafka/bin/kafka-server-start on each kafka then where I need to add/change other values - "-XX:MetaspaceSize=96m -XX:+UseG1GC-XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:G1HeapRegionSize=16M-XX:MinMetaspaceFreeRatio=50 -XX:MaxMetaspaceFreeRatio=80" ? ... View more

@Jay after I update the lines in ambari I get: ps -ef | grep kafka<br>kafka 58614 1 99 11:49 ? 00:00:44 /usr/jdk64/jdk1.8.0_112/bin/java -Xms3g -Xmx3g -Xms3g -Xmx3g -Xms3g -Xmx3g instead to get: ps -ef | grep kafka<br>kafka 58614 1 99 11:49 ? 00:00:44 /usr/jdk64/jdk1.8.0_112/bin/java -Xms3g -Xmx3g ... View more

@Jay why not to update the script - /usr/hdp/2.6.4 /kafka/bin/kafka-server-start on each kafka ? and change there the Xmx and Xms insted to add the new Xmx and Xms inside the ambari in kafka-env template ... View more

@Jay I ask all these questions because seems that the default of 1G inst enough , and we have in the past critical errors as the following from kafka.err ( this was very big problem and causes restarting of kafka broker ) Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "kafka-network-thread-1002-PLAINTEXT-2" Exception in thread "ExpirationReaper-1002" Exception in thread "ExpirationReaper-1002" java.lang.OutOfMemoryError: Java heap space java.lang.OutOfMemoryError: Java heap space ... View more

$Jay , anyway what is your personal recommendation for Xmx and Xms values ? ( according to jafka memory size ) ... View more