question Re: kafka broker + recommendation for JVM config in Archives of Support Questions (Read Only)

kafka broker + recommendation for JVM config

mike_bronson7 — Fri, 25 May 2018 14:30:38 GMT

hi all

according to the article in - https://community.hortonworks.com/articles/80813/kafka-best-practices-1.html

We recommend latest java 1.8 with G1 collector ( which is default in new version). If you are using Java 1.7 and G1 collector make sure you are on u51 or higher.

A recommended setting for JVM looks like following

-Xmx8g -Xms8g -XX:MetaspaceSize=96m -XX:+UseG1GC-XX:MaxGCPauseMillis=20 - XX:InitiatingHeapOccupancyPercent=35 -XX:G1HeapRegionSize=16M-XX:MinMetaspaceFreeRatio=50 - XX:MaxMetaspaceFreeRatio=80

my question - in which variable we need to set the values -> "-Xmx8g -Xms8g -XX:MetaspaceSize=96m -XX:+UseG1GC-XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:G1HeapRegionSize=16M-XX:MinMetaspaceFreeRatio=50 -XX:MaxMetaspaceFreeRatio=80"

second - is it possible to set all recomended JVM by API ?

Re: kafka broker + recommendation for JVM config

asirna — Fri, 25 May 2018 17:10:13 GMT

@Michael Bronson,

The config has to be added in kafka-env template. Yes. This can be done through API. Please look at the similar question to do the change using API

https://community.hortonworks.com/questions/193769/how-to-add-variable-in-kafka-env-template-by-api.html

-Aditya

Re: kafka broker + recommendation for JVM config

mike_bronson7 — Fri, 25 May 2018 17:57:11 GMT

@Aditya thank you so much for the great answer , I am really appropriate this , about the "-XX:MetaspaceSize=96m -XX:+UseG1GC-XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:G1HeapRegionSize=16M-XX:MinMetaspaceFreeRatio=50 -XX:MaxMetaspaceFreeRatio=80" , where we can put thiose values , in which variables?

Re: kafka broker + recommendation for JVM config

mike_bronson7 — Fri, 25 May 2018 18:02:54 GMT

Or in simple words can we set all "-Xmx8g -Xms8g -XX:MetaspaceSize=96m -XX:+UseG1GC-XX:MaxGCPauseMillis=20 - XX:InitiatingHeapOccupancyPercent=35 -XX:G1HeapRegionSize=16M-XX:MinMetaspaceFreeRatio=50 - XX:MaxMetaspaceFreeRatio=80" in KAFKA_HEAP_OPTS ?

Re: kafka broker + recommendation for JVM config

mike_bronson7 — Fri, 25 May 2018 18:03:53 GMT

export KAFKA_HEAP_OPTS="-Xmx8g -Xms8g -XX:MetaspaceSize=96m -XX:+UseG1GC-XX:MaxGCPauseMillis=20 - XX:InitiatingHeapOccupancyPercent=35 -XX:G1HeapRegionSize=16M-XX:MinMetaspaceFreeRatio=50 - XX:MaxMetaspaceFreeRatio=80"

Re: kafka broker + recommendation for JVM config

asirna — Fri, 25 May 2018 18:10:11 GMT

@Michael Bronson,

You can try setting

export KAFKA_JVM_PERFORMANCE_OPTS="-Xmx8g -Xms8g -XX:MetaspaceSize=96m -XX:+UseG1GC-XX:MaxGCPauseMillis=20 - XX:InitiatingHeapOccupancyPercent=35 -XX:G1HeapRegionSize=16M-XX:MinMetaspaceFreeRatio=50 - XX:MaxMetaspaceFreeRatio=80"

Re: kafka broker + recommendation for JVM config

mike_bronson7 — Fri, 25 May 2018 18:17:06 GMT

so the steps in https://community.hortonworks.com/questions/193769/how-to-add-variable-in-kafka-env-template-by-api.html are the same steps for ?

export KAFKA_HEAP_OPTS="-Xmx8g -Xms8g -XX:MetaspaceSize=96m -XX:+UseG1GC-XX:MaxGCPauseMillis=20 - XX:InitiatingHeapOccupancyPercent=35 -XX:G1HeapRegionSize=16M-XX:MinMetaspaceFreeRatio=50 - XX:MaxMetaspaceFreeRatio=80"

Re: kafka broker + recommendation for JVM config

asirna — Fri, 25 May 2018 18:53:23 GMT

@Michael Bronson,

Yes. The steps are same for any config change in kafka-env.

The key is KAFKA_JVM_PERFORMANCE_OPTS and not KAFKA_HEAP_OPTS.

Re: kafka broker + recommendation for JVM config

mike_bronson7 — Fri, 25 May 2018 19:38:27 GMT

so in that case I prefer to separate the variables as the following:

export KAFKA_HEAP_OPTS="-Xmx8g -Xms8g"


AND

export KAFKA_JVM_PERFORMANCE_OPTS="-XX:MetaspaceSize=96m -XX:+UseG1GC-XX:MaxGCPauseMillis=20 - XX:InitiatingHeapOccupancyPercent=35 -XX:G1HeapRegionSize=16M-XX:MinMetaspaceFreeRatio=50 - XX:MaxMetaspaceFreeRatio=80"

can you please confirm that?

so both them will be added to kafka-env

Re: kafka broker + recommendation for JVM config

mike_bronson7 — Fri, 25 May 2018 20:29:28 GMT

example of final file

<br>{
  "href" : "http://master02:8080/api/v1/clusters/HDP/configurations?type=kafka-env&tag=version1527250007610",
  "items" : [
    {
      "href" : "http://master02:8080/api/v1/clusters/HDP26/configurations?type=kafka-env&tag=version1527250007610",
      "tag" : "version1527250007610",
      "type" : "kafka-env",
      "version" : 8,
      "Config" : {
        "cluster_name" : "HDP",
        "stack_id" : "HDP-2.6"
      },
      "properties" : {
        "content" : "\n#!/bin/bash\n\n# Set KAFKA specific environment variables here.\n\n# The java implementation to use.\nexport JAVA_HOME={{java64_home}}\nexport PATH=$PATH:$JAVA_HOME/bin\nexport PID_DIR={{kafka_pid_dir}}\nexport LOG_DIR={{kafka_log_dir}}\nexport KAFKA_KERBEROS_PARAMS={{kafka_kerberos_params}}\nexport JMX_PORT=9997\n# Add kafka sink to classpath and related depenencies\nif [ -e \"/usr/lib/ambari-metrics-kafka-sink/ambari-metrics-kafka-sink.jar\" ]; then\n  export CLASSPATH=$CLASSPATH:/usr/lib/ambari-metrics-kafka-sink/ambari-metrics-kafka-sink.jar\n  export CLASSPATH=$CLASSPATH:/usr/lib/ambari-metrics-kafka-sink/lib/*\nfi\n\nif [ -f /etc/kafka/conf/kafka-ranger-env.sh ]; then\n. /etc/kafka/conf/kafka-ranger-env.sh\nfi\nexport KAFKA_HEAP_OPTS=\"-Xmx8g -Xms8g\"\nKAFKA_JVM_PERFORMANCE_OPTS=\"-XX:MetaspaceSize=96m -XX:+UseG1GC-XX:MaxGCPauseMillis=20 - XX:InitiatingHeapOccupancyPercent=35 -XX:G1HeapRegionSize=16M-XX:MinMetaspaceFreeRatio=50 - XX:MaxMetaspaceFreeRatio=80\n"",
        "is_supported_kafka_ranger" : "true",
        "kafka_log_dir" : "/var/log/kafka",
        "kafka_pid_dir" : "/var/run/kafka",
        "kafka_user" : "kafka",
        "kafka_user_nofile_limit" : "128000",
        "kafka_user_nproc_limit" : "65536"
      }
    }
  ]

Re: kafka broker + recommendation for JVM config

asirna — Fri, 25 May 2018 22:28:37 GMT

@Michael Bronson,

Yes. Your both the above configs looks file and your final curl call will look like this

curl -u admin:admin -H "X-Requested-By: ambari" -X PUT -d '[
  {
    "Clusters": {
      "desired_config": [
        {
          "type": "kafka-env",
          "tag": "unique value",
           "properties" : {
        "content" : "\n#!/bin/bash\n\n# Set KAFKA specific environment variables here.\n\n# The java implementation to use.\nexport JAVA_HOME={{java64_home}}\nexport PATH=$PATH:$JAVA_HOME/bin\nexport PID_DIR={{kafka_pid_dir}}\nexport LOG_DIR={{kafka_log_dir}}\nexport KAFKA_KERBEROS_PARAMS={{kafka_kerberos_params}}\nexport JMX_PORT=9997\n# Add kafka sink to classpath and related depenencies\nif [ -e \"/usr/lib/ambari-metrics-kafka-sink/ambari-metrics-kafka-sink.jar\" ]; then\n  export CLASSPATH=$CLASSPATH:/usr/lib/ambari-metrics-kafka-sink/ambari-metrics-kafka-sink.jar\n  export CLASSPATH=$CLASSPATH:/usr/lib/ambari-metrics-kafka-sink/lib/*\nfi\n\nif [ -f /etc/kafka/conf/kafka-ranger-env.sh ]; then\n. /etc/kafka/conf/kafka-ranger-env.sh\nfi\nexport KAFKA_HEAP_OPTS=\"-Xmx8g -Xms8g\"\nKAFKA_JVM_PERFORMANCE_OPTS=\"-XX:MetaspaceSize=96m -XX:+UseG1GC-XX:MaxGCPauseMillis=20 - XX:InitiatingHeapOccupancyPercent=35 -XX:G1HeapRegionSize=16M-XX:MinMetaspaceFreeRatio=50 - XX:MaxMetaspaceFreeRatio=80\n"",
        "is_supported_kafka_ranger" : "true",
        "kafka_log_dir" : "/var/log/kafka",
        "kafka_pid_dir" : "/var/run/kafka",
        "kafka_user" : "kafka",
        "kafka_user_nofile_limit" : "128000",
        "kafka_user_nproc_limit" : "65536"
        },
        "service_config_version_note": "New config version"
        }
      ]
    }
  }
]' "http://master02:8080/api/v1/clusters/HDP"