Kafka Best Practices + KAFKA_JVM_PERFORMANCE_OPTS

[#1]

dear friends and colleges

we set the KAFKA_JVM_PERFORMANCE_OPTS on all ours kafka physical machines , according to the following Link:

https://community.hortonworks.com/articles/80813/kafka-best-practices-1.html

we set in ambari - Advanced kafka-env the following:

 export KAFKA_JVM_PERFORMANCE_OPTS="-XX:MetaspaceSize=96m -XX:+UseG1GC-XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:G1HeapRegionSize=16M-XX:MinMetaspaceFreeRatio=50 -XX:MaxMetaspaceFreeRatio=80"

after we set that variable we restart the kafka broker to take affect

but kafka broker fails after this settings , please advice what is wrong on the settings from the article - https://community.hortonworks.com/articles/80813/kafka-best-practices-1.html

from the logs we can see that:

from kafka.err

more kafka.err
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Unrecognized VM option 'UseG1GC-XX:MaxGCPauseMillis=20'
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
Unrecognized VM option 'UseG1GC-XX:MaxGCPauseMillis=20'
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
Unrecognized VM option 'UseG1GC-XX:MaxGCPauseMillis=20'
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
Unrecognized VM option 'UseG1GC-XX:MaxGCPauseMillis=20'
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

from server.log

[2018-05-31 09:03:19,334] INFO starting (kafka.server.KafkaServer)
[2018-05-31 09:03:19,350] INFO [ThrottledRequestReaper-Fetch], Starting  (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2018-05-31 09:03:19,351] INFO [ThrottledRequestReaper-Produce], Starting  (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2018-05-31 09:03:19,358] INFO Connecting to zookeeper on master01.sys674.com:2181,master02.sys674.com:2181,master03.sys674.com:2181 (kafka.server.KafkaServer)
[2018-05-31 09:03:19,639] INFO Cluster ID = 7BUsZLW1RJa54GJ0CDCtcg (kafka.server.KafkaServer)
[2018-05-31 09:03:19,682] INFO Loading logs. (kafka.log.LogManager)
[2018-05-31 09:03:19,690] INFO Logs loading complete in 8 ms. (kafka.log.LogManager)
[2018-05-31 09:03:19,848] INFO Starting log cleanup with a period of 300000 ms. (kafka.log.LogManager)
[2018-05-31 09:03:19,850] INFO Starting log flusher with a default period of 9223372036854775807 ms. (kafka.log.LogManager)
[2018-05-31 09:03:19,855] WARN No meta.properties file under dir /var/kafka/kafka-logs/meta.properties (kafka.server.BrokerMetadataCheckpoint)
[2018-05-31 09:03:19,913] INFO Awaiting socket connections on kafka01.sys674.com:6667. (kafka.network.Acceptor)
[2018-05-31 09:03:19,917] INFO [Socket Server on Broker 1003], Started 1 acceptor threads (kafka.network.SocketServer)
[2018-05-31 09:03:19,943] INFO [ExpirationReaper-1003], Starting  (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2018-05-31 09:03:19,944] INFO [ExpirationReaper-1003], Starting  (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2018-05-31 09:03:19,991] INFO [ExpirationReaper-1003], Starting  (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2018-05-31 09:03:19,994] INFO [ExpirationReaper-1003], Starting  (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2018-05-31 09:03:19,995] INFO [ExpirationReaper-1003], Starting  (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2018-05-31 09:03:20,077] INFO [GroupCoordinator 1003]: Starting up. (kafka.coordinator.GroupCoordinator)
[2018-05-31 09:03:20,081] INFO [GroupCoordinator 1003]: Startup complete. (kafka.coordinator.GroupCoordinator)
[2018-05-31 09:03:20,082] INFO [Group Metadata Manager on Broker 1003]: Removed 0 expired offsets in 1 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2018-05-31 09:03:20,118] INFO Will not load MX4J, mx4j-tools.jar is not in the classpath (kafka.utils.Mx4jLoader$)
[2018-05-31 09:03:20,172] INFO Creating /brokers/ids/1003 (is it secure? false) (kafka.utils.ZKCheckedEphemeral)
[2018-05-31 09:03:20,187] INFO Result of znode creation is: OK (kafka.utils.ZKCheckedEphemeral)
[2018-05-31 09:03:20,189] INFO Registered broker 1003 at path /brokers/ids/1003 with addresses: PLAINTEXT -> EndPoint(kafka01.sys674.com,6667,PLAINTEXT) (kafka.utils.ZkUtils)
[2018-05-31 09:03:20,190] WARN No meta.properties file under dir /var/kafka/kafka-logs/meta.properties (kafka.server.BrokerMetadataCheckpoint)
[2018-05-31 09:03:20,273] INFO [Kafka Server 1003], started (kafka.server.KafkaServer)
[2018-05-31 09:13:20,078] INFO [Group Metadata Manager on Broker 1003]: Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2018-05-31 09:23:20,078] INFO [Group Metadata Manager on Broker 1003]: Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2018-05-31 09:33:20,078] INFO [Group Metadata Manager on Broker 1003]: Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2018-05-31 09:43:20,078] INFO [Group Metadata Manager on Broker 1003]: Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2018-05-31 09:46:25,337] INFO [Kafka Server 1003], shutting down (kafka.server.KafkaServer)
[2018-05-31 09:46:25,339] INFO [Kafka Server 1003], Starting controlled shutdown (kafka.server.KafkaServer)
[2018-05-31 09:46:25,393] INFO [Kafka Server 1003], Controlled shutdown succeeded (kafka.server.KafkaServer)
[2018-05-31 09:46:25,396] INFO [Socket Server on Broker 1003], Shutting down (kafka.network.SocketServer)
[2018-05-31 09:46:25,406] INFO [Socket Server on Broker 1003], Shutdown completed (kafka.network.SocketServer)
[2018-05-31 09:46:25,407] INFO [Kafka Request Handler on Broker 1003], shutting down (kafka.server.KafkaRequestHandlerPool)
[2018-05-31 09:46:25,413] INFO [Kafka Request Handler on Broker 1003], shut down completely (kafka.server.KafkaRequestHandlerPool)
[2018-05-31 09:46:25,420] INFO [ThrottledRequestReaper-Fetch], Shutting down (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2018-05-31 09:46:25,595] INFO [ThrottledRequestReaper-Fetch], Stopped  (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2018-05-31 09:46:25,595] INFO [ThrottledRequestReaper-Fetch], Shutdown completed (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2018-05-31 09:46:25,595] INFO [ThrottledRequestReaper-Produce], Shutting down (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2018-05-31 09:46:25,621] INFO [ThrottledRequestReaper-Produce], Stopped  (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2018-05-31 09:46:25,621] INFO [ThrottledRequestReaper-Produce], Shutdown completed (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2018-05-31 09:46:25,622] INFO [KafkaApi-1003] Shutdown complete. (kafka.server.KafkaApis)
[2018-05-31 09:46:25,624] INFO [Replica Manager on Broker 1003]: Shutting down (kafka.server.ReplicaManager)
[2018-05-31 09:46:25,624] INFO [ReplicaFetcherManager on broker 1003] shutting down (kafka.server.ReplicaFetcherManager)
[2018-05-31 09:46:25,626] INFO [ReplicaFetcherManager on broker 1003] shutdown completed (kafka.server.ReplicaFetcherManager)
[2018-05-31 09:46:25,626] INFO [ExpirationReaper-1003], Shutting down (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2018-05-31 09:46:25,671] INFO [ExpirationReaper-1003], Stopped  (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2018-05-31 09:46:25,671] INFO [ExpirationReaper-1003], Shutdown completed (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2018-05-31 09:46:25,671] INFO [ExpirationReaper-1003], Shutting down (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2018-05-31 09:46:25,832] INFO [ExpirationReaper-1003], Stopped  (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2018-05-31 09:46:25,832] INFO [ExpirationReaper-1003], Shutdown completed (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2018-05-31 09:46:25,835] INFO [Replica Manager on Broker 1003]: Shut down completely (kafka.server.ReplicaManager)
[2018-05-31 09:46:25,835] INFO [ExpirationReaper-1003], Shutting down (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2018-05-31 09:46:25,873] INFO [ExpirationReaper-1003], Stopped  (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2018-05-31 09:46:25,873] INFO [ExpirationReaper-1003], Shutdown completed (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2018-05-31 09:46:25,874] INFO [GroupCoordinator 1003]: Shutting down. (kafka.coordinator.GroupCoordinator)
[2018-05-31 09:46:25,874] INFO [ExpirationReaper-1003], Shutting down (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2018-05-31 09:46:26,067] INFO [ExpirationReaper-1003], Stopped  (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2018-05-31 09:46:26,067] INFO [ExpirationReaper-1003], Shutdown completed (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2018-05-31 09:46:26,067] INFO [ExpirationReaper-1003], Shutting down (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2018-05-31 09:46:26,125] INFO [ExpirationReaper-1003], Stopped  (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2018-05-31 09:46:26,125] INFO [ExpirationReaper-1003], Shutdown completed (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2018-05-31 09:46:26,126] INFO [GroupCoordinator 1003]: Shutdown complete. (kafka.coordinator.GroupCoordinator)
[2018-05-31 09:46:26,127] INFO Shutting down. (kafka.log.LogManager)
[2018-05-31 09:46:26,138] INFO Shutdown complete. (kafka.log.LogManager)
[2018-05-31 09:46:26,151] INFO [Kafka Server 1003], shut down completed (kafka.server.KafkaServer)
<br>

[#2 · Accepted solution]

@Michael Bronson

Between two JVM parameters you need to give a space.

There are no space between the following parametyers:

1.

-XX:+UseG1GC-XX:MaxGCPauseMillis=20

2.

-XX:G1HeapRegionSize=16M-XX:MinMetaspaceFreeRatio=50

.

Correct option should be:

export KAFKA_JVM_PERFORMANCE_OPTS="-XX:MetaspaceSize=96m -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:G1HeapRegionSize=16M -XX:MinMetaspaceFreeRatio=50 -XX:MaxMetaspaceFreeRatio=80"

.

[#3]

@Jay I think its better to fix the syntax as described in the article also , other users can faced the same problem

[#4]

@Michael Bronson

Thank you for your feedback. I have fixed the article typo in the referrenced article..

[#5]

@Jay thank you so much , now all users can enjoy the tight syntax

[#6]

@jay can you help me please , from some unclear reason I get "This post is currently awaiting moderation. If you believe this to be in error, contact a system administrator." on new question , what we can do to resolve this ?