Member since
08-08-2017
1652
Posts
30
Kudos Received
11
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 1950 | 06-15-2020 05:23 AM | |
| 15881 | 01-30-2020 08:04 PM | |
| 2093 | 07-07-2019 09:06 PM | |
| 8189 | 01-27-2018 10:17 PM | |
| 4646 | 12-31-2017 10:12 PM |
05-27-2018
09:40 AM
we have hadoop cluster version 2.6.4 with 3 kafka machines we are thinking what is the best filesystem for the kafka disk - sdb ( 25T ) according to the article - https://community.hortonworks.com/articles/80813/kafka-best-practices-1.html "FileSystem Selection Kafka uses regular files on disk, and such it has no hard dependency on a specific file system. We recommend EXT4 or XFS. Recent improvements to the XFS file system have shown it to have the better performance characteristics for Kafka’s workload without any compromise in stability. Note: Do not use mounted shared drives and any network file systems. In our experience Kafka is known to have index failures on such file systems. Kafka uses MemoryMapped files to store the offset index which has known issues on anetwork file systems." seems that they recommended on xfs file system and not ext 4 but xfs is old file system , so I am little confuse I will happy to know more opiones on that my sdb ( kafka disk ) df | grep sdb NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sdb 8:16 0 25.5T 0 disk
... View more
Labels:
05-25-2018
01:29 PM
example of final file <br>{
"href" : "http://master02:8080/api/v1/clusters/HDP/configurations?type=kafka-env&tag=version1527250007610",
"items" : [
{
"href" : "http://master02:8080/api/v1/clusters/HDP26/configurations?type=kafka-env&tag=version1527250007610",
"tag" : "version1527250007610",
"type" : "kafka-env",
"version" : 8,
"Config" : {
"cluster_name" : "HDP",
"stack_id" : "HDP-2.6"
},
"properties" : {
"content" : "\n#!/bin/bash\n\n# Set KAFKA specific environment variables here.\n\n# The java implementation to use.\nexport JAVA_HOME={{java64_home}}\nexport PATH=$PATH:$JAVA_HOME/bin\nexport PID_DIR={{kafka_pid_dir}}\nexport LOG_DIR={{kafka_log_dir}}\nexport KAFKA_KERBEROS_PARAMS={{kafka_kerberos_params}}\nexport JMX_PORT=9997\n# Add kafka sink to classpath and related depenencies\nif [ -e \"/usr/lib/ambari-metrics-kafka-sink/ambari-metrics-kafka-sink.jar\" ]; then\n export CLASSPATH=$CLASSPATH:/usr/lib/ambari-metrics-kafka-sink/ambari-metrics-kafka-sink.jar\n export CLASSPATH=$CLASSPATH:/usr/lib/ambari-metrics-kafka-sink/lib/*\nfi\n\nif [ -f /etc/kafka/conf/kafka-ranger-env.sh ]; then\n. /etc/kafka/conf/kafka-ranger-env.sh\nfi\nexport KAFKA_HEAP_OPTS=\"-Xmx8g -Xms8g\"\nKAFKA_JVM_PERFORMANCE_OPTS=\"-XX:MetaspaceSize=96m -XX:+UseG1GC-XX:MaxGCPauseMillis=20 - XX:InitiatingHeapOccupancyPercent=35 -XX:G1HeapRegionSize=16M-XX:MinMetaspaceFreeRatio=50 - XX:MaxMetaspaceFreeRatio=80\n"",
"is_supported_kafka_ranger" : "true",
"kafka_log_dir" : "/var/log/kafka",
"kafka_pid_dir" : "/var/run/kafka",
"kafka_user" : "kafka",
"kafka_user_nofile_limit" : "128000",
"kafka_user_nproc_limit" : "65536"
}
}
]
... View more
05-25-2018
12:38 PM
so in that case I prefer to separate the variables as the following: export KAFKA_HEAP_OPTS="-Xmx8g -Xms8g"
AND
export KAFKA_JVM_PERFORMANCE_OPTS="-XX:MetaspaceSize=96m -XX:+UseG1GC-XX:MaxGCPauseMillis=20 - XX:InitiatingHeapOccupancyPercent=35 -XX:G1HeapRegionSize=16M-XX:MinMetaspaceFreeRatio=50 - XX:MaxMetaspaceFreeRatio=80" can you please confirm that? so both them will be added to kafka-env
... View more
05-25-2018
11:17 AM
so the steps in https://community.hortonworks.com/questions/193769/how-to-add-variable-in-kafka-env-template-by-api.html are the same steps for ? export KAFKA_HEAP_OPTS="-Xmx8g -Xms8g -XX:MetaspaceSize=96m -XX:+UseG1GC-XX:MaxGCPauseMillis=20 - XX:InitiatingHeapOccupancyPercent=35 -XX:G1HeapRegionSize=16M-XX:MinMetaspaceFreeRatio=50 - XX:MaxMetaspaceFreeRatio=80"
... View more
05-25-2018
11:03 AM
as export KAFKA_HEAP_OPTS="-Xmx8g -Xms8g -XX:MetaspaceSize=96m -XX:+UseG1GC-XX:MaxGCPauseMillis=20 - XX:InitiatingHeapOccupancyPercent=35 -XX:G1HeapRegionSize=16M-XX:MinMetaspaceFreeRatio=50 - XX:MaxMetaspaceFreeRatio=80" ?
... View more
05-25-2018
11:02 AM
Or in simple words can we set all "-Xmx8g -Xms8g -XX:MetaspaceSize=96m -XX:+UseG1GC-XX:MaxGCPauseMillis=20 - XX:InitiatingHeapOccupancyPercent=35 -XX:G1HeapRegionSize=16M-XX:MinMetaspaceFreeRatio=50 - XX:MaxMetaspaceFreeRatio=80" in KAFKA_HEAP_OPTS ?
... View more
05-25-2018
10:57 AM
@Aditya thank you so much for the great answer , I am really appropriate this , about the "-XX:MetaspaceSize=96m -XX:+UseG1GC-XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:G1HeapRegionSize=16M-XX:MinMetaspaceFreeRatio=50 -XX:MaxMetaspaceFreeRatio=80" , where we can put thiose values , in which variables?
... View more
05-25-2018
07:30 AM
hi all according to the article in - https://community.hortonworks.com/articles/80813/kafka-best-practices-1.html We recommend latest java 1.8 with G1 collector ( which is default in new version). If you are using Java 1.7 and G1 collector make sure you are on u51 or higher. A recommended setting for JVM looks like following -Xmx8g -Xms8g -XX:MetaspaceSize=96m -XX:+UseG1GC-XX:MaxGCPauseMillis=20 - XX:InitiatingHeapOccupancyPercent=35 -XX:G1HeapRegionSize=16M-XX:MinMetaspaceFreeRatio=50 - XX:MaxMetaspaceFreeRatio=80 my question - in which variable we need to set the values -> "-Xmx8g -Xms8g -XX:MetaspaceSize=96m -XX:+UseG1GC-XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:G1HeapRegionSize=16M-XX:MinMetaspaceFreeRatio=50 -XX:MaxMetaspaceFreeRatio=80" ? second - is it possible to set all recomended JVM by API ?
... View more
Labels:
05-25-2018
07:07 AM
Did you check the repo path on ambari GUI? , if not created by API and you not see any repo PATH on ambari GUI , you can create the repo manual on ambari GUI , go to admin --> stack and version --> versions --> manage version and click on the version link then you get repo window
... View more
05-24-2018
09:27 PM
how to add the following variable ( export KAFKA_HEAP_OPTS="-Xms3g -Xmx3g" ) in kafka-env template by API? the target is to update the file kafka-env template in ambari GUI by API from ambari GUI Ambari UI --> Configs --> Advanced --> "Advanced kafka-env" --> kafka-env template the variable that we want to add by API: export KAFKA_HEAP_OPTS="-Xms3g -Xmx3g"
... View more
Labels: