hi all we have 3 kafka brokers ( HDP version - 2.6.0.1 and ambari verdion - 2.6.0 ) we cant start them from ambari , and we saw the following errors on ambari-agent logs the error about - Fail: Configuration parameter 'kafka-env' was not found in configurations dictionary! what chould be the problem here ? Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/ambari_agent/PythonReflectiveExecutor.py", line 59, in run_file imp.load_source('__main__', script) File "/var/lib/ambari-agent/cache/common-services/KAFKA/0.8.1/package/scripts/kafka_broker.py", line 141, in <module> KafkaBroker().execute() File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 314, in execute method(env) File "/var/lib/ambari-agent/cache/common-services/KAFKA/0.8.1/package/scripts/kafka_broker.py", line 128, in status import status_params File "/var/lib/ambari-agent/cache/common-services/KAFKA/0.8.1/package/scripts/status_params.py", line 26, in <module> kafka_pid_file = format("{kafka_pid_dir}/kafka.pid") File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/format.py", line 95, in format return ConfigurationFormatter().format(format_string, args, **result) File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/format.py", line 59, in format result_protected = self.vformat(format_string, args, all_params) File "/usr/lib64/python2.7/string.py", line 549, in vformat result = self._vformat(format_string, args, kwargs, used_args, 2) File "/usr/lib64/python2.7/string.py", line 582, in _vformat result.append(self.format_field(obj, format_spec)) File "/usr/lib64/python2.7/string.py", line 599, in format_field return format(value, format_spec) File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/config_dictionary.py", line 73, in __getattr__ raise Fail("Configuration parameter '" + self.name + "' was not found in configurations dictionary!") Fail: Configuration parameter 'kafka-env' was not found in configurations dictionary! ... View more

hi all I want to validate the balanced of the topics , include replica and Isr I will give some example of wrong balanced ( master01 is the zookeper server , and we defined 3 replicate ) on that example we can see that kafka brokers are missing from some Isr /usr/hdp/current/kafka-broker/bin/kafka-topics.sh --describe --zookeeper master01:2181 --topic mno.de.pola.trump Topic:mno.de.pola.trump PartitionCount:100 ReplicationFactor:3 Configs: Topic: mno.de.pola.trump Partition: 0 Leader: 1017 Replicas: 1017,1018,1016 Isr: 1017,1018,1016 Topic: mno.de.pola.trump Partition: 1 Leader: 1018 Replicas: 1018,1016,1017 Isr: 1018,1017,1016 Topic: mno.de.pola.trump Partition: 2 Leader: 1016 Replicas: 1016,1017,1018 Isr: 1016,1018 Topic: mno.de.pola.trump Partition: 3 Leader: 1017 Replicas: 1017,1016,1018 Isr: 1017,1018,1016 Topic: mno.de.pola.trump Partition: 4 Leader: 1018 Replicas: 1018,1017,1016 Isr: 1018,1017,1016 Topic: mno.de.pola.trump Partition: 5 Leader: 1016 Replicas: 1016,1018,1017 Isr: 1016,1018,1017 Topic: mno.de.pola.trump Partition: 6 Leader: 1017 Replicas: 1017,1018,1016 Isr: 1017,1018 on the following example we can see that Leader on partitions are more on broker 1017 /usr/hdp/current/kafka-broker/bin/kafka-topics.sh --describe --zookeeper master01:2181 --topic mno.de.pola.trump Topic:mno.de.pola.trump PartitionCount:100 ReplicationFactor:3 Configs: Topic: mno.de.pola.trump Partition: 0 Leader: 1017 Replicas: 1017,1018,1016 Isr: 1017,1018,1016 Topic: mno.de.pola.trump Partition: 1 Leader: 1018 Replicas: 1018,1016,1017 Isr: 1018,1017,1016 Topic: mno.de.pola.trump Partition: 2 Leader: 1017 Replicas: 1016,1017,1018 Isr: 1016,1018.1017 Topic: mno.de.pola.trump Partition: 3 Leader: 1017 Replicas: 1017,1016,1018 Isr: 1017,1018,1016 Topic: mno.de.pola.trump Partition: 4 Leader: 1018 Replicas: 1018,1017,1016 Isr: 1018,1017,1016 Topic: mno.de.pola.trump Partition: 5 Leader: 1016 Replicas: 1016,1018,1017 Isr: 1016,1018,1017 Topic: mno.de.pola.trump Partition: 6 Leader: 1017 Replicas: 1017,1018,1016 Isr: 1017,1018,1016 on the following example we can see that brokers are missing from Replica /usr/hdp/current/kafka-broker/bin/kafka-topics.sh --describe --zookeeper master01:2181 --topic mno.de.pola.trump Topic:mno.de.pola.trump PartitionCount:100 ReplicationFactor:3 Configs: Topic: mno.de.pola.trump Partition: 0 Leader: 1017 Replicas: 1017,1018 Isr: 1017,1018,1016 Topic: mno.de.pola.trump Partition: 1 Leader: 1018 Replicas: 1018,1016,1017 Isr: 1018,1017,1016 Topic: mno.de.pola.trump Partition: 2 Leader: 1017 Replicas: 1016,1017,1018 Isr: 1016,1018,1017 Topic: mno.de.pola.trump Partition: 3 Leader: 1017 Replicas: 1017,1016 Isr: 1017,1018,1016 Topic: mno.de.pola.trump Partition: 4 Leader: 1018 Replicas: 1018,1017,1016 Isr: 1018,1017,1016 Topic: mno.de.pola.trump Partition: 5 Leader: 1016 Replicas: 1016,1018,1017 Isr: 1016,1018,1017 Topic: mno.de.pola.trump Partition: 6 Leader: 1017 Replicas: 1017,1018,1016 Isr: 1017,1018,1016 And so on . . . so my question - I will happy to know about some tool that can validate the output from kafka-topics.sh --describe any ideas about this example of good balanced configuration /usr/hdp/current/kafka-broker/bin/kafka-topics.sh --describe --zookeeper master01:2181 --topic mno.de.pola.trump Topic: mno.de.pola.trump PartitionCount:100 ReplicationFactor:3 Configs: Topic: mno.de.pola.trump Partition: 0 Leader: 1017 Replicas: 1017,1018,1016 Isr: 1017,1018,1016 Topic: mno.de.pola.trump Partition: 1 Leader: 1018 Replicas: 1018,1016,1017 Isr: 1018,1017,1016 Topic: mno.de.pola.trump Partition: 2 Leader: 1016 Replicas: 1016,1017,1018 Isr: 1016,1018,1017 Topic: mno.de.pola.trump Partition: 3 Leader: 1017 Replicas: 1017,1016,1018 Isr: 1017,1018,1016 Topic: mno.de.pola.trump Partition: 4 Leader: 1018 Replicas: 1018,1017,1016 Isr: 1018,1017,1016 Topic: mno.de.pola.trump Partition: 5 Leader: 1016 Replicas: 1016,1018,1017 Isr: 1016,1018,1017 ... View more

hi all we run the following command on kafka machine in order to view/print the topics names we can see that its fails on Caused by: java.net.BindException: Address already in use but without to print also the port number ./kafka-topics.sh --zookeeper zookeeper1:2181 --list To enable GC log rotation, use -Xloggc:<filename> -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=<num_of_files> where num_of_file > 0 GC log rotation is turned off Exception in thread "main" java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:386) at sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:401) Caused by: java.net.BindException: Address already in use any suggestion how to find the port ? that is use ... View more