Hello Mike   Likely your existing 3 Zookeeper nodes can serve your expansion requirements You can monitor the CPU and network of the Zookeeper nodes when your Kafka cluster is growing, when reaching the throughput limit, you can expand your zookeeper to 5 nodes Remember the zookeeper nodes need to keep in sync all the time, so the more zookeeper nodes the more traffic will be added to keep them in sync, while those nodes handling the Kafka requests; so it doesn't mean the more the better   I would suggest to stay with 3 zookeeper nodes while expanding your kafka cluster with close monitoring, and consider to grow to 5 when the CPU/network throughput reaching the limit   You can also consider to tune the zookeeper nodes e.g. dedicated disks, better network throughput, isolate zookeeper process, disable swaps  ... View more

I want to say also that node-manager restart or fully restart of yarn service fixed the problem , but as you know this isn't the right solution that should be every time that one of the node manager became die  ... View more

I have the same problem, but in my case JVM pause detections are happening every 15 mins and detecting between 28337ms, .. 853466ms pauses. ... View more

@mike_bronson7, as this is an older post, you would have a better chance of receiving a resolution by starting a new thread. This will also be an opportunity to provide details specific to your environment that could aid others in assisting you with a more accurate answer to your question. You can link this thread as a reference in your new post. ... View more

first thank you so much , for your help ,   I see in the post the following example:   [{"ConfigGroup":{"id":2,"cluster_name":"c1","group_name":"A config group","tag":"HDFS","description":"A config group","hosts":[{"host_name":"host1"}],"service_config_version_note":"change","desired_configs":[{"type":"hdfs-site","tag":"version1443587493807","properties":{"dfs.replication":"2","dfs.datanode.du.reserved":"1073741822"}}]}}]   I will appreciate , to get full example about how to run this API , by using curl or full Ambari API    note - about - version1443587493807 , is this version number is "random" number that I need to set ?    ... View more

What is stored inside this blockmgr-* files ? It has any relation to the input files spark reading ? ... View more

@mike_bronson7    In kafka 0.1x we will see this statement (Consumer group ‘deeg_data’ is rebalancing) when the group is rebalancing but in newer versions, we will see something like:   GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID GroupName topicName 0 0 0 0 - - - Which means no active consumers in this group (or rebalancing).   Regarding rebalancing of a group this can be triggered for multiple reasons, but mostly because of:   1. A new consumer is added/joined to the group 2. A consumer was removed from the group (because of client shutdown, timeout, network glitches) 3. Timeout issues between brokers/client   To get more details about consumers rebalancing (if no errors from the broker side) checking the application log files might provide some details about the underlying issue. ... View more

Hi @mike_bronson7    1. Do you see anything interesting from the broker 1010 log file? this is to try to understand why 1010 is not able to register in zookeeper.   2. Try forcing a new controller by using: [zk: localhost:2181(CONNECTED) 11] rmr /controller   3. Are these broker ids unique? if you describe other topics, do you see the same brokers ids and same behavior (leader none for some partitions)?   4. Finally, if this is dev env:  4.1 You can enable unclean leader election = true and restart the brokers   Or:  4.2 (if this happening just for this topic) remove __consumer_offsets topic (just from zookeeper) and restart kafka    ... View more