Member since
08-08-2017
1652
Posts
30
Kudos Received
11
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 1943 | 06-15-2020 05:23 AM | |
| 15761 | 01-30-2020 08:04 PM | |
| 2091 | 07-07-2019 09:06 PM | |
| 8161 | 01-27-2018 10:17 PM | |
| 4638 | 12-31-2017 10:12 PM |
10-04-2018
09:38 AM
after we build a new cluster HDP - 2.6.4 cluster include - 3 masters machines and 3 workers machines ( data node ) and starting the services we get the following error when starting the start the ZKFailoverController on master machines 2018-10-04 08:41:42,298 ERROR datanode.DataNode (DataNode.java:secureMain(2692)) - Exception in secureMain
java.lang.IllegalArgumentException: Does not contain a valid host:port authority: master01.krysve_ci90.com:8020
at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:213)
at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:164)
at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:153)
at org.apache.hadoop.hdfs.DFSUtil.getAddressesForNameserviceId(DFSUtil.java:699)
at org.apache.hadoop.hdfs.DFSUtil.getAddressesForNsIds(DFSUtil.java:668)
at org.apache.hadoop.hdfs.DFSUtil.getNNServiceRpcAddressesForCluster(DFSUtil.java:885)
at org.apache.hadoop.hdfs.server.datanode.BlockPoolManager.refreshNamenodes(BlockPoolManager.java:156)
at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:1287)
at org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:456)
at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2591)
at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2493)
at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2540)
at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2685)
at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:2709)
2018-10-04 08:41:42,300 INFO util.ExitUtil (ExitUtil.java:terminate(124)) - Exiting with status 1
2018-10-04 08:41:42,304 INFO datanode.DataNode (LogAdapter.java:info(47)) - SHUTDOWN_MSG:
/************************************************************ we got - "Does not contain a valid host:port authority" what we need to check in order to understand this issue? , and resolving it
... View more
Labels:
09-30-2018
06:54 AM
hi Sandepp . can you also advice my thred - https://community.hortonworks.com/questions/222627/how-re-balance-the-partitions-to-available-brokers.html
... View more
09-30-2018
05:27 AM
but broker 1001 is up and I see also port is in listen
... View more
09-29-2018
07:20 PM
hi all we have ambari cluster , HDP version - 2.6.4 with 3 kafka machines and 3 zookeeper's kafka1 - 1001 kafka2 - 1002 kafka3 - 1003 we issue is about to re-balance
the partitions to available brokers , we can see ( down ) that some partitions under-replicated
with only two brokers in isr. Instead with 3 1. what is the best way to re balance the kafka topic partitions in ISR 2. second issue is that leader - 1002 is missing - how to fix it ? remark - we have 23 topics ( the list down is partial ) example what we get : [kafka@kafka01 bin]$ ./kafka-topics.sh -describe --zookeeper master:2181
Topic:lop_gt PartitionCount:50 ReplicationFactor:3 Configs:segment.bytes=104857600,cleanup.policy=compact,compression.type=producer
Topic: lop_gt Partition: 0 Leader: 1001 Replicas: 1002,1003,1001 Isr: 1001,1003,1002
Topic: lop_gt Partition: 1 Leader: 1003 Replicas: 1003,1001,1002 Isr: 1003,1001
Topic: lop_gt Partition: 2 Leader: 1001 Replicas: 1001,1002,1003 Isr: 1003,1001,1002
Topic: lop_gt Partition: 3 Leader: 1001 Replicas: 1002,1001,1003 Isr: 1001,1003,1002
Topic: lop_gt Partition: 4 Leader: 1003 Replicas: 1003,1002,1001 Isr: 1003,1001
Topic: lop_gt Partition: 5 Leader: 1001 Replicas: 1001,1003,1002 Isr: 1003,1001,1002
Topic: lop_gt Partition: 6 Leader: 1001 Replicas: 1002,1003,1001 Isr: 1001,1003,1002
Topic: lop_gt Partition: 7 Leader: 1003 Replicas: 1003,1001,1002 Isr: 1003,1001
Topic: lop_gt Partition: 8 Leader: 1001 Replicas: 1001,1002,1003 Isr: 1003,1001,1002
Topic: lop_gt Partition: 9 Leader: 1001 Replicas: 1002,1001,1003 Isr: 1001,1003,1002
Topic: lop_gt Partition: 10 Leader: 1003 Replicas: 1003,1002,1001 Isr: 1003,1001
Topic: lop_gt Partition: 11 Leader: 1001 Replicas: 1001,1003,1002 Isr: 1003,1001,1002
Topic: lop_gt Partition: 12 Leader: 1001 Replicas: 1002,1003,1001 Isr: 1001,1003,1002
Topic: lop_gt Partition: 13 Leader: 1003 Replicas: 1003,1001,1002 Isr: 1003,1001
Topic: lop_gt Partition: 14 Leader: 1001 Replicas: 1001,1002,1003 Isr: 1003,1001,1002
Topic: lop_gt Partition: 15 Leader: 1001 Replicas: 1002,1001,1003 Isr: 1001,1003,1002
Topic: lop_gt Partition: 16 Leader: 1003 Replicas: 1003,1002,1001 Isr: 1003,1001
Topic: lop_gt Partition: 17 Leader: 1001 Replicas: 1001,1003,1002 Isr: 1003,1001,1002
Topic: lop_gt Partition: 18 Leader: 1001 Replicas: 1002,1003,1001 Isr: 1001,1003,1002
Topic: lop_gt Partition: 19 Leader: 1003 Replicas: 1003,1001,1002 Isr: 1003,1001
Topic: lop_gt Partition: 20 Leader: 1001 Replicas: 1001,1002,1003 Isr: 1003,1001,1002
Topic: lop_gt Partition: 21 Leader: 1001 Replicas: 1002,1001,1003 Isr: 1001,1003,1002
Topic: lop_gt Partition: 22 Leader: 1003 Replicas: 1003,1002,1001 Isr: 1003,1001
Topic: lop_gt Partition: 23 Leader: 1001 Replicas: 1001,1003,1002 Isr: 1003,1001,1002
Topic: lop_gt Partition: 24 Leader: 1001 Replicas: 1002,1003,1001 Isr: 1001,1003,1002
Topic: lop_gt Partition: 25 Leader: 1003 Replicas: 1003,1001,1002 Isr: 1003,1001
Topic: lop_gt Partition: 26 Leader: 1001 Replicas: 1001,1002,1003 Isr: 1003,1001,1002
Topic: lop_gt Partition: 27 Leader: 1001 Replicas: 1002,1001,1003 Isr: 1001,1003,1002
Topic: lop_gt Partition: 28 Leader: 1003 Replicas: 1003,1002,1001 Isr: 1003,1001
Topic: lop_gt Partition: 29 Leader: 1001 Replicas: 1001,1003,1002 Isr: 1003,1001,1002
Topic: lop_gt Partition: 30 Leader: 1001 Replicas: 1002,1003,1001 Isr: 1001,1003,1002
Topic: lop_gt Partition: 31 Leader: 1003 Replicas: 1003,1001,1002 Isr: 1003,1001
Topic: lop_gt Partition: 32 Leader: 1001 Replicas: 1001,1002,1003 Isr: 1003,1001,1002
Topic: lop_gt Partition: 33 Leader: 1001 Replicas: 1002,1001,1003 Isr: 1001,1003,1002
Topic: lop_gt Partition: 34 Leader: 1003 Replicas: 1003,1002,1001 Isr: 1003,1001
Topic: lop_gt Partition: 35 Leader: 1001 Replicas: 1001,1003,1002 Isr: 1003,1001,1002
Topic: lop_gt Partition: 36 Leader: 1001 Replicas: 1002,1003,1001 Isr: 1001,1003,1002
Topic: lop_gt Partition: 37 Leader: 1003 Replicas: 1003,1001,1002 Isr: 1003,1001
Topic: lop_gt Partition: 38 Leader: 1001 Replicas: 1001,1002,1003 Isr: 1003,1001,1002
Topic: lop_gt Partition: 39 Leader: 1001 Replicas: 1002,1001,1003 Isr: 1001,1003,1002
Topic: lop_gt Partition: 40 Leader: 1003 Replicas: 1003,1002,1001 Isr: 1003,1001
Topic: lop_gt Partition: 41 Leader: 1001 Replicas: 1001,1003,1002 Isr: 1003,1001,1002
Topic: lop_gt Partition: 42 Leader: 1001 Replicas: 1002,1003,1001 Isr: 1001,1003,1002
Topic: lop_gt Partition: 43 Leader: 1003 Replicas: 1003,1001,1002 Isr: 1003,1001
Topic: lop_gt Partition: 44 Leader: 1001 Replicas: 1001,1002,1003 Isr: 1003,1001,1002
Topic: lop_gt Partition: 45 Leader: 1001 Replicas: 1002,1001,1003 Isr: 1001,1003,1002
Topic: lop_gt Partition: 46 Leader: 1003 Replicas: 1003,1002,1001 Isr: 1003,1001
Topic: lop_gt Partition: 47 Leader: 1001 Replicas: 1001,1003,1002 Isr: 1003,1001,1002
Topic: lop_gt Partition: 48 Leader: 1001 Replicas: 1002,1003,1001 Isr: 1001,1003,1002
Topic: lop_gt Partition: 49 Leader: 1003 Replicas: 1003,1001,1002 Isr: 1003,1001
Topic:_schemas PartitionCount:1 ReplicationFactor:3 Configs:cleanup.policy=compact
Topic: _schemas Partition: 0 Leader: 1001 Replicas: 1001,1002,1003 Isr: 1003,1001,1002
Topic:ambari_kafka_service_check PartitionCount:1 ReplicationFactor:1 Configs:
Topic: ambari_kafka_service_check Partition: 0 Leader: 1002 Replicas: 1002 Isr: 1002
Topic:jr_dfse PartitionCount:10 ReplicationFactor:3 Configs:
Topic: jr_dfse Partition: 0 Leader: 1003 Replicas: 1003,1001,1002 Isr: 1003,1001
Topic: jr_dfse Partition: 1 Leader: 1001 Replicas: 1001,1002,1003 Isr: 1003,1001,1002
Topic: jr_dfse Partition: 2 Leader: 1001 Replicas: 1002,1003,1001 Isr: 1001,1003,1002
Topic: jr_dfse Partition: 3 Leader: 1003 Replicas: 1003,1002,1001 Isr: 1003,1001
Topic: jr_dfse Partition: 4 Leader: 1001 Replicas: 1001,1003,1002 Isr: 1003,1001,1002
Topic: jr_dfse Partition: 5 Leader: 1001 Replicas: 1002,1001,1003 Isr: 1001,1003,1002
Topic: jr_dfse Partition: 6 Leader: 1003 Replicas: 1003,1001,1002 Isr: 1003,1001
Topic: jr_dfse Partition: 7 Leader: 1001 Replicas: 1001,1002,1003 Isr: 1003,1001,1002
Topic: jr_dfse Partition: 8 Leader: 1001 Replicas: 1002,1003,1001 Isr: 1001,1003,1002
Topic: jr_dfse Partition: 9 Leader: 1003 Replicas: 1003,1002,1001 Isr: 1003,1001
Topic:frte_nnc PartitionCount:6 ReplicationFactor:3 Configs:
Topic: frte_nnc Partition: 0 Leader: 1001 Replicas: 1001,1002,1003 Isr: 1003,1001,1002
Topic: frte_nnc Partition: 1 Leader: 1003 Replicas: 1002,1003,1001 Isr: 1003,1001
Topic: frte_nnc Partition: 2 Leader: 1003 Replicas: 1003,1001,1002 Isr: 1003,1001
Topic: frte_nnc Partition: 3 Leader: 1001 Replicas: 1001,1003,1002 Isr: 1003,1001,1002
Topic: frte_nnc Partition: 4 Leader: 1001 Replicas: 1002,1001,1003 Isr: 1001,1003,1002
Topic: frte_nnc Partition: 5 Leader: 1003 Replicas: 1003,1002,1001 Isr: 1003,1001
Topic:erw_plk PartitionCount:100 ReplicationFactor:3 Configs:
Topic: erw_plk Partition: 0 Leader: 1003 Replicas: 1002,1003,1001 Isr: 1003,1001
Topic: erw_plk Partition: 1 Leader: 1003 Replicas: 1003,1001,1002 Isr: 1003,1001
Topic: erw_plk Partition: 2 Leader: 1001 Replicas: 1001,1002,1003 Isr: 1003,1001,1002
Topic: erw_plk Partition: 3 Leader: 1001 Replicas: 1002,1001,1003 Isr: 1001,1003,1002
Topic: erw_plk Partition: 4 Leader: 1003 Replicas: 1003,1002,1001 Isr: 1003,1001
Topic: erw_plk Partition: 5 Leader: 1001 Replicas: 1001,1003,1002 Isr: 1003,1001,1002
Topic: erw_plk Partition: 6 Leader: 1003 Replicas: 1002,1003,1001 Isr: 1003,1001
Topic: erw_plk Partition: 7 Leader: 1003 Replicas: 1003,1001,1002 Isr: 1003,1001
Topic: erw_plk Partition: 8 Leader: 1001 Replicas: 1001,1002,1003 Isr: 1003,1001,1002
Topic: erw_plk Partition: 9 Leader: 1001 Replicas: 1002,1001,1003 Isr: 1001,1003,1002
Topic: erw_plk Partition: 10 Leader: 1003 Replicas: 1003,1002,1001 Isr: 1003,1001
Topic: erw_plk Partition: 11 Leader: 1001 Replicas: 1001,1003,1002 Isr: 1003,1001,1002
Topic: erw_plk Partition: 12 Leader: 1003 Replicas: 1002,1003,1001 Isr: 1003,1001
Topic: erw_plk Partition: 13 Leader: 1003 Replicas: 1003,1001,1002 Isr: 1003,1001
Topic: erw_plk Partition: 14 Leader: 1001 Replicas: 1001,1002,1003 Isr: 1003,1001,1002
Topic: erw_plk Partition: 15 Leader: 1001 Replicas: 1002,1001,1003 Isr: 1001,1003,1002
Topic: erw_plk Partition: 16 Leader: 1003 Replicas: 1003,1002,1001 Isr: 1003,1001
Topic: erw_plk Partition: 17 Leader: 1001 Replicas: 1001,1003,1002 Isr: 1003,1001,1002
Topic: erw_plk Partition: 18 Leader: 1003 Replicas: 1002,1003,1001 Isr: 1003,1001
Topic: erw_plk Partition: 19 Leader: 1003 Replicas: 1003,1001,1002 Isr: 1003,1001
Topic: erw_plk Partition: 20 Leader: 1001 Replicas: 1001,1002,1003 Isr: 1003,1001,1002
Topic: erw_plk Partition: 21 Leader: 1001 Replicas: 1002,1001,1003 Isr: 1001,1003,1002
Topic: erw_plk Partition: 22 Leader: 1003 Replicas: 1003,1002,1001 Isr: 1003,1001
Topic: erw_plk Partition: 23 Leader: 1001 Replicas: 1001,1003,1002 Isr: 1003,1001,1002
Topic: erw_plk Partition: 24 Leader: 1003 Replicas: 1002,1003,1001 Isr: 1003,1001
Topic: erw_plk Partition: 25 Leader: 1003 Replicas: 1003,1001,1002 Isr: 1003,1001
Topic: erw_plk Partition: 26 Leader: 1001 Replicas: 1001,1002,1003 Isr: 1003,1001,1002
Topic: erw_plk Partition: 27 Leader: 1001 Replicas: 1002,1001,1003 Isr: 1001,1003,1002
Topic: erw_plk Partition: 28 Leader: 1003 Replicas: 1003,1002,1001 Isr: 1003,1001
Topic: erw_plk Partition: 29 Leader: 1001 Replicas: 1001,1003,1002 Isr: 1003,1001,1002
Topic: erw_plk Partition: 30 Leader: 1003 Replicas: 1002,1003,1001 Isr: 1003,1001
Topic: erw_plk Partition: 31 Leader: 1003 Replicas: 1003,1001,1002 Isr: 1003,1001
Topic: erw_plk Partition: 32 Leader: 1001 Replicas: 1001,1002,1003 Isr: 1003,1001,1002
Topic: erw_plk Partition: 33 Leader: 1001 Replicas: 1002,1001,1003 Isr: 1001,1003,1002
Topic: erw_plk Partition: 34 Leader: 1003 Replicas: 1003,1002,1001 Isr: 1003,1001
Topic: erw_plk Partition: 35 Leader: 1001 Replicas: 1001,1003,1002 Isr: 1003,1001,1002
Topic: erw_plk Partition: 36 Leader: 1003 Replicas: 1002,1003,1001 Isr: 1003,1001
Topic: erw_plk Partition: 37 Leader: 1003 Replicas: 1003,1001,1002 Isr: 1003,1001
Topic: erw_plk Partition: 38 Leader: 1001 Replicas: 1001,1002,1003 Isr: 1003,1001,1002
Topic: erw_plk Partition: 39 Leader: 1001 Replicas: 1002,1001,1003 Isr: 1001,1003,1002
Topic: erw_plk Partition: 40 Leader: 1003 Replicas: 1003,1002,1001 Isr: 1003,1001
Topic: erw_plk Partition: 41 Leader: 1001 Replicas: 1001,1003,1002 Isr: 1003,1001,1002
Topic: erw_plk Partition: 42 Leader: 1003 Replicas: 1002,1003,1001 Isr: 1003,1001
Topic: erw_plk Partition: 43 Leader: 1003 Replicas: 1003,1001,1002 Isr: 1003,1001
Topic: erw_plk Partition: 44 Leader: 1001 Replicas: 1001,1002,1003 Isr: 1003,1001,1002
Topic: erw_plk Partition: 45 Leader: 1001 Replicas: 1002,1001,1003 Isr: 1001,1003,1002
Topic: erw_plk Partition: 46 Leader: 1003 Replicas: 1003,1002,1001 Isr: 1003,1001
Topic: erw_plk Partition: 47 Leader: 1001 Replicas: 1001,1003,1002 Isr: 1003,1001,1002
Topic: erw_plk Partition: 48 Leader: 1003 Replicas: 1002,1003,1001 Isr: 1003,1001
Topic: erw_plk Partition: 49 Leader: 1003 Replicas: 1003,1001,1002 Isr: 1003,1001
Topic: erw_plk Partition: 50 Leader: 1001 Replicas: 1001,1002,1003 Isr: 1003,1001,1002
Topic: erw_plk Partition: 51 Leader: 1001 Replicas: 1002,1001,1003 Isr: 1001,1003,1002
Topic: erw_plk Partition: 52 Leader: 1003 Replicas: 1003,1002,1001 Isr: 1003,1001
Topic: erw_plk Partition: 53 Leader: 1001 Replicas: 1001,1003,1002 Isr: 1003,1001,1002
Topic: erw_plk Partition: 54 Leader: 1003 Replicas: 1002,1003,1001 Isr: 1003,1001
Topic: erw_plk Partition: 55 Leader: 1003 Replicas: 1003,1001,1002 Isr: 1003,1001
Topic: erw_plk Partition: 56 Leader: 1001 Replicas: 1001,1002,1003 Isr: 1003,1001,1002
Topic: erw_plk Partition: 57 Leader: 1001 Replicas: 1002,1001,1003 Isr: 1001,1003,1002
Topic: erw_plk Partition: 58 Leader: 1003 Replicas: 1003,1002,1001 Isr: 1003,1001
Topic: erw_plk Partition: 59 Leader: 1001 Replicas: 1001,1003,1002 Isr: 1003,1001,1002
Topic: erw_plk Partition: 60 Leader: 1003 Replicas: 1002,1003,1001 Isr: 1003,1001
Topic: erw_plk Partition: 61 Leader: 1003 Replicas: 1003,1001,1002 Isr: 1003,1001
Topic: erw_plk Partition: 62 Leader: 1001 Replicas: 1001,1002,1003 Isr: 1003,1001,1002
Topic: erw_plk Partition: 63 Leader: 1001 Replicas: 1002,1001,1003 Isr: 1001,1003,1002
Topic: erw_plk Partition: 64 Leader: 1003 Replicas: 1003,1002,1001 Isr: 1003,1001
Topic: erw_plk Partition: 65 Leader: 1001 Replicas: 1001,1003,1002 Isr: 1003,1001,1002
Topic: erw_plk Partition: 66 Leader: 1003 Replicas: 1002,1003,1001 Isr: 1003,1001
Topic: erw_plk Partition: 67 Leader: 1003 Replicas: 1003,1001,1002 Isr: 1003,1001
Topic: erw_plk Partition: 68 Leader: 1001 Replicas: 1001,1002,1003 Isr: 1003,1001,1002
Topic: erw_plk Partition: 69 Leader: 1001 Replicas: 1002,1001,1003 Isr: 1001,1003,1002
Topic: erw_plk Partition: 70 Leader: 1003 Replicas: 1003,1002,1001 Isr: 1003,1001
Topic: erw_plk Partition: 71 Leader: 1001 Replicas: 1001,1003,1002 Isr: 1003,1001,1002
Topic: erw_plk Partition: 72 Leader: 1003 Replicas: 1002,1003,1001 Isr: 1003,1001
Topic: erw_plk Partition: 73 Leader: 1003 Replicas: 1003,1001,1002 Isr: 1003,1001
Topic: erw_plk Partition: 74 Leader: 1001 Replicas: 1001,1002,1003 Isr: 1003,1001,1002
Topic: erw_plk Partition: 75 Leader: 1001 Replicas: 1002,1001,1003 Isr: 1001,1003,1002
Topic: erw_plk Partition: 76 Leader: 1003 Replicas: 1003,1002,1001 Isr: 1003,1001
Topic: erw_plk Partition: 77 Leader: 1001 Replicas: 1001,1003,1002 Isr: 1003,1001,1002
Topic: erw_plk Partition: 78 Leader: 1003 Replicas: 1002,1003,1001 Isr: 1003,1001
Topic: erw_plk Partition: 79 Leader: 1003 Replicas: 1003,1001,1002 Isr: 1003,1001
Topic: erw_plk Partition: 80 Leader: 1001 Replicas: 1001,1002,1003 Isr: 1003,1001,1002
Topic: erw_plk Partition: 81 Leader: 1001 Replicas: 1002,1001,1003 Isr: 1001,1003,1002
Topic: erw_plk Partition: 82 Leader: 1003 Replicas: 1003,1002,1001 Isr: 1003,1001
Topic: erw_plk Partition: 83 Leader: 1001 Replicas: 1001,1003,1002 Isr: 1003,1001,1002
Topic: erw_plk Partition: 84 Leader: 1003 Replicas: 1002,1003,1001 Isr: 1003,1001
Topic: erw_plk Partition: 85 Leader: 1003 Replicas: 1003,1001,1002 Isr: 1003,1001
Topic: erw_plk Partition: 86 Leader: 1001 Replicas: 1001,1002,1003 Isr: 1003,1001,1002
Topic: erw_plk Partition: 87 Leader: 1001 Replicas: 1002,1001,1003 Isr: 1001,1003,1002
Topic: erw_plk Partition: 88 Leader: 1003 Replicas: 1003,1002,1001 Isr: 1003,1001
Topic: erw_plk Partition: 89 Leader: 1001 Replicas: 1001,1003,1002 Isr: 1003,1001,1002
Topic: erw_plk Partition: 90 Leader: 1003 Replicas: 1002,1003,1001 Isr: 1003,1001
Topic: erw_plk Partition: 91 Leader: 1003 Replicas: 1003,1001,1002 Isr: 1003,1001
Topic: erw_plk Partition: 92 Leader: 1001 Replicas: 1001,1002,1003 Isr: 1003,1001,1002
Topic: erw_plk Partition: 93 Leader: 1001 Replicas: 1002,1001,1003 Isr: 1001,1003,1002
Topic: erw_plk Partition: 94 Leader: 1003 Replicas: 1003,1002,1001 Isr: 1003,1001
Topic: erw_plk Partition: 95 Leader: 1001 Replicas: 1001,1003,1002 Isr: 1003,1001,1002
Topic: erw_plk Partition: 96 Leader: 1003 Replicas: 1002,1003,1001 Isr: 1003,1001
Topic: erw_plk Partition: 97 Leader: 1003 Replicas: 1003,1001,1002 Isr: 1003,1001
Topic: erw_plk Partition: 98 Leader: 1001 Replicas: 1001,1002,1003 Isr: 1003,1001,1002
Topic: erw_plk Partition: 99 Leader: 1001 Replicas: 1002,1001,1003 Isr: 1001,1003,1002
Topic:loe_sd PartitionCount:6 ReplicationFactor:3 Configs:
Topic: loe_sd Partition: 0 Leader: 1001 Replicas: 1002,1001,1003 Isr: 1001,1003,1002
Topic: loe_sd Partition: 1 Leader: 1003 Replicas: 1003,1002,1001 Isr: 1003,1001
Topic: loe_sd Partition: 2 Leader: 1001 Replicas: 1001,1003,1002 Isr: 1003,1001,1002
Topic: loe_sd Partition: 3 Leader: 1001 Replicas: 1002,1003,1001 Isr: 1001,1003,1002
Topic: loe_sd Partition: 4 Leader: 1003 Replicas: 1003,1001,1002 Isr: 1003,1001
Topic: loe_sd Partition: 5 Leader: 1001 Replicas: 1001,1002,1003 Isr: 1003,1001,1002
... View more
Labels:
09-28-2018
06:55 AM
hi all we have ambari cluster , version 2.6.x , with 3 kafka machine and 3 zookeper servers by the command ls /brokers/ids , we need to get all brokers ids as 1011 , 1012 , 1013 but in our case we get only the brokers id's - 1013 , 1012 what chould be the problem ? cd /usr/hdp/current/zookeeper-server/bin
You have mail in /var/spool/mail/root
[root@master01 bin]# ./zkCli.sh
Connecting to localhost:2181
Welcome to ZooKeeper!
JLine support is enabled
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
[zk: localhost:2181(CONNECTED) 0] ls /brokers/ids
[1013, 1012]
... View more
Labels:
- Labels:
-
Apache Ambari
-
Apache Kafka
09-13-2018
11:25 AM
@Tarun , can we ask you to post article about this ? , to explain what we should set as you said here , I think this will be useful to explain all aspects because some users think that the number of thread in lscpu needed to consider to calculate the Vcore
... View more
09-13-2018
11:06 AM
@Tarun , sorry , yes you rught all cluster are twice then the Vcore value , its my fault
... View more
09-13-2018
10:31 AM
@Tarun , so the final answer is to set 80% from 32 second , I will give you interesting info some of our ambari clusters the scrollbar is limit as the vcore ( for example if we have 16 vcore then scroll bar is until 16 and some cluster the scrolbar is twice then the real vcore ( for example if we have 16 vcore the scroll bar is 32 ) so how you explain this?
... View more
09-13-2018
10:00 AM
@Tarun , first thank you , but I will give you interesting example we have ambari cluster with 3 datanode each data node have 32 Vcore with 2 thread ( as we get from lscpu ) so according to your answer we need to set 32 to yarn.nodemanager.resource.cpu-vcores but as you can see from the following picture the scrollbar is until 64Vcore , and seems that during ambari installation , ambari calculate 2 X 32 and that gives us the scroll-bar to 64 so regarding my example , do you still with opinion that we need to set 80% of 32 ?
... View more
09-13-2018
07:26 AM
@Jay , can you look on my question on thred - https://community.hortonworks.com/questions/221835/how-to-calculate-the-yarnnodemanagerresourcecpu-vc-1.html , I will really apritiate your opinion
... View more