Member since
03-23-2017
103
Posts
2
Kudos Received
5
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
406 | 06-01-2017 09:30 AM | |
253 | 05-31-2017 08:41 AM | |
395 | 05-15-2017 06:15 AM | |
334 | 05-11-2017 02:40 AM | |
162 | 05-11-2017 02:36 AM |
08-14-2019
11:33 AM
We have a HDF 3.1 cluster where the users have to use the brokers NAT IP instead of real IP. I have added below. listeners=INTERNAL://x.x.x.x:6667,EXTERNAL://x.x.x.x:9092 listener.security.protocol.map=INTERNAL:PLAINTEXTSASL,EXTERNAL:PLAINTEXTSASL advertised.listeners=INTERNAL://xxxxxx.corp:6667,EXTERNAL://xxxxxxx.corp:9092 inter.broker.listener.name=INTERNAL but since security.inter.broker.protocol and inter.broker.listener.name can't be set at the same time, i have removed inter.broker.listener.name and added INTERNAL for security.inter.broker.protocol this has caused beow error. Invalid security protocol `INTERNAL` defined in security.inter.broker.protocol
... View more
Labels:
07-28-2019
08:25 PM
Hi, I have two HDP clusters in different DC's with same domain name and same realm but the KDC's are different. How can i enable trust between these two KDC's ??
... View more
05-31-2019
09:05 AM
Hi, We have few flume agents which have kafka as channel and hdfs as sink. Recently, we added few more flume agents and all the sources have re-balanced and have detected new flume agents however the channel topics consumer group are not re-balancing with the new flume agents. Below is my channel configuration. TAAS.channels.taasChnl2.type = org.apache.flume.channel.kafka.KafkaChannel
TAAS.channels.taasChnl2.brokerList = kafka1.test.com:6667
TAAS.channels.taasChnl2.zookeeperConnect = server1.test.com:2181,server2.test.com:2181,server3.test.com:2181/kafka1
TAAS.channels.taasChnl2.topic = CHNL_LOG_TRANSFORM
TAAS.channels.taasChnl2.groupId = TAAS_FLUME_P_1
TAAS.channels.taasChnl2.parseAsFlumeEvent = true
... View more
Labels:
05-31-2019
08:44 AM
@Geoffrey Shelton Okot Yes, i'm aware of that. I just wanted to know the service that would require restart before that change so that we can plan accordingly.
... View more
05-30-2019
04:39 PM
What all services need to be restarted when a kafka broker is added in HDP?
... View more
04-01-2019
08:58 AM
If we run a simple select on an hbase table limiting the result to 1 row using phoenix, how many read request does the jmx metrics display and how is it calculated.
... View more
Labels:
10-27-2018
07:35 PM
Hi, We often see NOT_LEADER_FOR_PARTITION in kafka. On what basis does kafka leader election happen ? What are all the reasons that cause kafka partition leader election to trigger?
... View more
Labels:
06-01-2018
10:16 AM
Thanks @William Brooks will try that. From my initial checks, it seems working
... View more
05-31-2018
03:27 PM
I'm using kafka 0.11. I'm trying to reset the offset of a topic in a consumer group but don't see consistency among the option. How is the offset calculated for different options in kafka-consumer-groups ? user1@xxxxxxxxxxxx:~$ kafka-consumer-groups.sh --bootstrap-server xxxxxxxxxxxx.devkafka.com:9092 --group testgroup --reset-offsets--by-duration PT1H30M0S--topic Test_Upgrade --export --command-config client.properties 2>/dev/null Test_Upgrade,1,615 Test_Upgrade,0,625 user1@xxxxxxxxxxxx:~$ dateThu May 31 15:30:45 BST 2018 user1@xxxxxxxxxxx:~$ kafka-consumer-groups.sh --bootstrap-server xxxxxxxxxxxx.devkafka.com:9092 --group testgroup --reset-offsets--to-datetime 2018-05-31T14:00:00.000--topic Test_Upgrade --export --command-config client.properties 2>/dev/null Test_Upgrade,1,754 Test_Upgrade,0,772
... View more
Labels:
05-22-2018
02:25 PM
I had to clean my zookeeper data directory and recreate the __consumer_offsets topic
... View more
05-22-2018
09:37 AM
Only brokers 1002 and 1005 are available.. Others are old brokers which were decommissioned. Topic: __consumer_offsets Partition: 0 Leader: -1 Replicas: 33,1005,30 Isr: 30
Topic: __consumer_offsets Partition: 1 Leader: -1 Replicas: 1002,30,32 Isr: 30
Topic: __consumer_offsets Partition: 2 Leader: -1 Replicas: 1005,32,33 Isr: 32
Topic: __consumer_offsets Partition: 3 Leader: -1 Replicas: 30,33,1002 Isr: 30
Topic: __consumer_offsets Partition: 4 Leader: -1 Replicas: 32,1002,1005 Isr: 32
Topic: __consumer_offsets Partition: 5 Leader: -1 Replicas: 33,30,32 Isr: 32
Topic: __consumer_offsets Partition: 6 Leader: -1 Replicas: 1002,32,33 Isr: 32
Topic: __consumer_offsets Partition: 8 Leader: -1 Replicas: 30,1002,1005 Isr: 30
Topic: __consumer_offsets Partition: 9 Leader: -1 Replicas: 32,1005,30 Isr: 30
Topic: __consumer_offsets Partition: 10 Leader: -1 Replicas: 33,32,1002 Isr: 32
Topic: __consumer_offsets Partition: 12 Leader: -1 Replicas: 1005,1002,30 Isr: 30
Topic: __consumer_offsets Partition: 13 Leader: -1 Replicas: 30,1005,32 Isr: 30
Topic: __consumer_offsets Partition: 14 Leader: -1 Replicas: 32,30,33 Isr: 30
Topic: __consumer_offsets Partition: 16 Leader: -1 Replicas: 1002,1005,30 Isr: 30
Topic: __consumer_offsets Partition: 17 Leader: -1 Replicas: 1005,30,32 Isr: 30
Topic: __consumer_offsets Partition: 18 Leader: -1 Replicas: 30,32,33 Isr: 32
Topic: __consumer_offsets Partition: 19 Leader: -1 Replicas: 32,33,1002 Isr: 32
Topic: __consumer_offsets Partition: 20 Leader: -1 Replicas: 33,1005,30 Isr: 30
Topic: __consumer_offsets Partition: 21 Leader: -1 Replicas: 1002,30,32 Isr: 32
Topic: __consumer_offsets Partition: 22 Leader: -1 Replicas: 1005,32,33 Isr: 32
Topic: __consumer_offsets Partition: 23 Leader: -1 Replicas: 30,33,1002 Isr: 30
Topic: __consumer_offsets Partition: 24 Leader: -1 Replicas: 32,1002,1005 Isr: 32
Topic: __consumer_offsets Partition: 25 Leader: -1 Replicas: 33,30,32 Isr: 30
Topic: __consumer_offsets Partition: 26 Leader: -1 Replicas: 1002,32,33 Isr: 32
Topic: __consumer_offsets Partition: 28 Leader: -1 Replicas: 30,1002,1005 Isr: 30
Topic: __consumer_offsets Partition: 29 Leader: -1 Replicas: 32,1005,30 Isr: 30
Topic: __consumer_offsets Partition: 30 Leader: -1 Replicas: 33,32,1002 Isr: 32
Topic: __consumer_offsets Partition: 32 Leader: -1 Replicas: 1005,1002,30 Isr: 30
Topic: __consumer_offsets Partition: 33 Leader: -1 Replicas: 30,1005,32 Isr: 32
Topic: __consumer_offsets Partition: 34 Leader: -1 Replicas: 32,30,33 Isr: 30
Topic: __consumer_offsets Partition: 36 Leader: -1 Replicas: 1002,1005,30 Isr: 30
Topic: __consumer_offsets Partition: 37 Leader: -1 Replicas: 1005,30,32 Isr: 32
Topic: __consumer_offsets Partition: 38 Leader: -1 Replicas: 30,32,33 Isr: 32
Topic: __consumer_offsets Partition: 39 Leader: -1 Replicas: 32,33,1002 Isr: 32
Topic: __consumer_offsets Partition: 40 Leader: -1 Replicas: 33,1005,30 Isr: 30
Topic: __consumer_offsets Partition: 41 Leader: -1 Replicas: 1002,30,32 Isr: 30
Topic: __consumer_offsets Partition: 42 Leader: -1 Replicas: 1005,32,33 Isr: 32
Topic: __consumer_offsets Partition: 43 Leader: -1 Replicas: 30,33,1002 Isr: 30
Topic: __consumer_offsets Partition: 44 Leader: -1 Replicas: 32,1002,1005 Isr: 32
Topic: __consumer_offsets Partition: 45 Leader: -1 Replicas: 33,30,32 Isr: 30
Topic: __consumer_offsets Partition: 46 Leader: -1 Replicas: 1002,32,33 Isr: 32
Topic: __consumer_offsets Partition: 48 Leader: -1 Replicas: 30,1002,1005 Isr: 30
Topic: __consumer_offsets Partition: 49 Leader: -1 Replicas: 32,1005,30 Isr: 30
... View more
05-22-2018
09:35 AM
Hi, How can i delete __consumer_offsetsnd recreate it? Among the 50 partitions, morethan half are outoff sync. And i dont need any data from the existing topics.
... View more
Labels:
05-18-2018
08:47 AM
Hi, We are using kafka 0.11. Recently we had an issue with one of our topics and had to delete and recreate the topic. Is there a way to rename the topic or clone?
... View more
Labels:
05-02-2018
06:51 PM
Hi @Sandeep Nemuri That was helpful but what if i want to add additional disks rather than removing one and adding other??
... View more
05-02-2018
05:50 PM
Hi @Sandeep Nemuri would this be the same when i add additional dara directories to kafka?
... View more
05-02-2018
05:25 PM
Hi, We currently have 3 kafka brokers and about 250 topics. How do i re-balance the partitions and data when a new broker is added? Is running manual rebalance the only option? I was thinkin gof stopping a broker and then since we have 3 replications, it woul copy the topics in the stopped broker to new broker. Is that a good idea?
... View more
Labels:
05-02-2018
02:37 PM
Hi, How do we reset the offset when using Kafka bootstrap consumer??
... View more
Labels:
04-25-2018
08:42 AM
Hi, Is it possible to install HDF without SAM and schema registry?
... View more
Labels:
04-24-2018
09:53 AM
Hortonworks support said it supports 9.4.
... View more
04-23-2018
10:20 AM
@Geoffrey Shelton Okot Ofcourse, we did that and its working and there is no warning in the ambari-server-check-database.log about the database version but since its not mentioned in support matrix, we are on crossroads.
... View more
04-23-2018
09:21 AM
@Geoffrey Shelton Okot By any chance does running ambari on 9.5 have any issue? We have a test cluster with ambari running on 9.6.8 with no issues till now. Want to make sure it doesn't have issues if used i Production.
... View more
04-23-2018
08:59 AM
Hi @Geoffrey Shelton Okot Yes, I have seen that, We want to install HDF ans SAM, Schema registry require 9.5. So was wondering if ambari supports it or not rather than having two versions of PostGres.
... View more
04-23-2018
08:36 AM
Hi, Does Ambari support PostGres9.5 or later since 9.4 support is not available(after Dec2019)?
... View more
Labels:
04-06-2018
05:40 PM
@Jordan thanks for the info.
... View more
04-06-2018
07:28 AM
Yes @Jordan Moore i can use them. But i was actually trying to test it with HTTP (wget). Not sure if it can be tested!
... View more
04-05-2018
03:15 PM
Hi, I have created two different vip's for kafka brokers and zookeeper servers. How do i test if the VIP's are working? Does this work?
... View more
Labels:
11-23-2017
10:04 AM
@SIOKEK YIM Do you mean all my topic creation, producing and consuming has to be done with kafka user? I'm not sure if kafka has any impersonation..!! Tried doing using kafka keytab. Below is the topic created with kafka service keytab. But its the same issue Topic:Hello1 PartitionCount:2 ReplicationFactor:2 Configs:min.insync.replicas=2 Topic: Hello1 Partition: 0 Leader: 35 Replicas: 35,38 Isr: 35,38 Topic: Hello1 Partition: 1 Leader: 38 Replicas: 38,33 Isr: 38,33 Producer: =======kafka-verifiable-producer.sh --topic Hello1 --broker-list server1.kafka2.pre.corp:9092,server2.kafka2.pre.corp:9092 --producer.config client.properties 17/11/23 09:58:07 INFO utils.AppInfoParser: Kafka version : 0.10.2-kafka-2.2.0
17/11/23 09:58:07 INFO utils.AppInfoParser: Kafka commitId : unknown 1
2
3
4
45 17/11/23 09:58:25 INFO producer.KafkaProducer: Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms.
{"name":"shutdown_complete"}
{"sent":1,"name":"tool_data","avg_throughput":0.0,"target_throughput":-1,"acked":0} Consumer: kafka-console-consumer --topic Hello1 --from-beginning --zookeeper server.kafka2.pre.corp:2181,server.kafka2.pre.corp:2181/kafka2dc1pre --consumer.config consumer.properties 17/11/23 09:59:04 INFO utils.ZKCheckedEphemeral: Creating /consumers/console-consumer-59653/ids/console-consumer-59653_server1.kafka2.pre.corp-1511431144657-52e4b1e7 (is it secure? false) 17/11/23 09:59:04 INFO utils.ZKCheckedEphemeral: Result of znode creation is: OK . . 17/11/23 09:59:05 INFO consumer.ZookeeperConsumerConnector: [console-consumer-59653_server1.kafka2.pre.corp-1511431144657-52e4b1e7], end rebalancing consumer console-consumer-59653_server1.kafka2.pre.corp-1511431144657-52e4b1e7 try #0 17/11/23 09:59:05 INFO consumer.ZookeeperConsumerConnector: [console-consumer-59653_server1.kafka2.pre.corp-1511431144657-52e4b1e7], Creating topic event watcher for topics Hello1 17/11/23 09:59:05 INFO consumer.ZookeeperConsumerConnector: [console-consumer-59653_server1.kafka2.pre.corp-1511431144657-52e4b1e7], Topics to consume = ArrayBuffer(Hello1)
17/11/23 09:59:05 WARN consumer.ConsumerFetcherManager$LeaderFinderThread: [console-consumer-59653_server1.kafka2.pre.corp-1511431144657-52e4b1e7-leader-finder-thread], Failed to find leader for Set(Hello1-0, Hello1-1)
kafka.common.BrokerEndPointNotAvailableException: End point with security protocol PLAINTEXT not found for broker 33
at kafka.client.ClientUtils$anonfun$getPlaintextBrokerEndPoints$1$anonfun$apply$5.apply(ClientUtils.scala:146)
at kafka.client.ClientUtils$anonfun$getPlaintextBrokerEndPoints$1$anonfun$apply$5.apply(ClientUtils.scala:146)
at scala.Option.getOrElse(Option.scala:121)
at kafka.client.ClientUtils$anonfun$getPlaintextBrokerEndPoints$1.apply(ClientUtils.scala:146)
at kafka.client.ClientUtils$anonfun$getPlaintextBrokerEndPoints$1.apply(ClientUtils.scala:142)
at scala.collection.TraversableLike$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.TraversableLike$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at kafka.client.ClientUtils$.getPlaintextBrokerEndPoints(ClientUtils.scala:142)
at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:67)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63) 17/11/23 09:59:05 INFO consumer.ConsumerFetcherManager: [ConsumerFetcherManager-1511431144680] Added fetcher for partitions ArrayBuffer() 17/11/23 09:59:05 WARN consumer.ConsumerFetcherManager$LeaderFinderThread: [console-consumer-59653_server1.kafka2.pre.corp-1511431144657-52e4b1e7-leader-finder-thread], Failed to find leader for Set(Hello1-0, Hello1-1)
kafka.common.BrokerEndPointNotAvailableException: End point with security protocol PLAINTEXT not found for broker 33
at kafka.client.ClientUtils$anonfun$getPlaintextBrokerEndPoints$1$anonfun$apply$5.apply(ClientUtils.scala:146)
... View more
11-23-2017
09:26 AM
The issue actually seems to be with the kafka root in zookeeper. Initially the kafka root was / and then we changed it to /kafkapre in zookeeper. But kafka seems to be searching for / itself still. 2017-11-23 09:16:12,539 WARN nl.techop.kafka.dao.zookeeper.KafkaZkClient: KeeperErrorCode = NoNode for /brokers/topics/TEST/partitions/1/state 2017-11-23 09:17:42,548 WARN nl.techop.kafka.dao.zookeeper.KafkaZkClient: KeeperErrorCode = NoNode for /brokers/topics/TEST/partitions/1/state 2017-11-23 09:19:12,533 WARN nl.techop.kafka.dao.zookeeper.KafkaZkClient: KeeperErrorCode = NoNode for /brokers/topics/TEST/partitions/1/state
... View more
11-23-2017
09:20 AM
@SIOKEK YIM Do i need to be using kafka keytab??
... View more
11-22-2017
02:33 PM
@Aditya Sirna Thanks for the prompt reply. Yes, we did all the mentioned above.
... View more