Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

kafka upgrade from 0.9 to 0.10 in HDP 2.6.0

avatar
Rising Star

HI,

ive upgrades HDP from 2.4 to 2.5 to 2.6. now kafka is at 0.10.1 . im have Kerberos configured and working with Ranger
I can create , produce , and consuming with zookeeper ( usinf kinit and a user that have permissions at Ranger rule)

WORKING OK
./kafka-console-producer.sh --broker-list ctl-t01.my.domain:6667,ctl-t02.my.domain:6667,ctl-t01.my.domain:6667 --topic bptopic --security-protocol PLAINTEXTSASL
./kafka-console-consumer.sh --bootstrap-server ctl-t01.my.domain:2181:6667 --topic bptopic --security-protocol PLAINTEXTSASL --from-beginning

when i use the Brokers instead of zookeper , the consumer , is not consuming

NOT WORKING

./kafka-console-consumer.sh --zookeeper  ctl-t01.my.domain:2181 --topic bptopic--security-protocol PLAINTEXTSASL --from-beginning

i have cheked logs , and documentation , without any solution.

Could be , that i need to delete at zookepper rmr /brokers? ( im not sure, but in another environment , that was the solution. deleting all the topics..). i want to see if i can keep all the topics , since we keep them for more than a week, and deleting them will be the last resource.

thanks much

@Ian Roberts

1 ACCEPTED SOLUTION

avatar
Rising Star

seem to be that deleting the brokers at zookeper do the trick .
yet all the topic need to be recreated, and the data of the topics will be lost

https://issues.apache.org/jira/browse/KAFKA-4987

View solution in original post

3 REPLIES 3

avatar
Rising Star

seem to be that deleting the brokers at zookeper do the trick .
yet all the topic need to be recreated, and the data of the topics will be lost

https://issues.apache.org/jira/browse/KAFKA-4987

avatar
Rising Star

First backup some metadata related to your topics

su - kafka

>sript>
for i in
`/usr/hdp/current/kafka-broker/bin/kafka-consumer-groups.sh --zookeeper server-02:2181  --list --security-protocol
PLAINTEXTSASL 2>&1 ` 
do 
     echo $i
     /usr/hdp/current/kafka-broker/bin/kafka-consumer-groups.sh --zookeeper server-01:2181 --group $i --describe 
done
<script<



describe topics

/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --describe --zookeeper sr-hadctl-xt01:2181 --topic bptopic

now remove the brokers metadata fron zookeeper


  • Delete topic (from zookeeper)
/usr/hdp/current/zookeeper-client/bin/zkCli.sh -server `hostname -f`:2181

rmr /brokers
  • Create Kafka tópic (at Broker server)
su - kafka

cd /usr/hdp/current/kafka-broker/bin

./kafka-topics.sh --create --zookeeper server-2:2181 --partitions 1 --replication-factor 1 --topic bptopic

Altering retention time

 OPTION 1
  [root@server-02 ~]#
  /usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper  server-01:2181 --alter --topic bptopic
  --config retention.ms=86400000

  WARNING:
  Altering topic configuration from this script has been deprecated and may be
  removed in future releases.

           Going forward, please use
  kafka-configs.sh for this functionality

  Updated
  config for topic "bptopic".

  OPTION 2
  [root@server-02 ~]# /usr/hdp/current/kafka-broker/bin/kafka-configs.sh
  --zookeeper  sr-hadctl-xt01:2181
  --alter --entity-type topics --entity-name bptopic --add-config
  'retention.ms=864000
  Check config
  [root@server-02 ~]# /usr/hdp/current/kafka-broker/bin/kafka-topics.sh
  --describe --zookeeper sr-hadctl-xt01:2181 --topic bptopic
  Topic:bptopic   PartitionCount:1        ReplicationFactor:1     Configs:retention.ms=86400000
          Topic: bptopic  Partition: 0    Leader: 1001    Replicas: 1001  Isr: 1001

avatar
Rising Star

There is a way to do the upgrade, and keeping the kafka-logs data?

@hgalante