question Re: kafka upgrade from 0.9 to 0.10 in HDP 2.6.0 in Archives of Support Questions (Read Only)

kafka upgrade from 0.9 to 0.10 in HDP 2.6.0

MarceloSaiedOK — Wed, 31 Jan 2018 02:11:57 GMT

HI,

ive upgrades HDP from 2.4 to 2.5 to 2.6. now kafka is at 0.10.1 . im have Kerberos configured and working with Ranger
I can create , produce , and consuming with zookeeper ( usinf kinit and a user that have permissions at Ranger rule)

WORKING OK
./kafka-console-producer.sh --broker-list ctl-t01.my.domain:6667,ctl-t02.my.domain:6667,ctl-t01.my.domain:6667 --topic bptopic --security-protocol PLAINTEXTSASL
./kafka-console-consumer.sh --bootstrap-server ctl-t01.my.domain:2181:6667 --topic bptopic --security-protocol PLAINTEXTSASL --from-beginning

when i use the Brokers instead of zookeper , the consumer , is not consuming

NOT WORKING

./kafka-console-consumer.sh --zookeeper  ctl-t01.my.domain:2181 --topic bptopic--security-protocol PLAINTEXTSASL --from-beginning

i have cheked logs , and documentation , without any solution.

Could be , that i need to delete at zookepper rmr /brokers? ( im not sure, but in another environment , that was the solution. deleting all the topics..). i want to see if i can keep all the topics , since we keep them for more than a week, and deleting them will be the last resource.

thanks much

@Ian Roberts

Re: kafka upgrade from 0.9 to 0.10 in HDP 2.6.0

MarceloSaiedOK — Thu, 01 Feb 2018 00:13:21 GMT

seem to be that deleting the brokers at zookeper do the trick .
yet all the topic need to be recreated, and the data of the topics will be lost

https://issues.apache.org/jira/browse/KAFKA-4987

Re: kafka upgrade from 0.9 to 0.10 in HDP 2.6.0

MarceloSaiedOK — Tue, 06 Feb 2018 00:52:11 GMT

First backup some metadata related to your topics

su - kafka

>sript>
for i in
`/usr/hdp/current/kafka-broker/bin/kafka-consumer-groups.sh --zookeeper server-02:2181  --list --security-protocol
PLAINTEXTSASL 2>&1 ` 
do 
     echo $i
     /usr/hdp/current/kafka-broker/bin/kafka-consumer-groups.sh --zookeeper server-01:2181 --group $i --describe 
done
<script<

describe topics

/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --describe --zookeeper sr-hadctl-xt01:2181 --topic bptopic

now remove the brokers metadata fron zookeeper

Delete topic (from zookeeper)

/usr/hdp/current/zookeeper-client/bin/zkCli.sh -server `hostname -f`:2181

rmr /brokers

Create Kafka tópic (at Broker server)

su - kafka

cd /usr/hdp/current/kafka-broker/bin

./kafka-topics.sh --create --zookeeper server-2:2181 --partitions 1 --replication-factor 1 --topic bptopic

Altering retention time

 OPTION 1
  [root@server-02 ~]#
  /usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper  server-01:2181 --alter --topic bptopic
  --config retention.ms=86400000

  WARNING:
  Altering topic configuration from this script has been deprecated and may be
  removed in future releases.

           Going forward, please use
  kafka-configs.sh for this functionality

  Updated
  config for topic "bptopic".

  OPTION 2
  [root@server-02 ~]# /usr/hdp/current/kafka-broker/bin/kafka-configs.sh
  --zookeeper  sr-hadctl-xt01:2181
  --alter --entity-type topics --entity-name bptopic --add-config
  'retention.ms=864000

  Check config
  [root@server-02 ~]# /usr/hdp/current/kafka-broker/bin/kafka-topics.sh
  --describe --zookeeper sr-hadctl-xt01:2181 --topic bptopic
  Topic:bptopic   PartitionCount:1        ReplicationFactor:1     Configs:retention.ms=86400000
          Topic: bptopic  Partition: 0    Leader: 1001    Replicas: 1001  Isr: 1001

Re: kafka upgrade from 0.9 to 0.10 in HDP 2.6.0

MarceloSaiedOK — Tue, 06 Feb 2018 23:28:07 GMT

There is a way to do the upgrade, and keeping the kafka-logs data?

@hgalante