how to set permanent kafka retention bytes on different topics

mike_bronson7 — Wed, 22 Jul 2020 22:25:13 GMT

we set `retention bytes` value - `104857600` for topic - `topic_test`

[root@confluent01 ~]# kafka-topics --zookeeper localhost:2181 --alter --topic topic_test --config retention.bytes=104857600
WARNING: Altering topic configuration from this script has been deprecated and may be removed in future releases.
Going forward, please use kafka-configs.sh for this functionality
Updated config for topic "topic_test".

Now we verify the `retention bytes` from the zookeeper:

[root@confluent01 ~]# zookeeper-shell confluent01:2181 get /config/topics/topic_test
Connecting to confluent1:2181
{"version":1,"config":{"retention.bytes":"104857600"}}
cZxid = 0xb30a00000038

WATCHER::

WatchedEvent state:SyncConnected type:None path:null
ctime = Mon Jun 29 11:42:30 GMT 2020
mZxid = 0xb31100008978
mtime = Wed Jul 22 19:22:20 GMT 2020
pZxid = 0xb30a00000038
cversion = 0
dataVersion = 7
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 54
numChildren = 0

now we performed `reboot` to the kafka confluent01 machine

after machines started and kafka service start successfully , we checked again the `retention-bytes` from zookeeper:

but now ( after machine reboot ) we can see that the `retention bytes` isn't configured in zookeeper

[root@confluent01 ~]#zookeeper-shell confluent01:2181 get /config/topics/topic_test
Connecting to confluent1:2181

WATCHER::

WatchedEvent state:SyncConnected type:None path:null  no retention bytes value

{"version":1,"config":{}}
cZxid = 0xb30a00000038
ctime = Mon Jun 29 11:42:30 GMT 2020
mZxid = 0xb3110000779b
mtime = Wed Jul 22 14:09:19 GMT 2020
pZxid = 0xb30a00000038
cversion = 0
dataVersion = 2
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 25
numChildren = 0

**the question is** - how to remain the `retention bytes` even after restart of kafka machine ?

***NOTE - we not want to use the retention bytes from `server.properties`
because we set different retention bytes to each topic***

Re: how to set permanent kafka retention bytes on different topics

Shelton — Sun, 26 Jul 2020 18:40:52 GMT

@mike_bronson7

log.retention.bytes is a size-based retention policy for logs, i.e the allowed size of the topic. Segments are pruned from the log as long as the remaining segments don't drop below log.retention.bytes.

You can also specify retention parameters at the topic level

To specify a retention time period per topic, use the following command.

kafka-configs.sh --zookeeper [ZooKeeperConnectionString] --alter --entity-type topics --entity-name [TopicName] --add-config retention.ms=[DesiredRetentionTimePeriod]

To specify a retention log size per topic, use the following command.

kafka-configs.sh --zookeeper [ZooKeeperConnectionString] --alter --entity-type topics --entity-name [TopicName] --add-config retention.bytes=[DesiredRetentionLogSize]

That should resolve your problem

Happy hadooping

question Re: how to set permanent kafka retention bytes on different topics in Support Questions

how to set permanent kafka retention bytes on different topics

Re: how to set permanent kafka retention bytes on different topics