Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

how to set permanent kafka retention bytes on different topics

Solved Go to solution
Highlighted

how to set permanent kafka retention bytes on different topics

we set `retention bytes` value - `104857600` for topic - `topic_test`


[root@confluent01 ~]# kafka-topics --zookeeper localhost:2181 --alter --topic topic_test --config retention.bytes=104857600
WARNING: Altering topic configuration from this script has been deprecated and may be removed in future releases.
Going forward, please use kafka-configs.sh for this functionality
Updated config for topic "topic_test".


Now we verify the `retention bytes` from the zookeeper:

[root@confluent01 ~]# zookeeper-shell confluent01:2181 get /config/topics/topic_test
Connecting to confluent1:2181
{"version":1,"config":{"retention.bytes":"104857600"}}
cZxid = 0xb30a00000038

WATCHER::

WatchedEvent state:SyncConnected type:None path:null
ctime = Mon Jun 29 11:42:30 GMT 2020
mZxid = 0xb31100008978
mtime = Wed Jul 22 19:22:20 GMT 2020
pZxid = 0xb30a00000038
cversion = 0
dataVersion = 7
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 54
numChildren = 0

now we performed `reboot` to the kafka confluent01 machine


after machines started and kafka service start successfully , we checked again the `retention-bytes` from zookeeper:

but now ( after machine reboot ) we can see that the `retention bytes` isn't configured in zookeeper

[root@confluent01 ~]#zookeeper-shell confluent01:2181 get /config/topics/topic_test
Connecting to confluent1:2181

WATCHER::

WatchedEvent state:SyncConnected type:None path:null  no retention bytes value

{"version":1,"config":{}}
cZxid = 0xb30a00000038
ctime = Mon Jun 29 11:42:30 GMT 2020
mZxid = 0xb3110000779b
mtime = Wed Jul 22 14:09:19 GMT 2020
pZxid = 0xb30a00000038
cversion = 0
dataVersion = 2
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 25
numChildren = 0


**the question is** - how to remain the `retention bytes` even after restart of kafka machine ?

 

***NOTE - we not want to use the retention bytes from `server.properties`
because we set different retention bytes to each topic***

Michael-Bronson
1 ACCEPTED SOLUTION

Accepted Solutions

Re: how to set permanent kafka retention bytes on different topics

Mentor

@mike_bronson7 

log.retention.bytes is a size-based retention policy for logs, i.e the allowed size of the topic. Segments are pruned from the log as long as the remaining segments don't drop below log.retention.bytes.

You can also specify retention parameters at the topic level

To specify a retention time period per topic, use the following command.

 

kafka-configs.sh --zookeeper [ZooKeeperConnectionString] --alter --entity-type topics --entity-name [TopicName] --add-config retention.ms=[DesiredRetentionTimePeriod]

 

To specify a retention log size per topic, use the following command.

 

 

kafka-configs.sh --zookeeper [ZooKeeperConnectionString] --alter --entity-type topics --entity-name [TopicName] --add-config retention.bytes=[DesiredRetentionLogSize]

 

 

That should resolve your problem

 

Happy hadooping

 

 

 

View solution in original post

1 REPLY 1

Re: how to set permanent kafka retention bytes on different topics

Mentor

@mike_bronson7 

log.retention.bytes is a size-based retention policy for logs, i.e the allowed size of the topic. Segments are pruned from the log as long as the remaining segments don't drop below log.retention.bytes.

You can also specify retention parameters at the topic level

To specify a retention time period per topic, use the following command.

 

kafka-configs.sh --zookeeper [ZooKeeperConnectionString] --alter --entity-type topics --entity-name [TopicName] --add-config retention.ms=[DesiredRetentionTimePeriod]

 

To specify a retention log size per topic, use the following command.

 

 

kafka-configs.sh --zookeeper [ZooKeeperConnectionString] --alter --entity-type topics --entity-name [TopicName] --add-config retention.bytes=[DesiredRetentionLogSize]

 

 

That should resolve your problem

 

Happy hadooping

 

 

 

View solution in original post

Don't have an account?
Coming from Hortonworks? Activate your account here