Kafka offset topic not following retention policy

manoj_ramakrish — Sat, 30 Apr 2016 01:10:11 GMT

Most topics (created by our app) are following the policy(deletes the data after 2 days). But, there is one which is internal to kafka which is used for consumer offset position(We use the client api to commit the offset). This is not being deleted after 2 days. kakfka broker version: 2.3.4.0-3485. Here are the retention policy values:

Converted the server.properties from kafka config to a txt and attaching it here:

Kafka broker retention policy configuration(from server.properties)server.txt:

############################# Log Retention Policy ############################# #

The following configurations control the disposal of log segments. The policy can # be set to delete segments after a period of time, or after a given size has accumulated. # A segment will be deleted whenever *either* of these criteria are met. Deletion always happens # from the end of the log. # The minimum age of a log file to be eligible for deletion log.retention.hours=48 # A size-based retention policy for logs. Segments are pruned from the log as long as the remaining # segments don't drop below log.retention.bytes. #log.retention.bytes=1073741824 # The maximum size of a log segment file. When this size is reached a new log segment will be created. log.segment.bytes=536870912 # The interval at which log segments are checked to see if they can be deleted according # to the retention policies log.retention.check.interval.ms=300000 # By default the log cleaner is disabled and the log retention policy will default to just delete segments after their retention expires. # If log.cleaner.enable=true is set the cleaner will be enabled and individual logs can then be marked for log compaction. log.cleaner.enable=false

Re: Kafka offset topic not following retention policy

bleonhardi — Sat, 30 Apr 2016 01:20:44 GMT

There is actually a specific parameter for offset retention

offsets.retention.minutes which by default is 1440 or one day. So it actually should be deleted sooner.

http://kafka.apache.org/documentation.html

question Re: Kafka offset topic not following retention policy in Archives of Support Questions (Read Only)

Kafka offset topic not following retention policy

Re: Kafka offset topic not following retention policy