Is it possible to force the cleaner to compact a partition log for partitions with very low traffic?
Is it possible to force the cleaner to compact a partition log for partitions with very low traffic?
For a topic with retention.policy
set to "compact, delete"
it is understood that compaction (and for null records - deletion) takes place when a cleaner thread decides to clean a log. This decision is based on a few things. Relevant to this question is the segment roll functionality; unless a new segment is created compaction won't run.
Segment roll may be configured via segment.ms
and segment.bytes
.
Now, for the question. Since active segments won't be cleaned and a new segment doesn't become active until a record has been written to it, is it possible to force the cleaner to compact a partition log for topics that no longer receive any write-traffic?
Example log:
$ kcat -b kafka:9092 -t foo -C -K:
1:hello
2:world
1:
Regardless of configuration, unless a records is written after 1:null
compaction does not run. However:
$ echo "3:compact" | kcat -b kafka:9092 -t foo -P -K:
# `segment.ms` time passes
$ kcat -b kafka:9092 -t foo -C -K:
2:world
1:
3:compact
Solution 1:
I've investigated further over at the Confluent Community and got it verified that a segment rolls when the timestamp of a written record differs more than segment.ms from the first record in the currently active segment.
In other words, a segment never rolls unless a record is written. So, the answer to the question is "no, it is not possible".