Database Reference
In-Depth Information
# log before we force a flush
log.flush.interval.ms=1000
# Per-topic overrides for log.flush.interval.ms
#log.flush.intervals.ms.per.topic=topic1:1000,
topic2:3000
The next set of parameters control the retention time for topics as well as
the size of log segments. Logs are always removed on deleting an entire
segment, so the size of the segment file combined with the retention time
defines how quickly expired data will be removed.
The basic parameter that controls the retention is the
log.retention.hours
setting. When a log segment exceeds this age it
is deleted. This happens on a per-partition basis. There is also a
log.retention.bytes
setting, but it has little utility in practice. First, as
discussed earlier, deleting log segments on space constraints is somewhat
more difficult to manage than time. Second,
log .retention.bytes
uses a 32-bit integer rather than Java's 64-bit long. This limits topics to at
most 2GB per-partition per-topic. Outside of development and testing, this
is rarely sufficiently large in a production environment.
# The minimum age of a log file to be eligible for
deletion
log.retention.hours=168
# A size-based retention policy for logs. Segments are
pruned from the
# log as long as the remaining segments don't drop
# below log.retention.bytes.
#log.retention.bytes=1073741824
The size of each segment is controlled by the
log.segment.bytes
and the
log .index.size.max.bytes
settings. The first parameter controls the
size of the log segment, which is the file that contains the actual messages as
they were sent to Kafka. The on-disk format is identical to the wire format
so that the messages can be quickly flushed to this disk. The default size of
this file is 1GB, and in the following example it has been set to 512MB for
development purposes.
The
log.index.size.max.bytes
parameter controls the size of the
index file that accompanies each log segment. This file is used to store
