Database Reference
In-Depth Information
requests
#num.replica.fetchers=1
# The frequency with with the high watermark is
flushed to disk
#replica.high.watermark.checkpoint.interval.ms=5000
ThenextgroupofsettingscontrolsthebehaviorofKafka'sdiskinput/output
(I/O). These settings are used to balance the durability of the data when it's
not using replication with Kafka's throughput for producers and consumers.
The two parameters,
log.flush.interval.messages
and
log.flush
.interval.ms
control the minimum and maximum times between flush
events. The
log.flush.interval.ms
setting is the longest Kafka waits
until flushing to disk, thereby defining an upper bound for flush events.
The
log.flush .interval.messages
parameter causes Kafka to flush
messages to disk after a number of messages have been received. If the
partition collects messages faster than the interval time, this defines the
lower bound on the flush rate.
Without replication, the time between flush events is the amount of data
that can be lost in the event of a broker failure. The default is 10,000
messages or 1,000 milliseconds (1 second), which is typically sufficient for
most use cases. Before setting these values to be larger or smaller, there are
two things to consider: throughput and latency. The process of flushing to
disk is the most expensive operation performed by Kafka, and increasing
this rate reduces the throughput of the system as a whole. Conversely, if
the flush is very large, it takes a relatively long time. Increasing the time
between flushes can lead to latency spikes in responses to producers and
consumers as a large bolus of data is flushed to disk.
You can control both these parameters on a per-topic basis using the
log
.flush.intervals.messages.per.topic
and
log.flush.intervals.ms.per .topic
parameters respectively.
These parameters take a comma-separated list of
<topic>:<value>
tuples.
# The number of messages to accept before forcing
# a flush of data to disk
log.flush.interval.messages=10000
# The maximum amount of time a message can sit in a
