Flushing is the process by which m3aggregator instances output the aggregated time series data (by using m3msg protocol). There are two targets to which the data is flushed:
- Persistence: the aggregated data is flushed to
m3coordinatorwhich then persists it in
- Forwarding: intermediate aggregation data is flushed to other nodes of
m3aggregator(in a multi-node setup) for further processing. This is necessary for rollup rule processing - eg. when a label is dropped, the resulting time series has a new id which potentially maps to a different shard (which may be owned by a different node).
Flushing coordination between leaders and followers
In a replicated setup, only the leader nodes are flushing the data (both persistence and forwarding). After each flush, for every combination of shard/metrics type/aggregation window the leader node updates last flushed timestamp in etcd.
Follower nodes are watching the changes of last flushed timestamps for the shardset that they own. The data that has been flushed by the leader node is considered as persisted, and the followers can discard the corresponding data they have accumulated for the given shard/metrics type/aggregation window.
If, for some reason, a follower node does not receive last flush timestamp updates from
the leader, it will keep the aggregated data for a while, but the follower will go into
forced flush mode
after the time period defined by
maxBufferSize and start discarding the
accumulated data in order to reclaim the memory. The default value for this setting is 5 minutes:
aggregator: flushManager: maxBufferSize: 5m
When a follower gets promoted to a leader, it initially flushes time windows it has buffered, following the last flush timestamp reported by the previous leader via etcd.
Cutoff/cutover are timestamps that can be used to control flushing from the given shard. They are fields that are defined for each shard in the placement structure.
If the shard has its cutover field set to some timestamp value, the shard will only start flushing once the wall clock will go past the given cutover timestamp. Until this happens, any aggregated data gets discarded.
Similarly, if the shard has its cutoff field set to some value, the shard will stop flushing once the wall clock will go past the given cutoff timestamp.
If the shard does not have cutover/cutoff fields it will flush indefinitely.