Navigation :

Mapping Rules

Mapping rules are used to configure the storage policy for metrics. The storage policy determines how long to store metrics for and at what resolution to keep them at. For example, a storage policy of 1m:48h tells M3 to keep the metrics for 48hrs at a 1min resolution. Mapping rules can be configured in the m3coordinator configuration file under the downsample > rules > mappingRules stanza. We will use the following as an example.

downsample:
  rules:
    mappingRules:
      - name: "mysql metrics"
        filter: "app:mysql*"
        aggregations: ["Last"]
        storagePolicies:
          - resolution: 1m
            retention: 48h
      - name: "nginx metrics"
        filter: "app:nginx*"
        aggregations: ["Last"]
        storagePolicies:
          - resolution: 30s
            retention: 24h
          - resolution: 1m
            retention: 48h

Here, we have two mapping rules configured – one for mysql metrics and one for nginx metrics. The filter determines what metrics each rule applies to. The mysql metrics rule will apply to any metrics where the app tag contains mysql* as the value (* being a wildcard). Similarly, the nginx metrics rule will apply to all metrics where the app tag contains nginx* as the value.

The aggregations field determines what functions to apply to the datapoints within a resolution tile. For example, if an application emits a metric every 10sec and the resolution for that metrics’s storage policy is 1min, M3 will need to combine 6 datapoints. If the aggregations policy is Last, M3 will take the last value in that 1min bucket. aggregations can be one of the following:

Last
Min
Max
Mean
Median
Count
Sum
SumSq
Stdev
P10
P20
P30
P40
P50
P60
P70
P80
P90
P95
P99
P999
P9999

Lastly, the storagePolicies field determines which namespaces to store the metrics in. For example, the mysql metrics will be sent to the 1m:48h namespace, while the nginx metrics will be sent to both the 1m:48h and 30s:24h namespaces.

Note: the namespaces listed under the storagePolicies stanza must exist in M3DB.

Rollup Rules

Rollup rules are used to rollup metrics and aggregate in different ways by arbitrary dimensions before they are stored.

Aggregating counters example

Here’s an example of creating a new monotonic counter called http_request_rollup_no_pod_bucket from a set of histogram metrics originally called http_request_bucket:

downsample:
  rules:
    rollupRules:
      - name: "http_request latency by route and git_sha without pod"
        filter: "__name__:http_request_bucket k8s_pod:* le:* git_sha:* route:*"
        transforms:
        - transform:
            type: "Increase"
        - rollup:
            metricName: "http_request_rollup_no_pod_bucket"
            groupBy: ["le", "git_sha", "route", "status_code", "region"]
            aggregations: ["Sum"]
        - transform:
            type: "Add"
        storagePolicies:
        - resolution: 30s
          retention: 720h

Note: only metrics that contain all of the group_by tags will be rolled up. For example, in the above config, only http_request_bucket metrics that have all of the group_by labels present will be rolled up into the new metric http_request_rollup_no_pod_bucket.

While the above example can be used to create a new rolled up metric, often times the goal of rollup rules is to eliminate the underlaying, raw metrics. In order to do this, a mappingRule will need to be added like in the following example (using the metric above as an example) with drop set to true. Additionally, if all of the underlaying metrics are being dropped, there is no need to change the metric name (e.g. in the rollupRule, the metricName field can be equal to the existing metric) – see below for an example.

downsample:
  rules:
    mappingRules:
      - name: "http_request latency by route and git_sha drop raw"
        filter: "__name__:http_request_bucket k8s_pod:* le:* git_sha:* route:*"
        drop: true
    rollupRules:
      - name: "http_request latency by route and git_sha without pod"
        filter: "__name__:http_request_bucket k8s_pod:* le:* git_sha:* route:*"
        transforms:
        - transform:
            type: "Increase"
        - rollup:
            metricName: "http_request_bucket" # metric name doesn't change
            groupBy: ["le", "git_sha", "route", "status_code", "region"]
            aggregations: ["Sum"]
        - transform:
            type: "Add"
        storagePolicies:
        - resolution: 30s
          retention: 720h

Storage policies and rollup rules

Note: In order to store rolled up metrics in an unaggregated namespace, the namespace’s aggregationOptions must have a matching aggregation. For example, if in the above rule, the 720h namespace under storagePolicies is unaggregated, the aggregationOptions for that namespace should resemble the following:

"aggregationOptions": {
  "aggregations": [
    {
      "aggregated": false
    },
    {
      "aggregated": true,
      "attributes": {
        "resolutionDuration": "30s",
        "downsampleOptions": { "all": false }
      }
    }
  ]
}

Aggregating gauges example

The following is an example of a sensible set of aggregations across an example metric which represents a job queue length. The aggregations provide the sum, average, max and min across all instances for the job queue length with different aggregate metric names.

downsample:
  rules:
    rollupRules:
      - name: "job queue length sum across pods pod"
        filter: "__name__:job_queue_length k8s_pod:*"
        transforms:
        - aggregate:
            type: "Last"
        - rollup:
            metricName: "job_queue_length:sum"
            excludeBy: ["k8s_pod"]
            aggregations: ["Sum"]
        storagePolicies:
        - resolution: 30s
          retention: 720h
      - name: "job queue length average across pods pod"
        filter: "__name__:job_queue_length k8s_pod:*"
        transforms:
        - aggregate:
            type: "Last"
        - rollup:
            metricName: "job_queue_length:avg"
            excludeBy: ["k8s_pod"]
            aggregations: ["Mean"]
        storagePolicies:
        - resolution: 30s
          retention: 720h
      - name: "job queue length max across pods pod"
        filter: "__name__:job_queue_length k8s_pod:*"
        transforms:
        - aggregate:
            type: "Last"
        - rollup:
            metricName: "job_queue_length:max"
            excludeBy: ["k8s_pod"]
            aggregations: ["Max"]
        storagePolicies:
        - resolution: 30s
          retention: 720h
      - name: "job queue length min across pods pod"
        filter: "__name__:job_queue_length k8s_pod:*"
        transforms:
        - aggregate:
            type: "Last"
        - rollup:
            metricName: "job_queue_length:min"
            excludeBy: ["k8s_pod"]
            aggregations: ["Min"]
        storagePolicies:
        - resolution: 30s
          retention: 720h