M3 Components

M3 consists of 3 simple components, all purposefully designed for scale, reliability and efficiency.

Ingestion & streaming aggregation

Ingest and aggregate a stream of metrics with dynamic configuration
Timeseries Database

Purpose built time series database (M3DB) for better performance, reliability and consistent results
Real time query engine

Real time, highly performant and optimized query engine

Getting Started

History

Intro

In 2014, following several years of battling a legacy metrics platform based on Graphite and WhisperDB, the Observability team at Uber began developing an open source metrics platform that would become known as M3. The goal of the project was to provide teams with a highly available and centralized metrics platform that could be easily integrated with existing tooling.

Storage

The first iteration of the storage layer featured Cassandra and ElasticSearch; however, as the ride-hailing business entered its hyper growth phase, the firefighting that had once plagued the team during the Graphite and WhisperDB years had resurfaced. The team then decided to build M3DB, a custom timeseries database with an embedded inverted index, from scratch which became the foundation of the platform. Over the course of about a year and a half, M3DB was developed and rolled out across Uber.

Learn more

Query

OOMs and slow queries became a regular occurrence, which prevented teams from effectively monitoring their services. The new query engine, dubbed M3Query, takes advantage of the fact that M3DB stores data in highly compressed blocks, and decompresses the data client side one datapoint at a time while functions are lazily applied. These optimizations minimize network, compute and memory resources. The new query engine, which supports PromQL and the legacy Graphite language, gave teams an extremely fast and powerful way to gain insights into their metrics.

Learn more

Ingestion

As the number of metrics being ingested skyrocketed, it became clear that there wasn’t a one-size-fits-all methodology when it came to data storage policies. Different teams wanted their metrics stored at different resolutions and retention periods. For example, infrastructure teams needed metrics stored at very high resolutions, but not for too long. Conversely, business teams didn’t need fine granularities, but wanted to retain data for much longer. The solution was a combination of two components, M3Coordinator and M3Aggregator, which work in tandem to ingest metrics and aggregate them in a highly customizable way all while providing high availability and efficiency.

Learn more

Today

Today, the M3 project is maintained by open source contributors around the globe, and is currently used by some of the largest enterprises in the world.

Getting Started