Creating a Single Node M3DB Cluster with Docker
This guide shows how to install and configure M3DB, create a single-node cluster, and read and write metrics to it.
Deploying a single-node M3DB cluster is a great way to experiment with M3DB and get an idea of what it has to offer, but is not designed for production use. To run M3DB in clustered mode with a separate M3Coordinator, read the clustered mode guide.
Prerequisites
- Docker: You don’t need Docker to run M3DB, but it is the simplest and quickest way.
- If you use Docker Desktop, we recommend the following minimum Resources settings.
- CPUs: 2
- Memory: 8GB
- Swap: 1GB
- Disk image size: 16GB
- If you use Docker Desktop, we recommend the following minimum Resources settings.
- JQ: This example uses jq to format the output of API calls. It is not essential for using M3DB.
- curl: This example uses curl for communicating with M3DB endpoints. You can also use alternatives such as Wget and HTTPie.
Start Docker Container
By default the official M3DB Docker image configures a single M3DB instance as one binary containing:
- An M3DB storage instance for time series storage. It includes an embedded tag-based metrics index and an etcd server for storing the cluster topology and runtime configuration.
- A coordinator instance for writing and querying tagged metrics, as well as managing cluster topology and runtime configuration.
The Docker container exposes three ports:
7201
to manage the cluster topology, you make most API calls to this endpoint7203
for Prometheus to scrape the metrics produced by M3DB and M3Coordinator
The command below creates a persistent data directory on the host operating system to maintain durability and persistence between container restarts.
docker run -p 7201:7201 -p 7203:7203 --name m3db -v $(pwd)/m3db_data:/var/lib/m3db quay.io/m3db/m3dbnode:v1.0.0
When running the command above on Docker for Mac, Docker for Windows, and some Linux distributions you may see errors about settings not being at recommended values. Unless you intend to run M3DB in production on macOS or Windows, you can ignore these warnings.
Configuration
The single-node cluster Docker image uses this sample configuration file by default.
The file groups configuration into coordinator
or db
sections that represent the M3Coordinator
and M3DB
instances of single-node cluster.
You can find more information on configuring M3DB in the operational guides section.
Organizing Data with Placements and Namespaces
A time series database (TSDB) typically consist of one node (or instance) to store metrics data. This setup is simple to use but has issues with scalability over time as the quantity of metrics data written and read increases.
As a distributed TSDB, M3 helps solve this problem by spreading metrics data, and demand for that data, across multiple nodes in a cluster. M3 does this by splitting data into segments that match certain criteria (such as above a certain value) across nodes into shards.
If you’ve worked with a distributed database before, then these concepts are probably familiar to you, but M3 uses different terminology to represent some concepts.
- Every cluster has one placement that maps shards to nodes in the cluster.
- A cluster can have 0 or more namespaces that are similar conceptually to tables in other databases, and each node serves every namespace for the shards it owns.
For example, if the cluster placement states that node A owns shards 1, 2, and 3, then node A owns shards 1, 2, 3 for all configured namespaces in the cluster. Each namespace has its own configuration options, including a name and retention time for the data.
Create a Placement and Namespace
This quickstart uses the http://localhost:7201/api/v1/database/create endpoint that creates a namespace, and the placement if it doesn’t already exist based on the type
argument.
You can create placements and namespaces separately if you need more control over their settings.
In another terminal, use the following command.
#!/bin/bash
curl -X POST http://localhost:7201/api/v1/database/create -d '{
"type": "local",
"namespaceName": "default",
"retentionTime": "12h"
}' | jq .
{
"namespace": {
"registry": {
"namespaces": {
"default": {
"bootstrapEnabled": true,
"flushEnabled": true,
"writesToCommitLog": true,
"cleanupEnabled": true,
"repairEnabled": false,
"retentionOptions": {
"retentionPeriodNanos": "43200000000000",
"blockSizeNanos": "1800000000000",
"bufferFutureNanos": "120000000000",
"bufferPastNanos": "600000000000",
"blockDataExpiry": true,
"blockDataExpiryAfterNotAccessPeriodNanos": "300000000000",
"futureRetentionPeriodNanos": "0"
},
"snapshotEnabled": true,
"indexOptions": {
"enabled": true,
"blockSizeNanos": "1800000000000"
},
"schemaOptions": null,
"coldWritesEnabled": false,
"runtimeOptions": null
}
}
}
},
"placement": {
"placement": {
"instances": {
"m3db_local": {
"id": "m3db_local",
"isolationGroup": "local",
"zone": "embedded",
"weight": 1,
"endpoint": "127.0.0.1:9000",
"shards": [
{
"id": 0,
"state": "INITIALIZING",
"sourceId": "",
"cutoverNanos": "0",
"cutoffNanos": "0"
},
…
{
"id": 63,
"state": "INITIALIZING",
"sourceId": "",
"cutoverNanos": "0",
"cutoffNanos": "0"
}
],
"shardSetId": 0,
"hostname": "localhost",
"port": 9000,
"metadata": {
"debugPort": 0
}
}
},
"replicaFactor": 1,
"numShards": 64,
"isSharded": true,
"cutoverTime": "0",
"isMirrored": false,
"maxShardSetId": 0
},
"version": 0
}
}
Placement initialization can take a minute or two. Once all the shards have the AVAILABLE
state, the node has finished bootstrapping, and you should see the following messages in the node console output.
{"level":"info","ts":1598367624.0117292,"msg":"bootstrap marking all shards as bootstrapped","namespace":"default","namespace":"default","numShards":64}
{"level":"info","ts":1598367624.0301404,"msg":"bootstrap index with bootstrapped index segments","namespace":"default","numIndexBlocks":0}
{"level":"info","ts":1598367624.0301914,"msg":"bootstrap success","numShards":64,"bootstrapDuration":0.049208827}
{"level":"info","ts":1598367624.03023,"msg":"bootstrapped"}
You can check on the status by calling the http://localhost:7201/api/v1/services/m3db/placement endpoint:
curl http://localhost:7201/api/v1/services/m3db/placement | jq .
{
"placement": {
"instances": {
"m3db_local": {
"id": "m3db_local",
"isolationGroup": "local",
"zone": "embedded",
"weight": 1,
"endpoint": "127.0.0.1:9000",
"shards": [
{
"id": 0,
"state": "AVAILABLE",
"sourceId": "",
"cutoverNanos": "0",
"cutoffNanos": "0"
},
…
{
"id": 63,
"state": "AVAILABLE",
"sourceId": "",
"cutoverNanos": "0",
"cutoffNanos": "0"
}
],
"shardSetId": 0,
"hostname": "localhost",
"port": 9000,
"metadata": {
"debugPort": 0
}
}
},
"replicaFactor": 1,
"numShards": 64,
"isSharded": true,
"cutoverTime": "0",
"isMirrored": false,
"maxShardSetId": 0
},
"version": 2
}
Ready a Namespace
Once a namespace has finished bootstrapping, you must mark it as ready before receiving traffic by using the http://localhost:7201/api/v1/services/m3db/namespace/ready.
#!/bin/bash
curl -X POST http://localhost:7201/api/v1/services/m3db/namespace/ready -d '{
"name": "default"
}' | jq .
{
"ready": true
}
View Details of a Namespace
You can also view the attributes of all namespaces by calling the http://localhost:7201/api/v1/services/m3db/namespace endpoint
curl http://localhost:7201/api/v1/services/m3db/namespace | jq .
Add ?debug=1
to the request to convert nano units in the output into standard units.
{
"registry": {
"namespaces": {
"default": {
"bootstrapEnabled": true,
"flushEnabled": true,
"writesToCommitLog": true,
"cleanupEnabled": true,
"repairEnabled": false,
"retentionOptions": {
"retentionPeriodNanos": "43200000000000",
"blockSizeNanos": "1800000000000",
"bufferFutureNanos": "120000000000",
"bufferPastNanos": "600000000000",
"blockDataExpiry": true,
"blockDataExpiryAfterNotAccessPeriodNanos": "300000000000",
"futureRetentionPeriodNanos": "0"
},
"snapshotEnabled": true,
"indexOptions": {
"enabled": true,
"blockSizeNanos": "1800000000000"
},
"schemaOptions": null,
"coldWritesEnabled": false,
"runtimeOptions": null
}
}
}
}
Writing and Querying Metrics
Writing Metrics
M3 supports ingesting statsd and Prometheus formatted metrics.
This quickstart focuses on Prometheus metrics which consist of a value, a timestamp, and tags to bring context and meaning to the metric.
You can write metrics using one of two endpoints:
- http://localhost:7201/api/v1/prom/remote/write - Write a Prometheus remote write query to M3DB with a binary snappy compressed Prometheus WriteRequest protobuf message.
- http://localhost:7201/api/v1/json/write - Write a JSON payload of metrics data. This endpoint is quick for testing purposes but is not as performant for production usage.
This quickstart uses the textfile collector feature of the Prometheus node exporter to export metrics to Prometheus that M3 then ingests. To follow the next steps, download node_exporter.
Configure and Start Prometheus
With M3 running and ready to receive metrics, change your Prometheus configuration to add M3 as remote_read
and remote_write
URLs, and as a job. With the configuration below, Prometheus scrapes metrics from two local nodes localhost:8080
and localhost:8081
in the production
group, and one local node localhost:8082
in the canary
group:
global:
scrape_interval: 15s
evaluation_interval: 15s
external_labels:
monitor: 'codelab-monitor'
remote_read:
- url: "http://localhost:7201/api/v1/prom/remote/read"
# To test reading even when local Prometheus has the data
read_recent: true
remote_write:
- url: "http://localhost:7201/api/v1/prom/remote/write"
scrape_configs:
- job_name: 'prometheus'
scrape_interval: 5s
static_configs:
- targets: ['localhost:9090']
- job_name: 'node'
scrape_interval: 5s
static_configs:
- targets: ['localhost:8080', 'localhost:8081']
labels:
group: 'production'
- targets: ['localhost:8082']
labels:
group: 'canary'
- job_name: 'm3'
static_configs:
- targets: ['localhost:7203']
Start Prometheus using the new configuration with the command below:
prometheus --config.file=prometheus.yml
Start Node Exporters
The three commands below simulate three point of sale (POS) devices reporting an hourly sales total for the POS. The node_exporter textfile collector uses .prom files for metrics, and only loads files from a directory, so this requires some extra steps for each node for this example.
Below is the text file for the first node, copy and paste it into a new .prom file in a directory named node-1, or download the file into that new folder.
# HELP third_avenue The total number of revenue from a POS
# TYPE third_avenue histogram
third_avenue_sum 3347.26
third_avenue_sum 3447.66
third_avenue_sum 3367.34
third_avenue_sum 4366.29
third_avenue_sum 4566.01
third_avenue_sum 4892.03
third_avenue_sum 5013.18
third_avenue_sum 5030.72
third_avenue_sum 5057.10
third_avenue_sum 5079.99
third_avenue_sum 5093.93
third_avenue_sum 5102.63
third_avenue_sum 5130.19
third_avenue_sum 5190.49
third_avenue_sum 5235.01
third_avenue_sum 5289.39
third_avenue_sum 5390.93
third_avenue_sum 5501.45
#!/bin/bash
./node_exporter --collector.textfile.directory=/node-1/ --web.listen-address 127.0.0.1:8081
Below is the text file for the second node, copy and paste it into a new .prom file in a directory named node-2, or download the file into that new folder.
# HELP third_avenue The total number of revenue from a POS
# TYPE third_avenue histogram
third_avenue_sum 8347.26
third_avenue_sum 9237.66
third_avenue_sum 10111.34
third_avenue_sum 11178.29
third_avenue_sum 11200.09
third_avenue_sum 12905.93
third_avenue_sum 13004.82
third_avenue_sum 13289.57
third_avenue_sum 13345.19
third_avenue_sum 13390.28
third_avenue_sum 13412.92
third_avenue_sum 13465.61
third_avenue_sum 13480.27
#!/bin/bash
./node_exporter --collector.textfile.directory=/node-2/ --web.listen-address 127.0.0.1:8082
Below is the text file for the second node, copy and paste it into a new .prom file in a directory named node-3, or download the file into that new folder.
# HELP third_avenue The total number of revenue from a POS
# TYPE third_avenue histogram
third_avenue_sum 1347.26
third_avenue_sum 1447.66
third_avenue_sum 2367.34
third_avenue_sum 3366.29
third_avenue_sum 3566.01
third_avenue_sum 3892.03
third_avenue_sum 4013.18
third_avenue_sum 4130.72
third_avenue_sum 4219.10
third_avenue_sum 5079.99
third_avenue_sum 5093.93
third_avenue_sum 5102.63
third_avenue_sum 5330.19
third_avenue_sum 5500.49
third_avenue_sum 5535.01
third_avenue_sum 5689.39
third_avenue_sum 5790.93
third_avenue_sum 5801.45
#!/bin/bash
./node_exporter --collector.textfile.directory=/node-3/ --web.listen-address 127.0.0.1:8083
You can now confirm that the node_exporter exported metrics to Prometheus by searching for third_avenue
in the Prometheus dashboard.
You can use the http://localhost:7201/api/v1/json/write endpoint to write a tagged metric to M3 with the following data in the request body, all fields are required:
tags
: An object of at least onename
/value
pairstimestamp
: The UNIX timestamp for the datavalue
: The float64 value for the data
The examples below use __name__
as the name for one of the tags, which is a Prometheus reserved tag that allows you to query metrics using the value of the tag to filter results.
Label names may contain ASCII letters, numbers, underscores, and Unicode characters. They must match the regex [a-zA-Z_][a-zA-Z0-9_]*
. Label names beginning with __
are reserved for internal use. Read more in the Prometheus documentation.
#!/bin/bash
curl -X POST http://localhost:7201/api/v1/json/write -d '{
"tags":
{
"__name__": "third_avenue",
"city": "new_york",
"checkout": "1"
},
"timestamp": '\"$(date "+%s")\"',
"value": 3347.26
}'
#!/bin/bash
curl -X POST http://localhost:7201/api/v1/json/write -d '{
"tags":
{
"__name__": "third_avenue",
"city": "new_york",
"checkout": "1"
},
"timestamp": '\"$(date "+%s")\"',
"value": 5347.26
}'
#!/bin/bash
curl -X POST http://localhost:7201/api/v1/json/write -d '{
"tags":
{
"__name__": "third_avenue",
"city": "new_york",
"checkout": "1"
},
"timestamp": '\"$(date "+%s")\"',
"value": 7347.26
}'
Querying metrics
M3 supports three query engines: Prometheus (default), Graphite, and the M3 Query Engine.
This quickstart uses Prometheus as the query engine, and you have access to all the features of PromQL queries.
To query metrics, use the http://localhost:7201/api/v1/query_range endpoint with the following data in the request body, all fields are required:
query
: A PromQL querystart
: Timestamp inRFC3339Nano
of start range for resultsend
: Timestamp inRFC3339Nano
of end range for resultsstep
: A duration or float of the query resolution, the interval between results in the timespan betweenstart
andend
.
Below are some examples using the metrics written above.
Return results in past 45 seconds
curl -X "POST" -G "http://localhost:7201/api/v1/query_range" \
-d "query=third_avenue" \
-d "start=$(date "+%s" -d "45 seconds ago")" \
-d "end=$( date +%s )" \
-d "step=5s" | jq .
curl -X "POST" -G "http://localhost:7201/api/v1/query_range" \
-d "query=third_avenue" \
-d "start=$( date -v -45S +%s )" \
-d "end=$( date +%s )" \
-d "step=5s" | jq .
{
"status": "success",
"data": {
"resultType": "matrix",
"result": [
{
"metric": {
"__name__": "third_avenue",
"checkout": "1",
"city": "new_york"
},
"values": [
[
1734656982,
"3347.26"
],
[
1734656982,
"5347.26"
],
[
1734656982,
"7347.26"
]
]
}
]
}
}
Values above a certain number
curl -X "POST" -G "http://localhost:7201/api/v1/query_range" \
-d "query=third_avenue > 6000" \
-d "start=$(date "+%s" -d "45 seconds ago")" \
-d "end=$( date +%s )" \
-d "step=5s" | jq .
curl -X "POST" -G "http://localhost:7201/api/v1/query_range" \
-d "query=third_avenue > 6000" \
-d "start=$(date -v -45S "+%s")" \
-d "end=$( date +%s )" \
-d "step=5s" | jq .
{
"status": "success",
"data": {
"resultType": "matrix",
"result": [
{
"metric": {
"__name__": "third_avenue",
"checkout": "1",
"city": "new_york"
},
"values": [
[
1734656982,
"7347.26"
]
]
}
]
}
}
Values collected from Prometheus
If you followed the steps above for collecting metrics from Prometheus, the examples above work, but don’t return any results. To query those results, use the following commands to return a sum of the values.
curl -X "POST" -G "http://localhost:7201/api/v1/query_range" \
-d "query=third_avenue_sum" \
-d "start=$(date "+%s" -d "45 seconds ago")" \
-d "end=$( date +%s )" \
-d "step=500s" | jq .
curl -X "POST" -G "http://localhost:7201/api/v1/query_range" \
-d "query=third_avenue_sum" \
-d "start=$( date -v -45S +%s )" \
-d "end=$( date +%s )" \
-d "step=500s" | jq .
{
"status": "success",
"data": {
"resultType": "matrix",
"result": [
{
"metric": {
"__name__": "third_avenue_sum",
"group": "canary",
"instance": "localhost:8082",
"job": "node",
"monitor": "codelab-monitor"
},
"values": [
[
1608737991,
"5801.45"
]
]
},
{
"metric": {
"__name__": "third_avenue_sum",
"group": "production",
"instance": "localhost:8080",
"job": "node",
"monitor": "codelab-monitor"
},
"values": [
[
1608737991,
"5501.45"
]
]
},
{
"metric": {
"__name__": "third_avenue_sum",
"group": "production",
"instance": "localhost:8081",
"job": "node",
"monitor": "codelab-monitor"
},
"values": [
[
1608737991,
"13480.27"
]
]
}
]
}
}