Updating a Cluster
After your cluster has been running for some time you may decide you want to change the cluster’s spec. For instance, you may want to upgrade to a newer release of M3DB or modify the cluster’s config file. The operator can be used to safely rollout such changes so you don’t need to do anything other than add an annotation to enable updates.
The first step in updating a cluster is to update the cluster’s
M3DBCluster CRD with the changes
you want to make. If you manage your cluster via manifests stored in YAML files then this is as
simple as updating the manifest and applying your changes:
kubectl apply -f example/my-cluster.yaml
As a precaution, the operator won’t immediately begin updating a cluster after your changes have
been applied. Instead, you’ll need to add the following annotation on each
StatefulSet in the
cluster to indicate to the operator that it is safe to update that
kubectl annotate statefulset my-cluster-rep0 operator.m3db.io/update=enabled
When the operator sees this annotation, it will check if the current state of the
differs from its desired state as defined by the
M3DBCluster CRD. If so, the operator will
StatefulSet to match its desired state, thereby triggering a rollout of the pods in
StatefulSet. The operator will also remove the
from the updated
If, on the other hand, the operator finds the update annotation on a
StatefulSet but it doesn’t
need to be updated then the operator will remove the annotation but perform no other actions.
Consequently, once you set the update annotation on a
StatefulSet, you can watch for the
annotation to be removed from it to know if the operator has seen and checked for an update.
Since M3DB rollouts can take longer periods of time, it’s often more convenient to set the
annotation to enable updates on each
StatefulSet in the cluster at once, and allow the operator
to perform the rollout safely. The operator will update only one
StatefulSet at a time and then
wait for it to bootstrap and become healthy again before moving onto the next
StatefulSet in the
cluster so that no two replicas are ever down at the same time.
kubectl annotate statefulset -l operator.m3db.io/cluster=my-cluster operator.m3db.io/update=enabled