Skip to content
Docs
Cancel

CircleCI Server v3.x Internal Database Volume Expansion

Last updated
Tags Server v3.x Server Admin

Overview

Persistent volume expansion of MongoDB and Postgres is available for server v3.2.0 and higher.

If you have chosen to deploy either of the CircleCI databases (MongoDB or Postgres) within the cluster, rather than externally provisioning these databases, there may come a point at which the storage space initially made available to these databases is no longer sufficient. Internal databases in your Kubernetes cluster make use of persistent volumes for persistent storage. The size of these volumes is determined by persistence volume claims (PVCs). These PVCs request storage space based on what has been made available to the nodes in your cluster.

This document runs through the steps required to increase PVCs to expand the space available to your internally deployed databases. This operation should not require any downtime, unless you need to restart your database pods.

Expanding persistent volumes does not affect the size of the storage attached to your nodes. Expanding node storage remains within the limitations of your cloud provider. Please refer to the docs for your chosen cloud provider for details on how to expand the storage attached to your cluster’s nodes.

Resizing persistent volume claims

Below are the steps detailing how to resize the persistent volume claims for Postgres and MongoDB. You will confirm the size of the claims and the disk space made available to your databases before and after this operation.

As a precaution, it is always a good idea to create a backup of your cluster first.

Step 0 - Confirm current volume size

By default, the persistent volume claims used by our internal databases have a capacity of 8Gi. However, this initial value can be set at the time of first deployment from the KOTS Admin Console. You can confirm the size of your persistent volume claim capacity using the command: kubectl get pvc <pvc-name>.

For Postgres:

circleci-user ~ $ kubectl get pvc data-postgresql-0
NAME                STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
data-postgresql-0   Bound    pvc-c2a2d97b-2b7d-47d3-ac77-d07c76c995a3   8Gi        RWO            gp2            1d

For MongoDB:

circleci-user ~ $ kubectl get pvc datadir-mongodb-0
NAME                STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
datadir-mongodb-0   Bound    pvc-58a2274c-31c0-487a-b329-0062426b5535   8Gi        RWO            gp2            1d

You can also confirm this capacity is made available to a database by checking the size of its data directory.

For Postgres, the directory is /bitnami/postgresql. You can confirm its size using the command below.

circleci-user ~ $ kubectl exec postgresql-0 -- df -h /bitnami/postgresql
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme4n1    7.8G  404M  7.4G   3% /bitnami/postgresql

For MongoDB, the directory is /bitnami/mongodb.

circleci-user ~ $ kubectl exec mongodb-0 -- df -h /bitnami/mongodb
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme1n1    7.8G  441M  7.4G   3% /bitnami/mongodb

From the examples above, the capacities are still 8Gi. The following steps show how to increase this to 10Gi.

Step 1 - Confirm volume expansion is allowed

First, confirm that volume expansion is allowed in your cluster.

circleci-user ~ $ kubectl get sc
NAME            PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
gp2 (default)   kubernetes.io/aws-ebs   Delete          WaitForFirstConsumer   false                  1d

As you can see, your default storage class does not allow volume expansion. However, you can change this with the kubectl patch command:

circleci-user ~ $ kubectl patch sc gp2 -p '{"allowVolumeExpansion": true}'
storageclass.storage.k8s.io/gp2 patched
circleci-user ~ $ kubectl get sc
NAME            PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
gp2 (default)   kubernetes.io/aws-ebs   Delete          WaitForFirstConsumer   true                  1d

Now you may proceed to expanding your volumes.

Step 2 - Delete the database’s stateful set

In this step, you will delete the stateful set, which controls your database pod. The command below deletes the referenced database’s stateful set without deleting the pod. You do not want to delete the pod itself, as this would cause downtime. In the following steps, you will redeploy your stateful set. You might chose to delete one or both stateful sets, depending on which database volumes you wish to expand. The --cascade=false flag is most important here.

For Postgres:

kubectl delete sts postgresql --cascade=false

For MongoDB:

kubectl delete sts mongodb --cascade=false

Step 3 - Update the size of the database’s PVC

Now that the stateful set has been removed, you can increase the size of our persistent volume claim to 10Gi.

For Postgres:

kubectl patch pvc data-postgresql-0 -p '{"spec": {"resources": {"requests": {"storage": "10Gi"}}}}'

For MongoDB:

kubectl patch pvc datadir-mongodb-0 -p '{"spec": {"resources": {"requests": {"storage": "10Gi"}}}}'

Step 4 - Update KOTS Admin Console with the new PVC size

Now you need to access the KOTS Admin Console to persist your changes. In the config section, you will update the values for your PVC size to 10Gi as shown below.

Postgres PVC size
Figure 1. Postgres
MongoDB PVC size
Figure 2. MongoDB

Now save and deploy your changes. This recreates the stateful set(s) that you destroyed earlier, but with the new PVC sizes, which will persist through new releases.

Step 5 - Validate new volume size

Once deployed, you can validate the size of the data directories assigned to our databases.

For Postgres the directory is /bitnami/postgresql.

circleci-user ~ $ kubectl exec postgresql-0 -- df -h /bitnami/postgresql
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme4n1    9.8G  404M  9.4G   5% /bitnami/postgresql

For MongoDB the directory is /bitnami/mongodb.

circleci-user ~ $ kubectl exec mongodb-0 -- df -h /bitnami/mongodb
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme1n1    9.8G  441M  9.3G   5% /bitnami/mongodb

As you can see, the size of your directories has been increased.

When completing these steps, if you find, as expected, that the new pods do show the resized volumes, it is still worth checking with the kubectl describe commands shown below. In some instances the resize will fail, but the only way to know is by viewing an event in the output from kubectl describe.

For Postgres:

kubectl describe pvc data-postgresql-0

For MongoDB:

kubectl describe pvc datadir-mongodb-0

Success looks like this example:

Events:
Type    Reason                      Age   From     Message

Normal  FileSystemResizeSuccessful  19m   kubelet  MountVolume.NodeExpandVolume succeeded for volume "pvc-b3382dd7-3ecc-45b0-aeff-45edc31f48aa"

Failure might look like this example:

Warning  VolumeResizeFailed  58m   volume_expand  error expanding volume "circleci-server/datadir-mongodb-0" of plugin "kubernetes.io/aws-ebs": AWS modifyVolume failed for vol-08d0861715c313887 with VolumeModificationRateExceeded: You've reached the maximum modification rate per volume limit. Wait at least 6 hours between modifications per EBS volume.
status code: 400, request id: 3bd43d1e-0420-4807-9c33-df26a4ca3f23
Normal   FileSystemResizeSuccessful  55m (x2 over 81m)  kubelet        MountVolume.NodeExpandVolume succeeded for volume "pvc-29456ce2-c7ff-492b-add4-fcf11872589f"

Troubleshooting

After following these steps, if you find that the disk size allocated to your data directories has not increased, then you may need to restart your database pods. This will cause downtime of 1-5 minutes while the databases restart. You can use the commands below to restart your databases.

For Postgres:

kubectl rollout restart sts postgresql

For MongoDB:

kubectl rollout restart sts mongodb

Help make this document better

This guide, as well as the rest of our docs, are open source and available on GitHub. We welcome your contributions.

Need support?

Our support engineers are available to help with service issues, billing, or account related questions, and can help troubleshoot build configurations. Contact our support engineers by opening a ticket.

You can also visit our support site to find support articles, community forums, and training resources.

Creative Commons License
CircleCI Documentation by CircleCI is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

6,000 Minutes Free

CircleCI, now with more build minutes
than anyone else for FREE 🎉

Sign up for Free

These teams get it

CCI customer teams