CircleCI Server v3.x Hardening Your Cluster
Server v3.x Server Admin This section provides supplemental information on hardening your Kubernetes cluster.
Network Topology
A server installation basically runs three different type of compute instances: The Kubernetes nodes, Nomad clients, and external VMs.
It is highly recommended that you deploy these into separate subnets with distinct CIDR blocks. This will make it easier for you to control traffic between the different components of the system and isolate them from each other.
As always, the rule is to make as many of the resources as private as possible, applies. If your users will access your CircleCI server installation via VPN, there is no need to assign any public IP addresses at all, as long as you have a working NAT gateway setup. Otherwise, you will need at least one public subnet for the circleci-server-traefik load balancer.
From server v3.3.0, we have replaced Traefik with Kong as our reverse proxy. However, in order to minimize disruption when upgrading, we chose not to rename the service used by Kong. Although you will see a service named circleci-server-traefik, this service is actually for Kong. |
However, in this case, it is also recommended to place Nomad clients and VMs in a public subnet to enable your users to SSH into jobs and scope access via networking rules.
Currently, custom subnetting is not supported for GCP. Custom subnetting support will be available in a future update/release.
Network Traffic
This section explains the minimum requirements for a server installation to work. Depending on your workloads, you might need to add additional rules to egress for Nomad clients and VMs. As nomenclature between cloud providers differs, you will probably need to implement these rules using firewall rules and/or security groups.
Where you see "external," this usually means all external IPv4 addresses. Depending on your particular setup, you might be able to be more specific (for example, if you are using a proxy for all external traffic).
It is assumed that you have configured the load balancers for Nomad, vm-service and output processor to be internal load balancers. This is the default.
The rules explained here are assumed to be stateful and for TCP connections only, unless stated otherwise. If you are working with stateless rules, you need to create matching ingress or egress rules for the ones listed here.
Kubernetes Load Balancers
Depending on your setup, your load balancers might be transparent (that is, they are not treated as a distinct layer in your networking topology). In this case, you can apply the rules from this section directly to the underlying destination or source of the network traffic. Refer to the documentation of your cloud provider to make sure you understand how to correctly apply networking security rules, given the type of load balancing you are using with your installation.
Ingress
If the traffic rules for your load balancers have not been created automatically, here are their respective ports:
| Name | Port | Source | Purpose |
|---|---|---|---|
*-server-traefik | 80 | External | User Interface & Frontend API |
*-server-traefik | 443 | External | User Interface & Frontend API |
vm-service | 3000 | Nomad clients | Communication with Nomad clients |
nomad | 4647 | Nomad clients | Communication with Nomad clients |
output-processor | 8585 | Nomad clients | Communication with Nomad clients |
Egress
The only type of egress needed is TCP traffic to the Kubernetes nodes on the Kubernetes load balancer ports (30000-32767). This is not needed if your load balancers are transparent.
Common Rules for Compute Instances
These rules apply to all compute instances, but not to the load balancers.
Ingress
If you want to access your instances using SSH, you will need to open port 22 for TCP connections for the instances in question. It is recommended to scope the rule as closely as possible to allowed source IPs and/or only add such a rule when needed.
Egress
You most likely want all of your instances to access internet resources. This requires you to allow egress for UDP and TCP on port 53 to the DNS server within your VPC, as well as TCP ports 80 and 443 for HTTP and HTTPS traffic, respectively. Instances building jobs (that is, the Nomad clients and external VMs) also will likely need to pull code from your VCS using SSH (TCP port 22). SSH is also used to communicate with external VMs, so it should be allowed for all instances with the destination of the VM subnet and your VCS, at the very least.
Kubernetes Nodes
Intra-node traffic
By default, the traffic within your Kuberntes cluster is regulated by networking policies. For most purposes, this should be sufficient to regulate the traffic between pods and there is no additional requirement to reduce traffic between Kubernetes nodes any further (it is fine to allow all traffic between Kubernetes nodes).
To make use of networking policies within your cluster, you may need to take additional steps, depending on your cloud provider and setup. Here are some resources to get you started:
Ingress
If you are using a managed service, you can check the rules created for the traffic coming from the load balancers and the allowed port range. The standard port range for Kubernetes load balancers (30000-32767) should be all that is needed here for ingress. If you are using transparent load balancers, you need to apply the ingress rules listed for load balancers above.
Egress
| Port | Destination | Purpose |
|---|---|---|
2376 | VMs | Communication with VMs |
4647 | Nomad clients | Communication with the Nomad clients |
all traffic | other nodes | Allow intra-cluster traffic |
Nomad Clients
Nomad clients do not need to communicate with each other. You can block traffic between Nomad client instances completely.
Ingress
| Port | Source | Purpose |
|---|---|---|
4647 | K8s nodes | Communication with Nomad server |
64535-65535 | External | Rerun jobs with SSH functionality |
Egress
| Port | Destination | Purpose |
|---|---|---|
2376 | VMs | Communication with VMs |
3000 | VM Service load balancers | Internal communication |
4647 | Nomad Load Balancer | Internal communication |
8585 | Output Processor Load Balancer | Internal communication |
External VMs
Similar to Nomad clients, there is no need for external VMs to communicate with each other.
Ingress
| Port | Source | Purpose |
|---|---|---|
22 | Kubernetes nodes | Internal communication |
22 | Nomad clients | Internal communication |
2376 | Kubernetes nodes | Internal communication |
2376 | Nomad clients | Internal communication |
54782 | External | Rerun jobs with SSH functionality |
Egress
You will only need the egress rules for internet access and SSH for your VCS.
What to read next
Help make this document better
This guide, as well as the rest of our docs, are open source and available on GitHub. We welcome your contributions.
- Suggest an edit to this page (please read the contributing guide first).
- To report a problem in the documentation, or to submit feedback and comments, please open an issue on GitHub.
- CircleCI is always seeking ways to improve your experience with our platform. If you would like to share feedback, please join our research community.
Need support?
Our support engineers are available to help with service issues, billing, or account related questions, and can help troubleshoot build configurations. Contact our support engineers by opening a ticket.
You can also visit our support site to find support articles, community forums, and training resources.
CircleCI Documentation by CircleCI is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
