Build a cost-optimized CI/CD pipeline with CircleCI and AWS CDK
Data Scientist
As organizations adopt cloud-native development, there is a growing need for scalable, automated, and cost-effective CI/CD pipelines. This tutorial guides you through how to use AWS CDK to define Infrastructure as Code and CircleCI to automate deployments, all while keeping cost optimization a top priority.
Traditional infrastructure management can lead to over-provisioning, forgotten resources, and unexpected bills. By combining Infrastructure as Code (IaC) with automated CI/CD practices, you can create a system that:
- Scales resources based on actual demand
- Monitors costs in real-time
- Cleans up unused resources
You will build an end-to-end workflow that provisions AWS resources, sets up auto-scaling, implements monitoring, and tears down unused infrastructure automatically. This approach ensures you can maintain high availability and performance, while paying only for what you use while .
Prerequisites
Before you begin, you will need:
- AWS account: Sign up for an AWS account if you do not already have one.
- AWS CLI installed and configured: Install the AWS Command Line Interface (CLI) and configure it with your AWS credentials (IAM Role Configuration). You can follow the AWS CLI setup guide.
- AWS access and secret keys: You will use them during CI/CD deployment. Refer to this AWS resource on how to access them.
- AWS CDK setup: Refer to the official getting started guide
- Familiarity with VPCs and EC2: You should have basic familiarity with Amazon VPC and Amazon EC2 concepts.
- GitHub and CircleCI accounts: You will need a GitHub account for version control and a CircleCI account to automate your CI/CD pipeline.
- Python virtual environment: You also need to create a Python virtual environment.
Once you have these items in place, you can set up your project and implement the serverless workflow.
Setting up the project
What is AWS CDK? The AWS Cloud Development Kit (CDK) lets you define infrastructure using familiar programming languages (e.g., Python). Instead of writing raw CloudFormation templates or clicking through the AWS Console, you get several key advantages:
- Type safety: Catch errors at compile time rather than deployment time
- IDE support: Get autocompletion, refactoring, and debugging capabilities
- Reusability: Create reusable components and share them across projects
- Version control: Track infrastructure changes alongside application code
CDK compiles your code into CloudFormation templates and deploys them, combining developer productivity with infrastructure best practices. This means you get the reliability of CloudFormation with the expressiveness of modern programming languages.
Set up the environment
Start by cloning the repository and setting up your development environment:
git clone https://github.com/CIRCLECI-GWP/cost-optimized-cicd-circleci-aws-cdk.git
cd cost-optimized-cicd-circleci-aws-cdk
Set up a virtual environment in the cdk folder and install dependencies:
cd cdk
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
If you started the project from scratch with an empty repository, you’ll need to create the cdk folder structure. From the root directory, run:
mkdir cdk
cd cdk
cdk init app --language python --profile <profile-name>
This will create all necessary dependencies, files and folders similar to the structure of the next section.
Set up the project structure
Here is the basic structure of the project if you cloned the repository. Under cdk you can find the AWS CDK Infrastructure. Each of the Stack that will be deployed can be found under infrastructure. The app.py file just imports each of them. To deploy the stack you need to run the corresponding command from the cdk directory where the app.py is located.
├── LICENSE
├── Makefile
├── README.md
├── cdk
│ ├── README.md
│ ├── app.py
│ ├── .env.example
│ ├── infrastructure
│ │ ├── __init__.py
│ │ ├── compute_stack.py
│ │ ├── monitoring_stack.py
│ │ └── vpc_stack.py
│ ├── requirements.txt
└── circleci
├── config.yml
If you initialized the project from scratch you would have a different structure under the main cdk folder. Your infrastructure folder, where the Stack is located, will have the name cdk. For this tutorial, I recommend that you change it to infrastructure, like in the cloned version because app.py take imports from that folder. The next sections of the tutorial use code based on that structure, but you are free to adapt it if you’d like. The requirements-dev.txt file and tests folder are not necessary for this project.
.
├── app.py
├── cdk
│ ├── cdk_stack.py
│ └── __init__.py
├── cdk.json
├── README.md
├── requirements-dev.txt
├── requirements.txt
├── source.bat
└── tests
├── __init__.py
└── unit
├── __init__.py
└── test_cdk_stack.py
You also need to add an environment variable into the .env.example file in the main cdk folder. Your email is used to receive cost alerts and notifications from AWS Budgets and SNS topics. Rename the file to .env:
EMAIL=your_email@example.com
Create a CDK stack
You will create three CDK stacks that work together to provide a complete, cost-optimized infrastructure:
- VPC stack: Defines a custom VPC with public and private subnets for network isolation.
- Compute stack: Launches EC2 instances using an Auto Scaling Group with intelligent scaling policies.
- Monitoring stack: Adds cost budgets, billing alarms, and dashboards for financial visibility.
This separation follows the single responsibility principle and makes it easier to manage each component independently.
CDK app entry point
When deploying the infrastructure, AWS CDK uses the app entry point under the app.py file. This file deploys all the stacks in the correct order. The next section of the tutorial has an explanation of each stack.
import aws_cdk as cdk
from infrastructure.compute_stack import ComputeStack
from infrastructure.monitoring_stack import MonitoringStack
from infrastructure.vpc_stack import VpcStack
app = cdk.App()
# Deploy infrastructure stacks
vpc_stack = VpcStack(app, "VpcStack")
compute_stack = ComputeStack(app, "ComputeStack", vpc=vpc_stack.vpc)
monitoring_stack = MonitoringStack(app, "MonitoringStack")
app.synth()
As shown in the previous code sample, the ComputeStack depends on the VPC created by VpcStack. CDK automatically handles the dependency order during deployment, ensuring the VPC is created before the compute resources that depend on it.
VPC stack (infrastructure/vpc_stack.py)
This stack is based on some key design decisions:
- Limited availability zones: Using only 2 AZs reduces NAT Gateway costs while maintaining high availability.
- Subnet types: Public subnets for load balancers and bastion hosts, private subnets for application servers.
- CIDR planning: /16 provides 65,536 IP addresses, allowing for future growth while using standard private IP ranges.
The VPC Stack creates the foundational network infrastructure keeping security and cost optimization top priorities:
from typing import Any
from aws_cdk import (
Stack,
)
from aws_cdk import (
aws_ec2 as ec2,
)
from constructs import Construct
class VpcStack(Stack):
def __init__(self, scope: Construct, construct_id: str, **kwargs: Any) -> None:
super().__init__(scope, construct_id, **kwargs)
# Create VPC with public and private subnets
self.vpc = ec2.Vpc(
self, "CostOptimizedVPC",
max_azs=2,
cidr="10.0.0.0/16",
subnet_configuration=[
ec2.SubnetConfiguration(
name="PublicSubnet",
subnet_type=ec2.SubnetType.PUBLIC,
cidr_mask=24
),
ec2.SubnetConfiguration(
name="PrivateSubnet",
subnet_type=ec2.SubnetType.PRIVATE_WITH_EGRESS,
cidr_mask=24
)
]
)
The VPC creates Internet Gateways, NAT Gateways, and route tables. Limiting AZs to 2 reduces the number of NAT Gateways (which cost ~$35/month each) while still achieving high availability.
Compute stack (infrastructure/compute_stack.py)
The Compute Stack handles all EC2-related resources with built-in auto-scaling and cost optimization. It provides these key cost optimization features:
- t3.micro instances: Burstable performance instances that provide baseline performance and can burst when needed
- Security group: The security group allows HTTP (port 80) and SSH (port 22) access.
- IAM role: The IAM role enables CloudWatch monitoring without hardcoding credentials.
- Auto scaling: Automatically adjusts capacity based on demand, scaling down during low traffic periods
- Launch templates: Enables easy updates and version control for instance configurations
- User data script: The user data script sets up a basic web server and installs
stress-ngfor load testing, which will be used to verify auto-scaling functionality. - Target tracking: Maintains 70% CPU utilization, adding up to 3 instances to optimize the balance between performance and cost
from typing import Any
from aws_cdk import Duration, Stack, Tags
from aws_cdk import aws_autoscaling as autoscaling
from aws_cdk import aws_ec2 as ec2
from aws_cdk import aws_iam as iam
from constructs import Construct
class ComputeStack(Stack):
def __init__(self, scope: Construct, construct_id: str, vpc: ec2.Vpc, **kwargs: Any) -> None:
super().__init__(scope, construct_id, **kwargs)
# Security Group
security_group = ec2.SecurityGroup(
self, "WebServerSG",
vpc=vpc,
description="Security group for web servers",
allow_all_outbound=True
)
security_group.add_ingress_rule(
ec2.Peer.any_ipv4(),
ec2.Port.tcp(80),
"Allow HTTP traffic"
)
security_group.add_ingress_rule(
ec2.Peer.any_ipv4(),
ec2.Port.tcp(22),
"Allow SSH access"
)
# IAM Role for EC2 instances
role = iam.Role(
self, "EC2Role",
assumed_by=iam.ServicePrincipal("ec2.amazonaws.com"),
managed_policies=[
iam.ManagedPolicy.from_aws_managed_policy_name("CloudWatchAgentServerPolicy")
]
)
# Launch Template
launch_template = ec2.LaunchTemplate(
self, "LaunchTemplate",
instance_type=ec2.InstanceType.of(
ec2.InstanceClass.T3,
ec2.InstanceSize.MICRO
),
machine_image=ec2.AmazonLinuxImage(
generation=ec2.AmazonLinuxGeneration.AMAZON_LINUX_2
),
security_group=security_group,
role=role,
user_data=ec2.UserData.for_linux()
)
# Add simple web server setup
launch_template.user_data.add_commands(
"yum update -y",
"yum install -y httpd",
"systemctl start httpd",
"systemctl enable httpd",
"echo '<h1>Cost-Optimized CI/CD Demo</h1>' > /var/www/html/index.html"
# Add stress-ng installation
"amazon-linux-extras install epel -y",
"yum install -y stress-ng",
)
# Auto Scaling Group
self.asg = autoscaling.AutoScalingGroup(
self, "WebServerASG",
vpc=vpc,
launch_template=launch_template,
min_capacity=1,
max_capacity=3,
desired_capacity=1,
vpc_subnets=ec2.SubnetSelection(
subnet_type=ec2.SubnetType.PUBLIC
)
)
Tags.of(self.asg).add("Name", "CostOptimizedWebServer")
# Target Tracking Scaling Policy
self.asg.scale_on_cpu_utilization(
"CpuScaling",
target_utilization_percent=70,
cooldown=Duration.minutes(5)
)
Monitoring stack (infrastructure/monitoring_stack.py)
The Monitoring Stack provides comprehensive cost visibility and automated alerting using these monitoring capabilities:
- SNS topic: Centralized notification system for all cost-related alerts
- Budget alerts: Proactive notifications at 80% of budget (actual) and 100% (forecasted)
- Billing alarms: Real-time alerts when charges exceed $30, evaluated every 6 hours
- Custom dashboard: Visual representation of spending trends and patterns
import os
from aws_cdk import Duration, Stack
from aws_cdk import aws_budgets as budgets
from aws_cdk import aws_cloudwatch as cloudwatch
from aws_cdk import aws_cloudwatch_actions as cw_actions
from aws_cdk import aws_sns as sns
from aws_cdk import aws_sns_subscriptions as sns_subscriptions
from constructs import Construct
from dotenv import load_dotenv
load_dotenv()
class MonitoringStack(Stack):
def __init__(self, scope: Construct, construct_id: str, **kwargs: object) -> None:
super().__init__(scope, construct_id, **kwargs)
# SNS Topic for alerts
alert_topic = sns.Topic(
self, "CostAlertTopic",
display_name="Cost Optimization Alerts"
)
# Subscribe to the topic using an email address from environment variable
# Ensure the EMAIL environment variable is set
# Needs confirmation via email after deployment
alert_topic.add_subscription(
sns_subscriptions.EmailSubscription(os.getenv("EMAIL"))
)
# Cost Budget Alert
budgets.CfnBudget(
self, "MonthlyBudget",
budget={
"budgetName": "Monthly-AWS-Budget",
"budgetLimit": {
"amount": 50, # $50 monthly limit
"unit": "USD"
},
"timeUnit": "MONTHLY",
"budgetType": "COST"
},
notifications_with_subscribers=[
{
"notification": {
"notificationType": "ACTUAL",
"comparisonOperator": "GREATER_THAN",
"threshold": 80 # Alert at 80% of budget ($40)
},
"subscribers": [
{
"subscriptionType": "EMAIL",
"address": os.getenv("EMAIL", "") # Use environment variable for email
}
]
},
{
"notification": {
"notificationType": "FORECASTED",
"comparisonOperator": "GREATER_THAN",
"threshold": 100 # Alert when forecasted to exceed budget
},
"subscribers": [
{
"subscriptionType": "EMAIL",
"address": os.getenv("EMAIL", "") # Use environment variable for email
}
]
}
]
)
# CloudWatch Billing Alarm
billing_alarm = cloudwatch.Alarm(
self, "BillingAlarm",
metric=cloudwatch.Metric(
namespace="AWS/Billing",
metric_name="EstimatedCharges",
dimensions_map={
"Currency": "USD"
},
statistic="Maximum",
period=Duration.hours(6)
),
threshold=30, # $30 threshold
evaluation_periods=1,
alarm_description="Alert when estimated charges exceed $30"
)
billing_alarm.add_alarm_action(
cw_actions.SnsAction(alert_topic)
)
# Custom Dashboard
dashboard = cloudwatch.Dashboard(
self, "CostDashboard",
dashboard_name="Cost-Optimization-Dashboard"
)
dashboard.add_widgets(
cloudwatch.GraphWidget(
title="Estimated Monthly Charges",
left=[billing_alarm.metric],
width=12,
height=6
)
)
The monitoring stack uses environment variables for email addresses to avoid hardcoding sensitive information. Make sure you add your email in the .env file. After deployment, you will need to confirm the email subscription using the confirmation email sent by AWS.
Automated cost optimization techniques
This architecture implements several automated cost optimization strategies:
- Auto scaling based on demand: The Auto Scaling Group automatically adjusts the number of EC2 instances based on CPU utilization. When traffic is low, it scales down to the minimum capacity (1 instance). During high traffic periods, it scales up to a maximum of 3 instances.
- Target tracking scaling policy: The target tracking policy maintains 70% CPU utilization across the fleet. This ensures optimal resource utilization without over-provisioning. The 5-minute cooldown prevents rapid scaling actions that could cause instability.
- Burstable performance instances: T3.micro instances provide baseline performance with the ability to burst when needed. This is perfect for workloads with variable performance requirements, as you only pay for the extra performance when you use it.
- Real-time cost monitoring: CloudWatch billing alarms and AWS Budgets provide multiple layers of cost protection. The system alerts you before costs spiral out of control, allowing for proactive cost management.
- Infrastructure as Code: By defining infrastructure in code, you can easily tear down environments when not needed and recreate them when required. This eliminates the common problem of “forgotten” resources that continue to accrue charges.
Deploying the application
To deploy all the infrastructure components defined in your AWS CDK project, you will need Bootstrap. Bootstrap prepares your AWS environment (e.g., S3 bucket, IAM roles) for CDK usage, and it is required if you run AWS CDK for the first time. Bootstrap creates a default CDKToolkit AWS CloudFormation stack.
Go to the root of the cdk directory (where your app.py file is). Run this command:
cdk bootstrap --profile <profile-name>
Once the CDKToolkit has been created you can deploy the application stack:
cdk deploy --all --profile <profile-name>
This command synthesizes and deploys all stacks in the correct dependency order. Once this process is complete, you can verify the deployment using the AWS Management Console. This command can be rerun as many times as you want. If the stack already exists, AWS will notify you and will not recreate it. If you have made any modification on the current deployed stack, the stack will be updated.
Stack formation
After deployment, each CDK stack will appear in the AWS CloudFormation console. You can inspect each stack to review its resources and status.
Cloudformation full stack:
Monitoring stack:
VPC stack:
Compute stack
Monitoring dashboards and alarms
You can access monitoring dashboards and alarms using Amazon CloudWatch. There are three alarms, one for billing and two for the CPU (high and low).
The CPU Low alarm shows an “In Alarm” status, indicating that the instances are underutilized. Meanwhile, there is a visible spike above the 70% threshold, which was configured in the compute stack. This spike results from a stress test used to verify whether autoscaling functions correctly.
For the stress test, you’ll run a 5-minute CPU load on 4 cores in the background. Connect to your EC2 instance and execute:
stress-ng --cpu 4 --timeout 300s
When CPU usage exceeds the threshold, the Auto Scaling Group will launch a second instance. This scaling activity can be observed in the CPU Dashboards and Auto Scaling Group views. After the cooldown period, if CPU usage drops below the target, the additional instance will be terminated.
In addition to CPU and scaling metrics, the budget and cost monitoring dashboards can be also found under Billing and Cost Management. These tools help ensure your deployed stack remains within financial limits and offers insights into cost optimization.
Setting up CI/CD with CircleCI
To automate the deployment pipeline using CircleCI, you need to define a CircleCI config file (.config.yml). This pipeline will handle tasks such as installing dependencies, validating your code, configuring AWS resources, and running tests.
Before CircleCI can deploy your application to AWS, you need to configure the required environment variables. These variables secure your AWS credentials and configuration settings. Here’s how to configure them:
- Go to your CircleCI dashboard.
- Create a new project and link it to your GitHub repository.
- Push it to GitHub in advance.
- Go to your project settings and add the AWS Credentials and Email environment variables:
AWS_ACCESS_KEY_ID: Your AWS access keyAWS_SECRET_ACCESS_KEY: Your AWS secret keyEMAIL: Email address for cost alertsAWS_DEFAULT_REGION: Your preferred AWS region (e.g.,eu-central-1)
![CircleCI env vars]2025-07-04-circleci-env-vars.png
Next, you’ll need to create a pipeline in your project and set up a trigger for that pipeline. If you want to trigger the pipeline on each push to the main branch, you can set up an all pushes event.
Once the environment variables, the pipeline, and the trigger are set up, you can define the CircleCI config file.
CircleCI configuration
The .circleci/config.yml file defines the complete deployment pipeline. Here’s a breakdown:
-
Orbs are pre-built CircleCI packages that provide common functionality. For example
nodeprovides Node.js runtime for CDK. - Deploy job steps:
Checkoutretrieves your code from the repository.Install Pythoninstalls Python 3, pip, and virtual environment tools.Install AWS CLI v2Installs the latest AWS CLI to interact with AWS services.Install CDKInstalls the AWS Cloud Development Kit (CDK) globally using npm.Set Up Python Environmentcreates a virtual environment and installs Python dependencies for your CDK app.Verify AWS Credentialsconfirms that the IAM role or access keys are valid.Bootstrap CDKprepares your AWS environment (e.g., S3 bucket, IAM roles) for CDK usage.Deploy stacksdeploys all CDK stacks without requiring manual approval.Output Infoprovides confirmation and useful logging once deployment is complete.Workflow: Thedeploy_appworkflow triggers the deploy job on every push to the repository.
-
Destroy job steps:
- Same initial steps as those for the deploy job: checkout, install dependencies, and verify credentials.
Destroy CDK stacksexecutescdk destroy --all --forceto clean up all infrastructure.Output teardown infologs confirmation that the teardown completed.
-
Workflows:
deploy_apptriggers the deploy job on every push to the repository.scheduled_teardownruns the destroy job every Friday at 15:30 UTC, tearing down deployed infrastructure to help manage cloud resource costs.
version: 2.1
orbs:
node: circleci/node@7.1.0
jobs:
deploy:
docker:
- image: cimg/node:24.3.0
steps:
- checkout
- run:
name: Install Python
command: |
sudo apt-get update
sudo apt-get install -y python3 python3-venv python3-pip unzip curl
- run:
name: Install AWS CLI v2
command: |
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install
aws --version
- run:
name: Install AWS CDK
command: npm install -g aws-cdk
- run:
name: Set up Python environment
command: |
cd cdk
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
- run:
name: Verify AWS credentials
command: aws sts get-caller-identity
- run:
name: Bootstrap CDK
command: |
cd cdk
source .venv/bin/activate
cdk bootstrap
- run:
name: Deploy CDK stacks
command: |
cd cdk
source .venv/bin/activate
cdk deploy --all --require-approval never
- run:
name: Output deployment info
command: |
echo "Deployment completed successfully"
echo "Check AWS Console for deployed resources"
destroy:
docker:
- image: cimg/node:24.3.0
steps:
- checkout
- run:
name: Install Python
command: |
sudo apt-get update
sudo apt-get install -y python3 python3-venv python3-pip unzip curl
- run:
name: Install AWS CLI v2
command: |
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install
aws --version
- run:
name: Install node.js and CDK
command: npm install -g aws-cdk
- run:
name: Set up Python environment
command: |
cd cdk
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
- run:
name: Verify AWS credentials
command: aws sts get-caller-identity
- run:
name: Destroy CDK stacks
command: |
cd cdk
source .venv/bin/activate
cdk destroy --all --force
- run:
name: Output teardown info
command: |
echo "Teardown completed successfully"
echo "Resources have been destroyed"
workflows:
deploy_app:
jobs:
- deploy
# Scheduled destroy job (e.g., Every Friday at 15:30 UTC)
scheduled_teardown:
triggers:
- schedule:
cron: "30 15 * * 5" # Every Friday at 15:30 UTC
filters:
branches:
only:
- main
jobs:
- destroy
Once this configuration is committed and pushed to your GitHub repository, CircleCI will kick off the deployment process. You will be able to monitor the steps directly in the CircleCI dashboard. If everything is set up correctly, you will get a green build.
Note: The scheduled teardown job will run every Friday at 15:30 UTC, automatically destroying all resources created by the deployment job. I recommend changing the schedule to a better time for you. You can disable it if you want to keep the resources running longer.
![CircleCI deploy]2025-07-04-circleci-deploy.png
![CircleCI destroy]2025-07-04-circleci-destroy.png
Cleaning up
The time for clean up is when you are done with the demo or want to avoid ongoing charges. If the CircleCI scheduled teardown job hasn’t cleaned everything up, you can do this manually. In the cdk directory, run:
cdk destroy --all --profile <profile-name>
Make sure you cross-check that all your resources have been deleted to avoid unnecessary charges. You can check in the console AWS EC2, Cloudwatch and Budget or delete them from the AWS Console.
Conclusion
This tutorial showed you how to build a cost-optimized CI/CD pipeline that combines the power of Infrastructure as Code with automated deployment and monitoring. The key achievements include:
- Using AWS CDK to define infrastructure as code: Providing type safety, reusability, and version control for your infrastructure.
- Leveraging CircleCI to automate deployment and teardown: Ensuring consistent, repeatable deployments while minimizing manual errors.
- Applying auto-scaling policies to optimize compute resource usage: Automatically adjusting capacity based on demand to minimize costs.
- Integrating cost monitoring and alerts for financial visibility: Proactive cost management through real-time alerts and budgets.
By implementing these practices, you can significantly reduce your AWS spending while maintaining high availability and performance. The infrastructure scales automatically based on demand, alerts you to cost anomalies, and can be easily reproduced or destroyed as needed. The combination of AWS CDK, CircleCI, and cost optimization techniques provides a robust foundation for cloud-native applications. As your applications grow, this architecture can be extended with additional features like blue-green deployments, canary releases, and multi-region setups.
Remember that cost optimization is an ongoing process. Regularly review your usage patterns, explore new AWS services and pricing models, and continuously refine your architecture based on actual usage data. With these practices in place, you’ll be well-equipped to build and maintain cost-effective cloud infrastructure that scales with your business needs.
