Take the stress out of deployment
City of San Diego automates data transparency using CircleCI
Why CircleCI?
In mid-2016, the City of San Diego launched DataSD, a Drupal-based portal hosted by a vendor and allowing citizens to access and download data about city operations, revenue, licensing, permits and more. In February 2017, the city replaced this portal with a custom-built Jekyll-based portal. The city’s chief data officer, Maksim Pecherskiy, uses CircleCI to deploy changes to the portal several times a day, while also keeping infrastructure to a minimum. “Going with CircleCI meant that we made a small investment upfront to make sure our systems run automatically,” Pecherskiy says. “That’s much better than us deploying ourselves, possibly making mistakes, and causing a lot of stress.”
The Challenge
The website data.sandiego.gov is updated multiple times a day. When data is updated, metadata changes get committed to the portal, and the new code needs to move to production. New datasets, blog posts, and data stories are continuously created as well. Updating such changes manually would be a time-consuming process.
“We’re a small team and we have to keep our infrastructure really light,” Pecherskiy says. “We like to let machines do what they do best, leaving humans to do what they’re really good at.”
The Solution
Pecherskiy uses CircleCI to automate deployment of the portal. Based on the branch where code was committed, code is deployed to a target site for the next step in the process. The team uses the Airflow workflow tool to manage data updates into DataSD; CircleCI then loads the data into a Netlify staging site, where it undergoes acceptance testing. “Once we merge into the production branch, CircleCI picks up the data and deploys it,” Pecherskiy says.
“While CircleCI is deploying to production, we’re in meetings, we’re looking for more data to release, and we’re doing analysis – the kinds of things humans are good it,” Pecherskiy adds. “With CircleCI and continuous integration, we let machines do what machines are good at.”
“CircleCI is one of my stress-free tools.”
Maksim Pecherskiy | Chief Data Officer at City of San Diego
About DataSD
DataSD is the official portal for City of San Diego. The portal includes data about city operations, including public safety, street repairs, public facilities, code enforcement, and business licensing. The goal of DataSD is to promote civic engagement, improve service delivery, allow for more effective communication with the public, and increase opportunities for economic development.