CI/CD: Automating our build and deploy process

• Introduction to CI/CD

• Importance of CI/CD in software development

• CircleCI and its alternatives

• How Mage uses CircleCI for our CI/CD pipeline

• CircleCI configuration

The “CI” stands for “continuous integration”, and it’s the practice of regularly building and testing code after each code change is merged to the main branch. Different tests can be run depending if code is committed to a secondary branch or the main branch.

The “CD” stands for either “continuous delivery” or “continuous deployment”. Continuous delivery and continuous deployment both build on top of continuous integration and enable automatic deployments of code changes to testing/staging and production environments

Delivering your app to the interwebs. (Source: Giphy/TikTok)

However, continuous delivery requires a manual step, such as an approval, before deploying to production, whereas continuous deployment automates the production deployment to completion without an approval step, assuming all tests and builds have successfully completed.

Using a CI/CD pipeline automatically tests new code commits at frequent intervals so bugs or breaking changes can be caught quickly and rectified before being deployed to production. The release process is much quicker, easier, and safer once the CI/CD pipeline is already set up, so it makes your team’s software development life cycle much more efficient.

is currently one of the most popular CI/CD tools on the market, and it supports several widely used programming languages, including Python, JavaScript, Java, Go, and Ruby on Rails. It also supports integration with both

and

.

Though CircleCI is the tool that Mage uses and that we will go into greater detail in the rest of this article, there are several other suitable CI/CD tools available, such as

,

, or

.

As stated in our

article, we previously used Travis CI but migrated our CI/CD pipeline to CircleCI because it allowed better control over configuring our pipeline (e.g. setting up manual approvals for certain jobs before the next job can run), was less restrictive in terms of number of concurrent jobs and users, and was a much better value for us (ended up costing five times less than Travis CI).

We have different CircleCI

for each of our services, but we’ll be discussing the different parts of our backend workflow below. A workflow defines a collection of

and the order in which they are run, and a job is a collection of steps. All steps in a job are executed in a single

or

(VM).

Github integration

We host all of our code repositories in GitHub, so we

with all of our services except the front-end service, which we use

for. We discuss how we use Vercel in the “Infrastructure” section of our

article.

When working on secondary branches, code commits pushed to the remote branch on GitHub will only build a development environment in a Docker container and run tests, but not make any deployments.

Testing/Debugging

In addition to the automated testing that CircleCI provides on every code commit pushed to the remote branch, CircleCI also allows us to SSH into test instances to check our code or test environment for debugging purposes.

Eliminating bugs in your pipeline. (Source: Giphy)

This SSH feature can be found by clicking on the “Rerun” button in the top right corner of the CircleCI job page and selecting “Rerun Job with SSH”:

Database migrations

On the main branch after all tests have successfully passed, CircleCI runs our database migrations for our

database hosted on

(

). This synchronizes the production database state with any updated models from our

backend.

Staging environment

After successfully running the database migrations, CircleCI deploys our

application to a staging environment in Amazon

and also runs some

automatically. With the staging environment deployed, we can run additional manual tests and confirm that any new or updated APIs are working properly.

Notifications

CircleCI can send

to members of a CircleCI project’s team whenever a

succeeds or fails to complete. There is also a

that can send Slack messages at any step during the workflow. We have a message sent to a specific Mage Slack channel whenever any manual approvals are required to remind us to go back into the CircleCI UI.

Manual approvals

We require a

in CircleCI for initiating any deployments to production. This allows us to do any additional custom testing in the staging environment first, and it also gives us control over the timing of our deployments.

Manual approvals actually require a team member to go into the CircleCI UI and click an “Approve” button, like in the screenshot below:

Building Docker images

As part of our CircleCI

, we can also create custom Docker images using our own

.

We use

to execute and quickly complete complex data processing tasks asynchronously. In order to configure our

functions, we need to build custom Docker images and push them to Amazon Elastic Container Registry (

). This is done automatically in CircleCI after approving the manual approval hold.

Production deployments

In addition to updating the AWS Lambda functions, simultaneous CircleCI

will be run for deploying our application and backend workers to production in

Elastic Beanstalk.

Deployment rollbacks

There may be situations where we want to rollback a deployment. For instance, if we already deployed our application to production but want to revert back to a previous version because we discovered an uncaught bug, we may be able to do that using CircleCI.

In order to rerun an individual job in a workflow after it has already successfully completed, CircleCI doesn’t have an explicit feature that allows us to do this from the UI. However, as a workaround, we can use the “rerun the job with SSH” feature like in the “Testing/Debugging” section above. We simply go back to the workflow for that particular code commit, and rerun the individual deployment job.

Note that the rerun deployment job should be able to reference the relevant Docker image with a unique version tag (e.g. a commit

value) if the job is deploying an application using Docker containers, which is the case for our

.

A

file using

is required for defining the steps, jobs, and workflows in CircleCI. We’ll go over some of the different parts of a

config.yml

file, but the best place for details on the different configuration keys would be the

.

Version

A minimum CircleCI version of 2.1 is required for several different features, so it’s not recommended to use a version lower than that.

1
version: 2.1

Orbs

are convenient packages of configuration elements (i.e.

,

,

) provided by CircleCI. There are many orbs available in CircleCI’s

.

Some of the orbs used by Mage include the Slack, AWS Elastic Beanstalk, AWS CLI, and AWS ECR orbs. The Slack orb allows CircleCI to send messages to one of our Slack channels, while the AWS orbs allows us to easily run CLI or other commands specific to those tools.

1
2
3
4
5
orbs:  
   slack: circleci/slack@4.3.1  
   aws-elastic-beanstalk: circleci/aws-elastic-beanstalk@1.0.2  
   aws-cli: circleci/aws-cli@1.4.0  
   aws-ecr: circleci/aws-ecr@6.15.3

Jobs

Most of the sections under “How Mage uses CircleCI for our CI/CD pipeline” above represent a single job in Mage’s backend workflow. For example, there is a job for running tests, a job for deploying to a staging environment, a job for building the AWS Lambda Docker image and deploying it to AWS ECR, etc.

Each job has a number of steps that get executed in a single Docker container or VM. In our CircleCI jobs, we generally use Docker containers whenever possible because they can spin up quicker than booting up a VM, and our jobs aren’t so resource-intensive that they require being run in a VM.

If your jobs only require a minimal amount of CPU and memory, a small Docker container uses less resources than a medium VM (the smallest size available in CircleCI), resulting in less

being used per minute.

The

slack/on-hold job

provided by a CircleCI orb has the following steps:

The CircleCI

have several examples of how the configuration should look like inside the

config.yml

file for jobs.

Workflows

There is a lot of customization regarding how a CircleCI

can run. Besides defining the order that jobs should run in, we can specify which branches we want certain jobs to run on, fan-out to run multiple jobs concurrently to save time, or require several jobs to successfully complete before a specific job can proceed.

We showed a diagram of Mage’s backend workflow earlier in the article at the very beginning of the “How Mage uses CircleCI for our CI/CD pipeline” section. This is just one of many ways to utilize the power of CircleCI workflows.

No matter which CI/CD tool you decide to use, the important thing is just having a CI/CD pipeline in place to begin with and then building off of it. Migrating to another platform can always happen later if absolutely necessary.

Relax while your pipeline takes care of the work for you. (Source: