How to do Continuous Integration and Continuous Delivery with Circleci

5-minute read

The idea of continuous integration and continuous delivery(CI/CD) becomes wildly used for version control repositories. There are tons of blogs defining what is CI/CD. I get familiar with CI/CD from my work experience in Circleci. In my understanding, CI is a procedure to verify each small changes will not affect the functionality of a project and not violate the tests of a project in a workflow or pipeline. After CI ends, CD picks up the procedure to achieve some deployment such as release packages, sync static website and publish docs etc. This is how I understand CI/CD according to my experience. With CI/CD procedure, development will reduce the breaking of functionality when numbers of people contributing to a same project and get deployed easily.

How to do CI/CD with Circleci

After we set up repositories in Circleci, Circleci always reminds us to add a config.yml file. So we need to create a .circleci folder at the top level of the repo. Under the .circleci folder, we create a config.yml file. With this config file, then Circleci will run its workflow defined in the config file.

In Circleci version 2.1, a config.yml file usually consists of three components, executors, jobs and workflows. executors is used to config the images in the Circleci tasks and work directory in Circleci. Defining executors is similar to write a Dockerfile. jobs is used to define all checks, usually consisting of build, test for CI, deploy, release for CD. workflows is used to define the dependency and order of the jobs. We can define the deployment branch in workflows as well.

Here is a sample template of config.yml

version: 2.1
executors:
    name:
      docker:
        # define image
      working_directory: # define working directory
jobs:
    build:
    # check the construction of dependency and environment
      executor: name
      steps:
        - checkout
        # define steps to construct environment
        - run:
            name: # define name
            command: # add command
    # add other jobs
    test:
    ...
workflows:
    version: 2.1
    flow_name:
      jobs:
        - build
        - test:
            requires:
              - build
        # above means test job will be executed after build step
        # if steps depends on the same step, those steps will run parallel in circleci

I am going to cover two examples to demonstrate details about how to write the config file.

  • How to add test database in Circleci
  • How to deploy static website content in Circleci

How to add test database in Circleci

When we have some database functionality needed to be tested, setting a test database in Circleci is necessary.

This is how the config.yml roughly looks like,

version: 2.1
executors:
    # can change to other name
    name:
      docker:
        # If the codebase is python
        - image: circleci/python:3.7.4
          environment:
            # add database url as environment variable in circle
            TEST_DATABSE_URL: postgresql://root@localhost:5432
        # database image is required as well for database test
        - image: circleci/postgres
      working_directory: ~/repo
jobs:
    build:
      executor: name
      steps:
        - checkout
        - run:
            name: Install depenedency
            command: |
                            # command to install dependency
        # following is how to set up database
        - run: sudo apt-get update
        - run: sudo apt-get install postgresql-client-11
        # then we can create user or database so that we can run test
        - run:
            name: Setup database
            command: |
                            # command to set up database with postgresql-cli
        - run:
            name: migrate table
            command: |
              # need to exist migration in reop so that all tables can be migrated in circleci
              # in other words, migration can guarantee the test database have the same tables as
              # real database              
    # then add other jobs
    test:
      ...

I only include the database set up part, which I spent most of my time on when I first tried to add database test in Circleci. For the rest of jobs, they depends on the actual requirements of CI/CD. The way to write the config file is like writing a bash file, adding what we would like to do in the Circleci workflow with specific syntax line by line. Finally, define the workflows with existing jobs and then it is good to go.

How to deploy static website content in Circleci

I also tried to add Circleci to my personal website repo so that every time I merge new content in master, Circleci can automatically help me sync my new content to my S3 bucket.

This is how the config.yml roughly looks like,

# similar with other config files
version: 2
jobs:
  build:
    docker:
      - image: cibuilds/hugo:latest
    working_directory: ~/repo
    environment:
      HUGO_BUILD_DIR: ~/repo/public
    steps:
      - checkout
      - run:
          name: Add Theme
          command: |
            mkdir themes
            cd themes
            git clone git@github.com:luizdepra/hugo-coder.git            
      - run:
          name: Generate Public Folder
          command: |
                        HUGO_ENV=production hugo -v -d $HUGO_BUILD_DIR
      - run:
          name: Install AWS Cli
          command: |
            apt-get update && apt-get -y install python-pip
            pip install awscli            
      - run:
          name: Test Generated HTML Files
          command: |
            htmlproofer $HUGO_BUILD_DIR --allow-hash-href --check-html \
            --empty-alt-ignore --disable-external            
      - deploy:
          name: deploy to AWS
          command: |
            if [ "${CIRCLE_BRANCH}" = "master" ]; then
              aws s3 sync $HUGO_BUILD_DIR s3://zhen404.com --delete
            else
              echo "Not master branch, dry run only"
            fi

This config file is a little bit different from previous one. I wrote all procedures in one pipeline instead of creating multiple components for a workflow. After I figure out how to restore the environment in build step, I will refactor this config file in the similar way as the previous example.

Lesson Learned: I felt like I failed because I didn’t pay for circleci for the disk space issue before. But the real reason is that I miss the -y flag while using apt-get. With this flag, the apt-get will not ask if I would like to continue when disk space reach to a limit. Also, when the workflow involve AWS operation, we need to add the aws credentials in Circleci setting.

Bottom Line

After knowing how config.yml is written, I think CI/CD looks not that hard in practice. But it’s worth writing more config files when I get chance so that I can have more experience on CI/CD different situations.

comments powered by Disqus