AWS CodePipeline - CI, the Amazon Way

I recently began work on an application, written in Golang, that is ultimately intended to replace our MongoDB cluster backup solution. I, being the curious type, decided to try out a few AWS services that I'd never used before. My objective was to setup a simple three stage pipeline - get source code, compile it, then deploy it. Note the glaring omission of "test it" - this is a choice made out of a current lack of knowledge of Golang by the author, not a limitation of these services. This article chronicles my experience with CodeBuild, CodeDeploy and CodePipeline.

I use multiple cloud providers both at work and on personal projects, so Terraform is a natural choice to deal with them. CodeDeploy and CodeBuild resources are already supported by TF - and while it isn't documented as of the time of writing since it was only merged this week, TF now also supports defining a CodePipeline...pipeline. I plan to use TF to create all three. I in no way claim this implementation is complete or perfect as I'm new to CI systems relative to many who may read this. The application that it'll be building is also my first real foray into Golang and very much in its infancy, so (constructive!) criticism is welcome!

Down to it, then. The code can be found here.

File Structure

.
├── appspec.yml
├── buildspec.yml
├── LICENSE
├── logger/
│   └── logger.go
├── lvmsnap/
│   └── lvmsnap.go
├── mongosnap.go
├── README.md
├── s3upload/
│   └── s3upload.go
└── terraform/
    ├── base/
    │   └── vars.tf
    ├── codebuild/
    │   ├── base-vars.tf -> ../base/vars.tf
    │   ├── iam-servicerole.tf
    │   ├── codebuild.tf
    │   └── .terraform/
    │       └── terraform.tfstate
    ├── codedeploy/
    │   ├── base-vars.tf -> ../base/vars.tf
    │   ├── iam-servicerole.tf
    │   ├── codedeploy.tf
    │   └── .terraform/
    │       └── terraform.tfstate
    └── codepipeline/
        ├── base-vars.tf -> ../base/vars.tf
        ├── iam-servicerole.tf
        ├── pipeline.tf
        ├── s3-bucket.tf
        └── .terraform/
            └── terraform.tfstate

As you can see, the repo contains a golang application, a terraform directory, and a pair of files - appspec.yml and buildspec.yml. Appspec.yml is used by CodeDeploy and buildspec.yml is used by CodeBuild. These are very straightforward and, for my use case, also very simple.

Base Variables

These are used across all three Terraform "environments" - I tend to separate Terraform components to limit blast my blast radius when making or automating changes. This does however mean that when things depend on one another - for instance, my CodePipeline depends on the presence of both a CodeBuild project and a CodeDeploy Application/Deployment Group - you need to build those components first. If it were one environment, TF would take care of resolving that dependency for you. FYI!

Those familiar with Terraform variables will notice that some of these are "blank" - this is because, since the repo is public, I choose to omit some secrets from source code and feed them in using terraform apply -var-file=/path/to/secrets.tfvars.

provider "aws" {  
  profile = "${var.aws-credentials-profile}"
}

variable "region" {  
  default = "us-east-1"
}

variable "aws-credentials-profile" {}

variable "artifact-bucket" {}

variable "state-bucket" {}

variable "codedeploy-slack-endpoint" {}

variable "github-oauth-token" {}

data "terraform_remote_state" "codebuild" {  
  backend = "s3"

  config {
    bucket  = "${var.state-bucket}"
    key     = "dev/codebuild.tfstate"
    profile = "${var.aws-credentials-profile}"
  }
}

data "terraform_remote_state" "codedeploy" {  
  backend = "s3"

  config {
    bucket  = "${var.state-bucket}"
    key     = "dev/codedeploy.tfstate"
    profile = "${var.aws-credentials-profile}"
  }
}

data "terraform_remote_state" "codepipeline" {  
  backend = "s3"

  config {
    bucket  = "${var.state-bucket}"
    key     = "dev/codepipeline.tfstate"
    profile = "${var.aws-credentials-profile}"
  }
}

You may alto note the presence of .terraform/terraform.tfstate files in each TF directory - while these will be disappearing in the soon to be released Terraform 0.9, it's worth mentioning that I'm not storing any state information in git - just skeletal files that reference the actual remote state, stored in s3:

{
    "version": 3,
    "serial": 1,
    "remote": {
        "type": "s3",
        "config": {
            "bucket": "mongosnap-tfstate",
            "key": "dev/codebuild.tfstate",
            "profile": "tastycidr",
            "region": "us-east-1"
        }
    },
    "modules": []
}

CodeBuild

You'll need an IAM role for the CodeBuild service to use - it logs to CloudWatch and uploads to S3.

resource "aws_iam_role" "codebuild_role" {  
  name = "codebuild"

  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "codebuild.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF  
}

resource "aws_iam_policy" "codebuild_policy" {  
  name        = "codebuild-policy"
  path        = "/service-role/"
  description = "Policy used in trust relationship with CodeBuild"

  policy = <<POLICY
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Resource": [
        "*"
      ],
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents",
        "s3:CreateBucket",
        "s3:GetObject",
        "s3:List*",
        "s3:PutObject"
      ]
    }
  ]
}
POLICY  
}

resource "aws_iam_policy_attachment" "codebuild_policy_attachment" {  
  name       = "codebuild-policy-attachment"
  policy_arn = "${aws_iam_policy.codebuild_policy.arn}"
  roles      = ["${aws_iam_role.codebuild_role.id}"]
}

And of course the CodeBuild project itself:

resource "aws_codebuild_project" "mongosnap" {  
  name         = "mongosnap"
  description  = "builds mongosnap golang binary"
  timeout      = "30"
  service_role = "${aws_iam_role.codebuild_role.arn}"

  artifacts {
    type = "CODEPIPELINE"
  }

  environment {
    compute_type = "BUILD_GENERAL1_SMALL"
    image        = "aws/codebuild/golang:1.7.3"
    type         = "LINUX_CONTAINER"
  }

  source {
    type = "CODEPIPELINE"
  }

  tags {
    "Environment" = "dev"
  }
}

output "project-name" {  
  value = "${aws_codebuild_project.mongosnap.name}"
}

The "image" - aws/codebuild/golang:1.7.3 - is a pre-existing Amazon managed container image and requires no setup. The artifact and source blocks list CODEPIPELINE as their types - you'll see why that is later on. I've chosen the smallest build container and the latest version of Golang. You'll note that I've also exported the project name as an output - we'll use this later on while defining our Pipeline.

Additionally, we need to tell CodeBuild what it actually needs to do to compile our Golang code:

buildspec.yml

version: 0.1

phases:  
  install:
    commands:
      - go get -u github.com/golang/lint/golint
      - go get -u github.com/crielly/mongosnap/logger
      - go get -u github.com/rlmcpherson/s3gof3r
      - go get -u github.com/docopt/docopt-go

  pre_build:
    commands:
      - golint -set_exit_status

  build:
    commands:
      - echo Build started on `date`
      - echo Compiling the Golang binary...
      - go build -o mongosnap
  post_build:
    commands:
      - echo Build completed on `date`
artifacts:  
  files:
    - mongosnap
    - appspec.yml

This is one area where I give a definite edge to CircleCI over CodeBuild - my project is small enough that it isn't a big deal, but it's a huge quality of life advantage on a bigger project - here I have to manually list imports and ensure they are installed, or the build will fail. Circle detects and installs them for you. Note that as artifacts, I'm exposing mongosnap (the compiled binary) and appspec.yml - the file that CodeDeploy needs in order to know what to do with our compiled binary.

CodeDeploy

We need to define a CodeDeploy Application and a Deployment Group. We can also, optionally, define a Deployment Configuration. First though, another IAM Service Role:

resource "aws_iam_role_policy" "codedeploy-policy" {  
  name = "codedeploy-policy"
  role = "${aws_iam_role.codedeploy-role.id}"

  policy = <<EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
          "Action": [
                "autoscaling:CompleteLifecycleAction",
                "autoscaling:DeleteLifecycleHook",
                "autoscaling:DescribeAutoScalingGroups",
                "autoscaling:DescribeLifecycleHooks",
                "autoscaling:PutLifecycleHook",
                "autoscaling:RecordLifecycleActionHeartbeat",
                "ec2:DescribeInstances",
                "ec2:DescribeInstanceStatus",
                "tag:GetTags",
                "tag:GetResources",
                "sns:Publish",
                "s3:GetObject",
                "s3:ListObject"
            ],
            "Resource": "*"
        }
    ]
}
EOF  
}

resource "aws_iam_role" "codedeploy-role" {  
  name = "codedeploy-role"

  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "Service": [
          "codedeploy.amazonaws.com"
        ]
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF  
}

And the Application/Deployment Group itself:

resource "aws_codedeploy_app" "mongosnap" {  
  name = "mongosnap"
}

resource "aws_codedeploy_deployment_group" "mongosnap-dev" {  
  app_name               = "${aws_codedeploy_app.mongosnap.name}"
  deployment_group_name  = "mongosnap-dev"
  service_role_arn       = "${aws_iam_role.codedeploy-role.arn}"
  deployment_config_name = "CodeDeployDefault.OneAtATime"

  ec2_tag_filter {
    key   = "mongosnap"
    type  = "KEY_AND_VALUE"
    value = "true"
  }

  auto_rollback_configuration {
    enabled = true
    events  = ["DEPLOYMENT_FAILURE"]
  }
}

output "application-name" {  
  value = "${aws_codedeploy_app.mongosnap.name}"
}

output "deployment-group-name" {  
  value = "${aws_codedeploy_deployment_group.mongosnap-dev.deployment_group_name}"
}

CodeDeploy also relies upon an appspec.yml being present in the root of your repo - mine is extremely simplistic as all I want to do is take the compiled binary and stick it in my server's $PATH for later use:

version: 0.0  
os: linux  
files:  
  - source: mongosnap
    destination: /usr/local/bin

The app definition is basically nothing. The deployment group in this case simply looks for a tag - mongosnap = true - on ec2 instances and deploys to them if it's found. Deploying to specific Autoscaling Groups is the other method I've used in the past, use whichever meets your needs. Because I have a very simple App and setup, I'm also choosing the CodeDeploy default - OneAtATime - for my deploymentconfigname. You can also select half at once, all at once, or create your own (custom). This can also be defined in TF code via aws_codedeploy_deployment_config.

There are a couple of additional requirements for servers with which you want to use CodeDeploy - first of all, it requires an agent! I've created a quick and easy Ansible role to deal with that requirement:

codedeploy-agent/defaults/main.yml:

codedeploy_apt_dependencies:  
  - python-pip
  - ruby

codedeploy_pip_dependencies:  
  - paramiko

pip_retry_num: 3  
pip_retry_delay: 5

apt_retry_num: 3  
apt_retry_delay: 5  

codedeploy-agent/tasks/main.yml:

- name: gather EC2 facts
  action: ec2_facts

- name: Ensure apt dependencies are present
  apt: 
    pkg: "{{ item }}"
    state: latest
  with_items: "{{ codedeploy_apt_dependencies }}"
  register: result
  until: result | success
  retries: "{{ apt_retry_num }}"
  delay: "{{ apt_retry_delay }}"

- name: Ensure pip dependencies are present
  pip: 
    name: "{{ item }}"
    state: latest
  with_items: "{{ codedeploy_pip_dependencies }}"
  register: result
  until: result | success
  retries: "{{ pip_retry_num }}"
  delay: "{{ pip_retry_delay }}"

- name: download codedeploy agent setup script
  get_url:
    url: https://aws-codedeploy-{{ ansible_ec2_placement_region }}.s3.amazonaws.com/latest/install
    dest: /tmp/codedeployagent-install.sh
    mode: 0500

- name: run codedeploy agent setup script
  shell: /tmp/codedeployagent-install.sh auto
  args:
    creates: /opt/codedeploy-agent/bin/codedeploy-agent

- name: run and enable codedeploy agent
  service:
    name: codedeploy-agent
    state: started
    enabled: yes

CodePipeline

This is the part that sews the other ones together. It has only just received Terraform support and isn't very well documented, but I didn't find it too difficult to figure out. One more IAM service role:

resource "aws_iam_role" "mongosnap-codepipeline" {  
  name = "mongosnap-codepipeline"

  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "codepipeline.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF  
}

resource "aws_iam_role_policy" "mongosnap-codepipeline" {  
  name = "mongosnap-codepipeline"
  role = "${aws_iam_role.mongosnap-codepipeline.id}"

  policy = <<EOF
{
    "Statement": [
        {
            "Action": [
                "s3:GetObject",
                "s3:GetObjectVersion",
                "s3:GetBucketVersioning",
                "s3:PutObject",
                "ec2:*",
                "iam:PassRole",
                "codepipeline:*",
                "codedeploy:CreateDeployment",
                "codedeploy:GetApplicationRevision",
                "codedeploy:GetDeployment",
                "codedeploy:GetDeploymentConfig",
                "codedeploy:RegisterApplicationRevision",
                "codebuild:BatchGetBuilds",
                "codebuild:StartBuild"
            ],
            "Resource": "*",
            "Effect": "Allow"
        }
    ],
    "Version": "2012-10-17"
}
EOF  
}

And the Pipeline:

resource "aws_codepipeline" "mongosnap" {  
  name     = "mongosnap"
  role_arn = "${aws_iam_role.mongosnap-codepipeline.arn}"

  artifact_store {
    location = "${aws_s3_bucket.mongosnap.bucket}"
    type     = "S3"
  }

  stage {
    name = "Source"

    action {
      name             = "github-checkout"
      category         = "Source"
      owner            = "ThirdParty"
      provider         = "GitHub"
      version          = "1"
      output_artifacts = ["mongosnap-source"]

      configuration {
        OAuthToken = "${var.github-oauth-token}"
        Owner      = "crielly"
        Repo       = "mongosnap"
        Branch     = "master"
      }
    }
  }

  stage {
    name = "Build"

    action {
      name             = "mongosnap-compile"
      category         = "Build"
      owner            = "AWS"
      provider         = "CodeBuild"
      input_artifacts  = ["mongosnap-source"]
      output_artifacts = ["mongosnap-build"]
      version          = "1"

      configuration {
        ProjectName = "${data.terraform_remote_state.codebuild.project-name}"
      }
    }
  }

  stage {
    name = "Deploy"

    action {
      name            = "binary-deploy"
      category        = "Deploy"
      owner           = "AWS"
      provider        = "CodeDeploy"
      input_artifacts = ["mongosnap-build"]
      version         = "1"

      configuration {
        ApplicationName     = "${data.terraform_remote_state.codedeploy.application-name}"
        DeploymentGroupName = "${data.terraform_remote_state.codedeploy.deployment-group-name}"
      }
    }
  }
}

The documentation on the Pipeline itself isn't on the Terraform website yet - but it can be found on Github. It is, however, very incomplete. I had to build this resource using a certain element of trial and error - luckily, the internal validation seems better than what the docs indicate and TF didn't let me apply any dumb configurations. Most of this should be self explanatory - but note the artifact names. You need to checkout code and pass that from the Source stage's Output Artifact to the Build stage's Input Artifact - then do the same thing from the Build to Deploy state. All artifact names must be unique unless you enjoy unpredictable results. The github oauth token is obviously generated within Github itself. Another thing I found confusing was the "repo" value in the Source stage - this isn't a repo .git address as you'd normally use for cloning, it's literally just the repo name.

Now you're ready for commits. As soon as you commit something to the branch you specified in the pipeline config, you'll see a build kick off automagically.

If you login to CodePipeline, you'll see this:
Imgur

Not the shiniest of GUIs, but functional - and if you logged into one of my servers tagged with mongosnap = true:

$ which mongosnap
/usr/local/bin/mongosnap

My binary has been successfully deployed. It appears to keep the permissions it's given when compiled:

-rwxr-xr-x  1 root root 6.0M Mar  5 04:00 mongosnap

Ready to execute.

Conclusion (UPDATED)

My experience with these three services was mostly positive. The configuration and methodology largely made sense to me. I don't care for the CodePipeline GUI, and I wish there was an easy way to integrate with Github to build from branches other than the default (aka usually master). Regardless, this problem is solvable if you like the services enough to choose them over their many worthy competitors. The main draw here is CodeDeploy - there are lots of build/CI services, but I am aware of no tool that provides the capabilities CodeDeploy has in terms of tying directly into AutoScaling groups/EC2 Tag groups and performing rolling deploys. Of course, if you're already performing Blue/Green deploys in some other fashion, you probably don't care.

It was an interesting learning experience and, before I choose a CI Pipeline method for a new project, I'd definitely consider using this one.