Opinionated DevOps Stuff

[Terragrunt GitOps - Part 2] Prerequisites & GCP set-up

Piotr's photo
Piotr
·

5 min read

Table of contents

Introduction

Welcome back - I hope you've become acquainted with the first article in the series. In this one, we'll continue our journey and focus on performing the initial GCP set-up. We'll create the "orange" resources (as noted in the previous article) and set the grounds for GitOps to operate smoothly.

However, the "meta" trigger will be the topic of the next article.

Set-up steps

Projects and folders

First of all, we need GCP projects. Recall from the design article that we need 3 projects on the SP perimeter. Then we need 2 projects in the "customer" perimeter. Of course, I'll simulate the latter part and create those projects under the same organization (in real life, however, those would be customer-managed projects belonging to another GCP org).

I'll delete my folders and projects before the publication of this article, so I'll be sharing their names and IDs.

To create them, follow publically available documentation (e.g. for folders), it's very simple.

Enable APIs

We need to enable APIs in the projects. First, let's handle the SP part.

list_of_apis=(cloudbuild.googleapis.com cloudresourcemanager.googleapis.com storage-component.googleapis.com storage-api.googleapis.com iam.googleapis.com secretmanager.googleapis.com)

list_of_projects=(prj-customer1-dedicated prj-customer2-dedicated prj-terragrunt-common)

for project_id in "${list_of_projects[@]}"
do
    for api in "${list_of_apis[@]}"
    do
        gcloud services enable $api --project=$project_id
    done
done

Then let's do the same for the customer projects:

list_of_apis=(cloudresourcemanager.googleapis.com storage-component.googleapis.com storage-api.googleapis.com iam.googleapis.com)
list_of_projects=(prj-customer1-outside-org prj-customer2-outside-org)

for project_id in "${list_of_projects[@]}"
do
    for api in "${list_of_apis[@]}"
    do
        gcloud services enable $api --project=$project_id
    done
done

Create Dockerfile, build, and push to Artifact Registry

When writing the Dockerfile for the Terragrunt runner, I got inspiration from Google's example foundation. I enhanced and adjusted to my needs a little, and that's the result:

ARG GCLOUD_VERSION=453.0.0-slim

# base image
FROM google/cloud-sdk:${GCLOUD_VERSION}

# Use ARG so that values can be overridden by user/cloudbuild
ARG TERRAFORM_VERSION="1.6.3"
ARG TERRAGRUNT_VERSION="0.53.2"

# install common tools
RUN apt-get update && \
    apt-get -y install curl jq unzip git ca-certificates gnupg

# intall terraform
RUN curl https://releases.hashicorp.com/terraform/${TERRAFORM_VERSION}/terraform_${TERRAFORM_VERSION}_linux_amd64.zip --output terraform_${TERRAFORM_VERSION}_linux_amd64.zip && \
    curl https://releases.hashicorp.com/terraform/${TERRAFORM_VERSION}/terraform_${TERRAFORM_VERSION}_SHA256SUMS.sig --output terraform_SHA256SUMS.sig  && \
    curl https://releases.hashicorp.com/terraform/${TERRAFORM_VERSION}/terraform_${TERRAFORM_VERSION}_SHA256SUMS --output terraform_SHA256SUMS && \
    curl https://keybase.io/hashicorp/pgp_keys.asc --output pgp_keys.asc && \
    gpg --import pgp_keys.asc && \
    gpg --verify terraform_SHA256SUMS.sig terraform_SHA256SUMS && \
    grep terraform_${TERRAFORM_VERSION}_linux_amd64.zip terraform_SHA256SUMS | shasum --algorithm 256 --check  && \
    mkdir -p /builder/terraform && \
    unzip terraform_${TERRAFORM_VERSION}_linux_amd64.zip -d /builder/terraform && \
    rm -f terraform_${TERRAFORM_VERSION}_linux_amd64.zip terraform_SHA256SUMS

# install terragrunt
RUN curl -L https://github.com/gruntwork-io/terragrunt/releases/download/v${TERRAGRUNT_VERSION}/terragrunt_linux_amd64 \
    -o /usr/local/bin/terragrunt && \
    chmod +x /usr/local/bin/terragrunt

# install Github CLI
RUN curl -fsSL https://cli.github.com/packages/githubcli-archive-keyring.gpg | dd of=/usr/share/keyrings/githubcli-archive-keyring.gpg && \
    chmod go+r /usr/share/keyrings/githubcli-archive-keyring.gpg && \
    echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/githubcli-archive-keyring.gpg] https://cli.github.com/packages stable main" | tee /etc/apt/sources.list.d/github-cli.list > /dev/null && \
    apt update && \
    apt install gh -y

# purge and remove
RUN apt-get --purge -y autoremove && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

ENV PATH=/builder/terraform/:$PATH

ENTRYPOINT ["/bin/bash"]

Comments for that Dockerfile:

  • It's based on gcloud-sdk image.

  • It contains a lot of tools, so not the slimmest you can use (as per Docker best practices). However, it's general enough for me to justify its usage - in the Cloud Builds steps I often run gcloud commands, curl, git so it comes in handy.

  • You can choose Terraform and Terragrunt versions. For Terraform download, you can see a lengthy part whose purpose is verification of the SHA sums of the package.

  • I install GitHub CLI tool - I'll explain why exactly a bit later. In short, it's necessary for the way I've chosen to authenticate when pulling from private repositories.

The procedure is as follows:

  • Create an Artifact Registry repo by following this article

  • Build & push the Docker image to the repository by following this article

Note. I advise to use of the regions listed here. Egress to the same region is free (so Artifact Registry -> Cloud Build in the same region). And we're a bit limited in terms of Cloud Build trigger location if we use the GCP free trial.

Prepare repositories

In the design article, I mentioned we'd be using 4 repositories. Of course, I share the "public" versions of what I used. But when you try to replicate the setup, you will need those repositories to be private (as they will be in an enterprise environment). In order not to delve into the code right now, I suggest you create 4 private repositories and name them similarly to what I used:

  • terraform-random-sample-module

  • terraform-storage-sample-module

  • terragrunt-runner-module

  • terragrunt-example-envs

They could be empty as of now.

Create a fine-grained token in GitHub

I've chosen GH fine-grained tokens as an authentication mechanism to pull private repositories. Your next step is to create a private access token and add read permission for the 4 repos in its scope.

Follow the guide in the official documentation to achieve it. I have something like:

And grant read access for contents, commit statuses, discussions, issues, merge queues, metadata, pull requests, and variables.

Note. It's a bit excessive, but those read-access permissions may come in handy if you want to expand the solution. Feel free to limit it, though.

Store the token in the Secret Manager

The next step is to create a new secret object in a common project and add a secret version containing GitHub's fine-grained PAT.

Follow the docs to complete it.

Add a Cloud Build Connection in each of the SP projects

For the Cloud Build triggers to respond to events in the terragrunt-example-envs repository, we need to create something that's called a connection. We'll use the "2nd" gen. It's much easier to do it via console than CLI/Terraform and also you don't have to manage the sensitive string in your code, so that part is even preferred to do in GUI in my opinion.

The result is that in each of your SP projects you have something like this:

Conclusion

This article explained how to perform some one-time operations needed for the solution. We didn't deal with "meta" triggers and customer-side set-up, which will be the topic of the next article.

GCPci-cdTerraformterragruntgitops