Opinionated DevOps Stuff

[Terragrunt GitOps - Part 5] Operations

Piotr's photo
Piotr
·

5 min read

Table of contents

Introduction

This article will present the juicy part and the one that would occupy most of the engineering time - operations.

The plan is to:

  • Run some modules in the check environment of Customer 1 (remember that check environment for me means "deploy all the infrastructure apart from the portion that is located in the customer's perimeter).

  • Add new prod environment for Customer 1.

  • Deploy all the modules that we have in the check environment, but add to that module invocation that provisions infra on the customer's perimeter - we'll use impersonation for that.

Running modules for a customer

We onboarded Customer 1 in the "check" environment (ref. to article 1 in the series to understand this type of "environment").

Now let's deploy some stuff for them!

This link presents lines added or amended in the commit. Let's begin explanations with the deepest level - envs/customers/customer1/check/random/terragrunt.hcl

This file describes which module we invoke and the associated parameters. First of all, we can see that by usinginclude blocks we can add files terragrunt.hcl and provider.hcl as if their content were to be copied into the current file. The first file in the parent directory will be added.

terraform block includes the path of the module to be invoked. Here we see one of the great qualities of Terragrunt - we can set a specific version (tag) of the module for a given customer and environment. Finally, we pass variables to the module in the inputs tag - notice that we're referring to the local variable defined in the parent terragrunt.hcl file. This local file is a return value of the merge function that ingested local variables at various levels and merged them, taking the most specific variables as the last argument (which really matters here).

Quite the same reasoning can be applied to the terragrunt.hcl file in the storage directory. There are some significant additions here, though.

We notice a generate block that creates a provider config. Again, a great feature of Terragrunt - we can override the provider config (present at the higher level) by writing a new config in a more specific file (module invocation in this case).

Moreover, we see the dependency block. This is necessary to create a "depends on" relationship between the modules' invocations. Here we come across a Terragrunt limitation:

Terragrunt will return an error indicating the dependency hasn’t been applied yet if the terraform module managed by the terragrunt config referenced in a dependency block has not been applied yet. This is because you cannot actually fetch outputs out of an unapplied Terraform module, even if there are no resources being created in the module.

(source)

Unlike vanilla Terraform, Terragrunt requires us to define the dependencies explicitly. There's a way to avoid such blocks, but I will come back to this topic in the last article. For now, let's just remember that plan job of a storage module will use mock values if the outputs of the module it depends on are not known at the time of execution.

Lastly, I added a few local values at 2 levels, customer.hcl and common_all_customers.hcl . Those define certain arguments for the modules (for example, random_string_length and and also some customer's variables we haven't used so far (like customer_given_sa - that's the customer-created SA, please refer to the previous article for details on it).

Add a new environment for a customer

Adding a new environment for a customer is very easy now. Let's add a single terragrunt.hcl file in a location envs/onboard/customer1/prod.

Content may be 100% the same as in the check environment; or we can utilize features of Terragrunt and adjust:

  • Docker tag and version of a Terragrunt runner image

  • a version of a runner module (doesn't have to be the same as meta triggers or other environments!)

That's a great benefit: slowly rolling new versions of infra modules for the customers. I'll use an older runner module version for demonstration.

The link to the commit is here.

We're not finished, though. After we merge this "onboard" code but before we deploy any modules in the customer's env, we have to grant the Token Creator role to the Cloud Build-attached SA over the customer-created SA (refer to the last article for details).

Let's do it now:

gcloud iam service-accounts add-iam-policy-binding customer1-solution-sa@prj-customer1-outside-org.iam.gserviceaccount.com --member="serviceAccount:customer-1-cb-exec-prod@prj-customer1-dedicated.iam.gserviceaccount.com" --role="roles/iam.serviceAccountTokenCreator" --project prj-customer1-outside-org

Deploying code using impersonation

This part is one of my favorites. In the previous step, we added the prod environment for Customer 1. This means that we can finally provision some stuff in their perimeter!

The code for this section is here. The content of other files has already been explained, so let's focus on envs/customers/customer1/prod/storage_impersonate/terragrunt.hcl

I have added the extra_arguments block inside the terraform one to pass certain environmental variables. You can see that I have included the GOOGLE_IMPERSONATE_SERVICE_ACCOUNT variable to impersonate SA (as described here). This means that for this specific module, we'll use an impersonated account to run Terraform. That's all the set-up you need (much easier than vanilla TF!). For details about the block used, check out the docs.

In the GCP audit logs, we have a clear trace of who created the GCS bucket in the customer's environment (pasting parts of the log entry):

{
  "protoPayload": {
    "@type": "type.googleapis.com/google.cloud.audit.AuditLog",
    "status": {},
    "authenticationInfo": {
      "principalEmail": "customer1-solution-sa@prj-customer1-outside-org.iam.gserviceaccount.com",
      "serviceAccountDelegationInfo": [
        {
          "firstPartyPrincipal": {
            "principalEmail": "customer-1-cb-exec-prod@prj-customer1-dedicated.iam.gserviceaccount.com"
          }
        }
      ]
    },
    "serviceName": "storage.googleapis.com",
    "methodName": "storage.buckets.create",
    "resourceLocation": {
      "currentLocations": [
        "europe-west3"
      ]
    }
  }
}

A clear trace of impersonation here (serviceAccountDelegationInfo vs. principalEmail).

Conclusion

Hopefully, I've been able to prove how easy it is now to specify module versions, deploy to various customers and environments, update variables (and override them), as well as use impersonation.

Adding a new customer/environment is now just a matter of adding one or a couple of files and we're good to go.

Let's jump onto the last article of the series to make some conclusions and say goodbyes!

GCPTerraformterragruntci-cdgitops