Google Cloud Landing Zone Series – Part 10: Automation

In the last blog posts, we mostly covered Google related Landing Zones, but never discussed by which tools and techniques landing zones should be built and maintained.

In this blog post, we will cover this topic and give some information about Infrastructure-as-Code (IaC), GitOps and so on.

Motivation

Within the last years, enterprises have already migrated large portions of their workload to the cloud – whether it is a private, public or hybrid cloud. However, many companies still fail to grasp all the benefits of cloud computing. According to the State of DevOps report released by Puppet and DORA, that highlights a DevOps maturity model, the overwhelming majority of companies are struggling with to reach the highest level of DevOps majority. This results in a set of problems to be solved: First, long lead times can be observed. Secondly, processes are often carried out manually, usually initiated by means of tickets created by developers. Third, the all overall complexity of the tools and the way how to integrate them is usually high leading to the fact that developers are overwhelmed. Last, waiting times slow done developers as many companies lack an internal self-service developer platform.

To tackle those issues, highly mature DevOps organizations tend to have their organization having structured in both application teams that are stream-aligned and platform teams Basically, stream-aligned teams are development teams that are building code and putting them into production. On the other hand, platform teams support that teams and might provide among other things:

• Platform servicing such as CI/CD and infrastructure provisioning by using industry-wide best practices such as Infrastructure-as-Code (IaC). Our Landing Zone infrastructure clearly belongs to this point.
• Evangelization and mentoring DevOps practices for promoting cultural values, such as communication, transparency and knowledge sharing.
• Rotary human resources, which means that platform teams might help and provide product teams with human resources when these teams lack of specific skills to accomplish their work.

Background

Platform Teams for cloud-native environments don’t come from nowhere – instead they use state of the art software engineering principles to accelerate software delivery. For infrastructure provisioning Infrastructure-as-Code (IaC) has become a de-facto standard. Basically, the idea behind IaC is to declaratively the describe the target-state of the cloud environment within a source control system. IaC tools then apply all necessary changes to rollout cloud resources in the target environment. IaC principles help to create consistency, speed up deployment, improve accountability, and increase efficiency while also being able to help mitigate unnecessary costs.

GitOps extends the idea of IaC for all kind of deployments – not only infrastructure provisioning. Git remains the single source of truth and deployment information is described in a declarative way as well. In addition, an application deployment tool is to be used and there is a monitoring tool in place, which continuously reconciles the states and carries out changes in the target system if necessary.

Platform Engineering as a basis for Cloud Infrastructure

In many organizations, DevOps teams are solely responsible for all the aspects within a software product. However, when dealing in cloud-environment and possible with Kubernetes, many application teams struggle with efficient processes for infrastructure provisioning and security issues. Platform teams help to address those issues by abstracting infrastructural issues from application teams.

Platform team and application teams can be part of an organization. However, companies might not have the size or the required skills to build up a platform team. In that case, platform engineering can be offered as a service across organizations.

For such offerings, standardization is a key requirement as it increases efficiency for supporting application teams. In addition, tooling is needed that provides support for consuming provided IaC modules as well as having self-service capabilities. For GitOps support and facilitating security best practices, we propose to set up a well-defined structure in a Git repository that enables to go infrastructure and application deployments hand in hand.

• Most important, a DevOps tool with pipeline support in order to run workflows is provided to run workflows.
• The Platform Team provides standardized modules that are optimized in terms of programming and security best practices and are compliant for the organization.
• Application Teams need a way to interact with the Platform Team. In most use cases, using Git is sufficient in order to support IaC pipelines and GitOps workflows. For advanced use cases, a self-service catalog or an internal developer platform can be used.
• Besides infrastructure deployment support, a CI/CD platform should also be provided. Functionality should include at least support for container images builds and app delivery.
• In order to support SRE, observability should be integrated in the platform tooling and all provisioned resources should have observability support included.
• Platform tooling should support product teams in the complete DevOps lifecycle – from building, testing to operating.
• If Application Teams need support, the Platform Team might provide consultancy and human resources to the respective teams.

Choosing the right Platform for Terraform

Using Terraform and IaC is great, but you also need a DevOps platform run running Terraform. Most companies use either GitLab, Azure DevOps or GitHub. We prefer GitLab, as it is the most comprehensible product for Terraform – as it comes with support for Terraform state, has a Terraform registry and provides built-in support for running Terraform. On top, the self-hosted GitLab is free of charge and provides those Terraform features. However, here is a short comparision:

Sample pipeline

We don’t want to dive into the details of Terraform pipelines, but the following shows a simplified Terraform pipeline:

stages:
  - init
  - plan
  - apply
 
variables:
  TF_ROOT: "path/to/terraform/code"
  TF_STATE: "terraform.tfstate"
  TF_PLAN: "terraform.tfplan"
 
# Cache Terraform files to speed up subsequent runs
cache:
  paths:
    - .terraform/
 
before_script:
  - cd $TF_ROOT
  - terraform --version
  - terraform init -input=false
 
# Terraform Init Stage
terraform:init:
  stage: init
  script:
    - terraform init -input=false
  artifacts:
    paths:
      - .terraform
 
# Terraform Plan Stage
terraform:plan:
  stage: plan
  script:
    - terraform plan -input=false -out=$TF_PLAN
  artifacts:
    paths:
      - $TF_PLAN
  dependencies:
    - terraform:init
 
# Terraform Apply Stage
terraform:apply:
  stage: apply
  script:
    - terraform apply -input=false -auto-approve $TF_PLAN
  dependencies:
    - terraform:plan
  when: manual  # Only run apply manually to avoid unintended changes

With this  simple pipline for Terraform as a starting point for GitOps, we will conclude today’s post on IaC and GitOps.

> Click here for Part 11: Landing Zone Logging