Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
25 changes: 25 additions & 0 deletions .devcontainer/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
<!--
Copyright 2026 Google LLC

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->

## Development Environment

**To use the Dev Container:**

1. Open project in **VS Code**.
2. Install the **Dev Containers** extension if not already installed.
3. When prompted (or via the Command Palette `F1`), select **Dev Containers: Reopen in Container**.
4. Once the container builds, you will have a terminal with all tools ready to use.
5. Run `gcloud auth login` and `gcloud auth application-default login` to authenticate.
105 changes: 105 additions & 0 deletions .gitlab/ci/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
# Terraform Landing Zone Automation Pipeline

## Overview
This repository contains the Continuous Deployment (CD) pipeline for our Terraform-based Landing Zone (LZ). The pipeline is designed to automate infrastructure provisioning, executing `terraform plan` and `terraform apply` whenever code is merged into the `main` branch.

To maintain security, separation of duties, and modularity, the pipeline is divided into distinct stages and utilizes dynamic child pipelines to manage multi-tenant infrastructure efficiently.

---

## Architecture & Design

### Authentication
All authentication from the GitLab instance to Google Cloud is handled securely through **Workload Identity Federation (WIF)**.
* GitLab authenticates via an OpenID Connect (OIDC) provider configured in Google Cloud.
* The WIF pool ID and prefix are passed as GitLab CI/CD variables to authorize the runner.
* This setup allows the pipeline to impersonate specific service accounts for each stage, ensuring strict separation of duties and least-privilege access.

### Pipeline Jobs & Stages
The pipeline is highly modular. Core infrastructural changes go through standard stages.

| Job Name | Description |
| :--- | :--- |
| `compliance` | Runs immediately after the Stage 1 plan. Cross-checks the plan against allowed YAML configurations to ensure no services outside the compliance catalog (or compliance COA decisions) are utilized. |
| `<Stage_number>-plan` | Runs for Stages 0-4. Previews infrastructure changes on Merge Requests and `main`. Downloads tools, tfvars, and providers. Saves the plan as a GitLab artifact. |
| `<Stage_number>-apply` | Executes **only** on the `main` branch. Uses the saved plan artifact from the previous step to provision approved changes. |

### Execution Flow & Dependencies
Because of the modular design, the pipeline intelligently determines which stages need to run based on the files changed:

* **Changes in Stage 0:** Runs only Stage 0.
* **Changes in Stage 1:** Runs Stage 1, 2, 3, 4, and 5.
* **Changes in Stage 2:** Runs Stage 2, 3, 4, and 5.
* **Changes in Stage 3:** Runs Stage 3 and 5.
* **Changes in Stage 4:** Runs Stage only stage 4.
* **Changes in stage 5 or `tenants.yml`:** Runs only Stage 5.

*(Note: The pipeline automatically triggers the **Plan** stage, but will pause and wait for a **manual trigger** before executing the **Apply** stage).*

---

## Setup & Integration Guide

The following steps guide you through bootstrapping a new environment so that the automated CI/CD pipeline can take over subsequent deployments.

> **Best Practices**
> * Keep variable names exactly the same as documented.
> * Maintain the exact format for variable values.

### 1. Local Configuration

Authenticate your local `gcloud` environment:
```bash
gcloud auth login
gcloud auth application-default login
```

Create and set up your `config.env` file (use `config.env.sample` for reference). Be sure to include the Workload Identity Federation variables:
```env
# Workload Identity Federation via GitLab
CI_PROJECT_PATH=google-cloud/enterprise-lz
# If deployment is for prod this should be main or the equivelent protected branch
CI_COMMIT_BRANCH=<Branch name you are testing on>
GITLAB_URL=[https://gitlab.example.com](https://gitlab.example.com)
```

Update the `tenants.yml` file located in the root directory to reflect the initial tenants you are deploying via `automation/tenant.sh`.

### 2. Initial Bootstrap (Stages 0 & 1)

To allow the CI/CD pipeline to assume control, you must manually deploy Stages 0 and 1 locally first. This establishes the Workload Identity Federation and impersonation service accounts.

**Run `./stellar-engine-deploy.sh`**
1. Go through all of stage 0
2. Run the through the first and second apply for stage one and update providers and migrate state, stop at the third apply without the bootstrap user.

🎉 **Success:** Workload Identity Federation should now be active. You can verify this by checking the `-il5-prod-iac` project in your GCP Console under **IAM > Workload Identity Federation**.

---

## GitLab Configuration

Now that WIF is set up in Google Cloud, you need to configure the connection variables in GitLab.

1. **Gather your WIF Provider Variable:**
Run the following command from the `1-assured-workload` directory to extract the exact WIF path:
```bash
cd fast/stages-aw/1-assured-workload
terraform output -json workload_identity_pool | jq -r '.providers."gitlab-fed".audiences[1] | sub("[https://iam.googleapis.com/](https://iam.googleapis.com/)"; "")'
```
*The output will look similar to: `projects/<project-ID>/locations/global/workloadIdentityPools/<WIF-pool-name>`*

2. **Set GitLab Variables:**
* Navigate to **Settings > CI/CD** in your GitLab repository and expand the **Variables** section.
* Edit the `PREFIX` variable and replace the value with your deployment's prefix.
* Edit the `GCP_WORKLOAD_IDENTITY_PROVIDER` variable and paste the filepath outputted from the `jq` command above.
* Click **Save Changes**.

---

## Usage

### Running the Pipeline
* Commits pushed to `fast/stages-aw` will trigger the pipeline automatically based on the dependency rules outlined in the Architecture section.
* To view a plan, click on the **logs** for the specific plan stage in the GitLab UI.
* To apply changes, click the **Play** button on the manual apply stage.
114 changes: 114 additions & 0 deletions .gitlab/ci/shared-services/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,114 @@
# Shared Services CI/CD

Each shared service runs as an independent child pipeline triggered from `.gitlab-ci.yml`. Services use shared templates from `templates.yml` to reduce boilerplate.

## Templates Reference

Defined in `.gitlab/ci/shared-services/templates.yml`:

| Hidden Job | Purpose | What child jobs override |
|---|---|---|
| `.shared-service-init` | `before_script` preamble: `cd`, WIF auth, tf_dependencies, sets `$GCS_BUCKET` | N/A — referenced via `!reference`, not `extends` |
| `.shared-service-plan` | Sets `stage: plan`, `extends: .terraform_plan_template`, `GCP_SERVICE_ACCOUNT` | `variables` (TF_DIR, SERVICE_NAME), `before_script`, `rules` |
| `.shared-service-apply` | Sets `stage: apply`, `extends: .terraform_apply_template`, `GCP_SERVICE_ACCOUNT`, full `before_script` | `variables` (TF_DIR, SERVICE_NAME), `rules` |

## Adding a New Shared Service

### 1. Create the Terraform module

Add the module under `fast/stages-aw/shared-services/<service>/`. Include a `data/config.yml` if the service needs YAML-based configuration.

### 2. Create the CI pipeline file

Create `.gitlab/ci/shared-services/<service>.yml`. Use an existing service as a starting point (e.g., `ntp.yml` for standard services, `bcap.yml` for simpler ones).

Required structure:

```yaml
stages:
- plan
- apply

include:
- local: '.gitlab/ci/templates/templates.yml'
- local: '.gitlab/ci/shared-services/templates.yml'
- local: '.gitlab/ci/templates/tf_dependencies_public.yml'
rules:
- if: $GCP_TF_REGISTRY_TYPE == "PUBLIC" || $GCP_TF_REGISTRY_TYPE == null || $GCP_TF_REGISTRY_TYPE == ""
- local: '.gitlab/ci/templates/tf_dependencies_internal.yml'
rules:
- if: $GCP_TF_REGISTRY_TYPE == "PRIVATE_DW"

plan-<service>:
extends: .shared-service-plan
variables:
TF_DIR: "fast/stages-aw/shared-services/<service>"
SERVICE_NAME: "<service>"
before_script:
- !reference [.shared-service-init, before_script]
# Download tfvars your service needs:
# - gcloud storage cp ${GCS_BUCKET}/tfvars/1-assured-workload.auto.tfvars.json ./
# - gcloud storage cp "${GCS_BUCKET}/tfvars/3-networking-shared-services.auto.tfvars.json" ./
- gcloud storage cp "${GCS_BUCKET}/providers/shared-services-providers.tf" ./
- yq '.' data/config.yml > <service>-config.auto.tfvars.json
- |
cat <<EOF ><service>-global.auto.tfvars
prefix = "${PREFIX}"
EOF
- terraform init -backend-config=prefix=$SERVICE_NAME
rules:
- if: '$DEPLOY_STAGES =~ /\b(SERVICES|<UPPER_SERVICE>)\b/'
when: on_success
- if: '$PREFIX != null && $PREFIX != "" && ($CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH || $CI_PIPELINE_SOURCE == "merge_request_event")'
when: on_success

apply-<service>:
extends: .shared-service-apply
variables:
TF_DIR: "fast/stages-aw/shared-services/<service>"
SERVICE_NAME: "<service>"
rules:
- if: '$DEPLOY_STAGES =~ /\b(SERVICES|<UPPER_SERVICE>)\b/'
when: manual
- if: '$PREFIX != null && $PREFIX != "" && $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'
when: manual
```

Replace `<service>` with the lowercase service name and `<UPPER_SERVICE>` with the uppercase name used in `DEPLOY_STAGES`.

### 3. Add the trigger in `.gitlab-ci.yml`

Add a trigger block under the `deploy-services` stage:

```yaml
pipeline-deploy-<service>:
stage: deploy-services
trigger:
include:
- local: .gitlab/ci/shared-services/<service>.yml
forward:
pipeline_variables: true
strategy: depend
rules:
- if: '$DEPLOY_VALID != "true"'
when: never
- if: '$DEPLOY_STAGES && $DEPLOY_STAGES !~ /\b(SERVICES|<UPPER_SERVICE>)\b/'
when: never
- if: '$DEPLOY_STAGES && $DEPLOY_STAGES =~ /\b(SERVICES|<UPPER_SERVICE>)\b/'
when: on_success
- if: $CI_COMMIT_BRANCH == "main" || $CI_PIPELINE_SOURCE == "merge_request_event"
changes:
- fast/stages-aw/3-networking/**/*
- fast/stages-aw/shared-services/<service>/**/*
when: on_success
- when: never
```

### 4. Update the deploy script

Add the service to `automation/shared-services-deploy.sh` following the existing pattern (GCS download, terraform init/plan/apply).

### 5. Triggering

- **Automatic**: pipelines trigger on changes to `fast/stages-aw/shared-services/<service>/**/*` or `fast/stages-aw/3-networking/**/*`
- **Manual**: set the `DEPLOY_STAGES` pipeline variable to include `<UPPER_SERVICE>` (e.g., `NTP`, `DNS,SMTP`) or `SERVICES` to trigger all shared services
Loading