GitOps Workflow: Cost Optimization and Automation

Introduction

This might be an unusual way to start an article, but have you ever heard of a developer accidentally deleting an application branch? It happens! When designing and implementing infrastructure, there are many moving parts, especially as developers keep deploying applications. It often becomes tricky to keep track of changes to the infrastructure and deployments. This lack of visibility can make it easy to miss out on important reviews, security checks, and branch rules, which are crucial for safely getting code changes into the live infrastructure.

However, challenges like these are what drive companies to evolve their operational workflows through a cultural shift known as DevOps, leading to the solution we call GitOps. GitOps is a workflow pattern that uses Git as a single source of truth, ensuring that the state of your current repository configuration matches your live infrastructure. Your infrastructure and deployments are version-controlled and auditable. GitOps also enhances security and collaboration by using pull requests to manage code changes, requiring reviews, status checks, and discussions before merging.

In this project, we’ll use GitOps principles to simplify infrastructure and application deployment. We’ll leverage GitHub Actions CI/CD with tools like Terraform, Ansible, and Docker to ensure efficient and automated infrastructure provisioning, cost estimation, monitoring setup, and application deployments. Additionally, we’ll implement security measures to ensure that changes are properly reviewed and pass necessary checks before being integrated into the live infrastructure.

This article will focus on the GitOps workflow and CI/CD. If you want to learn more about IAC, monitoring, and application deployment, check out the following articles.

Prerequisites

To follow this tutorial, you need:

Basic knowledge of Git and GitHub Actions
Familiarity with Terraform and Ansible

Objectives

The primary goals of this project include:

Cloud Cost Optimization: Leverage InfraCost to analyze and optimize infrastructure costs before provisioning.
GitOps Workflows: Implement a declarative and auditable approach for seamless automation.
Terraform and Ansible Integration: Automate infrastructure provisioning and configure a monitoring stack.
Effective Branching Strategies: Separate branches for infrastructure provisioning and application deployment.

Technologies

Infrastructure Tools

Terraform: Infrastructure as Code tool for cloud resource provisioning.
InfraCost: Provides cloud cost estimation and optimization.
Ansible: Configuration management tool for provisioning and configuring monitoring services.

Containerization and Deployment

Docker: Packages applications into containers for consistent deployment.
Traefik: Reverse proxy to manage container traffic.

Monitoring and Logging Stack

Prometheus: Metrics collection for infrastructure and containers.
Grafana: Visualization of metrics and logs.
cAdvisor: Provides container-level metrics.
Loki: Log aggregation for better observability.
Promtail: Collects and ships logs to Loki.

CI/CD

GitHub Actions: Automates infrastructure provisioning and application deployment.

Project Architecture

The project integrates infrastructure provisioning with CI/CD pipelines, monitoring, and deployment in two phases:

Infrastructure Management: Provisioning and configuring cloud resources and monitoring tools.
Application Deployment: Building, testing, and deploying a three-tier full-stack application.

The monitoring stack runs alongside the application and provides real-time insights into logs and metrics.

Project Structure

This project uses an efficient branching strategy to manage the CI/CD workflows. Below is a summary of the configurations present in the branches.

Infrastructure Branches: infra_main and infra_features

📦 Project Root
│
├── .github/
│   └── workflows/              # GitHub Actions CI/CD workflows
│       ├── ansible-monitoring.yml      # Workflow to trigger Ansible for monitoring stack
│       ├── terraform-apply.yml         # Workflow to apply Terraform configurations
│       ├── terraform-plan.yml          # Workflow to run 'terraform plan' and output cost estimations
│       └── terraform-validate.yml      # Workflow to validate Terraform configurations
│
├── ansible/
│   ├── roles/                  # Directory for Ansible roles (future role definitions)
│   ├── compose.monitoring.yml  # Docker Compose configuration for monitoring services
│   └── playbook.yml            # Main playbook for configuring the monitoring stack
│
├── monitoring/
│   ├── dashboards/             # Directory for custom Grafana dashboards
│   ├── dashboard-providers.yml # Grafana dashboard provider configuration
│   ├── loki-config.yml         # Loki log aggregation configuration
│   ├── loki-datasource.yml     # Grafana datasource configuration for Loki
│   ├── prometheus-datasource.yml # Prometheus datasource configuration for Grafana
│   ├── prometheus.yml          # Prometheus main configuration file
│   └── promtail-config.yml     # Promtail configuration for log scraping
│
├── terraform/
│   ├── ansible.tf              # Terraform resource for triggering Ansible
│   ├── backend.tf              # Backend configuration for Terraform state management
│   ├── dns.tf                  # DNS configurations
│   ├── ec2.tf                  # EC2 resource definitions
│   ├── main.tf                 # Main Terraform entry point
│   ├── output.tf               # Outputs from Terraform resources
│   ├── variables.tf            # Input variables for Terraform configurations
│   └── vpc.tf                  # VPC configuration for network setup
│
└── README.md                   # Documentation and project overview

Application Branches: integration and deployment

📦 Project Root
│
├── .github/
│   └── workflows/             # GitHub Actions CI/CD workflows
│       ├── cd-backend.yml      # Continuous Deployment workflow for backend
│       ├── cd-frontend.yml     # Continuous Deployment workflow for frontend
│       ├── ci-backend.yml      # Continuous Integration workflow for backend
│       └── ci-frontend.yml     # Continuous Integration workflow for frontend
│
├── backend/                   # Backend code and resources
│
├── frontend/                  # Frontend code and resources
│
├── compose.yml                # Docker Compose configuration file for full-stack services
│
└── README.md                  # Project documentation and usage instructions

Pipeline Configuration Files

Infrastructure Pipelines

terraform-validate.yml: This workflow validates Terraform configurations when changes are pushed to the infra_features branch.

name: Terraform Validate

on:
  workflow_dispatch:
  push:
    branches:
      - 'infra_features'

env:
  AWS_ACCESS_KEY: ${{ secrets.AWS_ACCESS_KEY }}
  AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
  TF_VAR_aws_region: ${{ vars.TF_AWS_REGION }}
  TF_VAR_ami_id: ${{ vars.TF_AMI_ID }}
  TF_VAR_instance_type: ${{ vars.TF_INSTANCE_TYPE }}
  TF_VAR_volume_size: ${{ vars.TF_VOLUME_SIZE }}
  TF_VAR_key_pair_name: ${{ vars.TF_KEY_PAIR_NAME }}
  TF_VAR_private_key: ${{ secrets.PRIVATE_KEY }}
  TF_VAR_domain_name: ${{ vars.TF_DOMAIN_NAME }}
  TF_VAR_frontend_domain: ${{ vars.TF_FRONTEND_DOMAIN }}
  TF_VAR_db_domain: ${{ vars.TF_DB_DOMAIN }}
  TF_VAR_traefik_domain: ${{ vars.TF_TRAEFIK_DOMAIN }}
  TF_VAR_cert_email: ${{ secrets.TF_CERT_EMAIL }}
  TF_VAR_private_key_path: ./${{ vars.TF_KEY_PAIR_NAME }}.pem

jobs:
  build-infra:
    name: terraform-ci-cd
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v2

      - name: Write Private Key to File
        run: |
          echo "${{ secrets.PRIVATE_KEY }}" > ${{ vars.TF_KEY_PAIR_NAME }}.pem
          chmod 600 ${{ vars.TF_KEY_PAIR_NAME }}.pem

      - name: Set up Terraform
        uses: hashicorp/setup-terraform@v2

      - name: Terraform Init
        id: init
        run: terraform init
        working-directory: ./terraform

      - name: Terraform Validate
        id: validate
        run: terraform validate
        working-directory: ./terraform

terraform-plan.yml: This workflow runs when a pull request is made to the infra_main branch. It executes terraform plan and generates cost estimates using InfraCost, which are posted as comments in the PRs. Here is a detailed step:
1. - Triggers: Runs on pull request events (opened, synchronize, reopened) and manual dispatch.
    - Environment Setup: Configures AWS and Terraform variables using secrets and repository variables.
    - Terraform Plan: Initializes Terraform, creates a plan, and saves the output.
    - Infracost Analysis:
      - Estimates costs on the base branch.
      - Compares it with the pull request branch.
    - PR Comment: Two comments are added to the pull request. The first shows the cost difference, estimation, and optimization tips. The second shows the Terraform plan and the cost breakdown of the plan.

Note: The Infracost breakdown for the base branch uses values set in the repository variables, while the breakdown for the current PR branch uses values set with -var for certain variables that affect cost.

          name: Terraform Plan and Cost Estimation

          on:
            workflow_dispatch:
            pull_request:
              types: [opened, synchronize, reopened]
              branches:
                - 'infra_main'

          permissions:
            pull-requests: write

          env:
            AWS_ACCESS_KEY: ${{ secrets.AWS_ACCESS_KEY }}
            AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
            TF_VAR_aws_region: ${{ vars.TF_AWS_REGION }}
            TF_VAR_ami_id: ${{ vars.TF_AMI_ID }}
            TF_VAR_instance_type: ${{ vars.TF_INSTANCE_TYPE }}
            TF_VAR_volume_size: ${{ vars.TF_VOLUME_SIZE }}
            TF_VAR_key_pair_name: ${{ vars.TF_KEY_PAIR_NAME }}
            TF_VAR_private_key: ${{ secrets.PRIVATE_KEY }}
            TF_VAR_domain_name: ${{ vars.TF_DOMAIN_NAME }}
            TF_VAR_frontend_domain: ${{ vars.TF_FRONTEND_DOMAIN }}
            TF_VAR_db_domain: ${{ vars.TF_DB_DOMAIN }}
            TF_VAR_traefik_domain: ${{ vars.TF_TRAEFIK_DOMAIN }}
            TF_VAR_cert_email: ${{ secrets.TF_CERT_EMAIL }}
            TF_VAR_private_key_path: ./${{ vars.TF_KEY_PAIR_NAME }}.pem


          jobs:
            terraform-plan:
              name: Terraform Plan
              runs-on: ubuntu-latest

              steps:
                - name: Checkout PR Branch
                  uses: actions/checkout@v2

                - name: Write Private Key to File
                  run: |
                    echo "${{ secrets.PRIVATE_KEY }}" > ${{ vars.TF_KEY_PAIR_NAME }}.pem
                    chmod 600 ${{ vars.TF_KEY_PAIR_NAME }}.pem

                - name: Setup Terraform
                  uses: hashicorp/setup-terraform@v2

                - name: Terraform Init
                  id: init
                  run: terraform init
                  working-directory: ./terraform

                - name: Terraform Plan
                  id: plan
                  run: | 
                    terraform plan -out=tfplan.out \
                    -var="ami_id=ami-005fc0f236362e99f" \
                    -var="instance_type=t2.large" \
                    -var="volume_size=20"
                  working-directory: ./terraform

                - name: Save Plan JSON
                  id: save-plan
                  run: terraform show -no-color tfplan.out > /tmp/tfplan.txt
                  working-directory: ./terraform

                - name: Setup Infracost
                  uses: infracost/actions/setup@v3
                  with:
                    api-key: ${{ secrets.INFRACOST_API_KEY }}

                # Checkout the branch you want Infracost to compare costs against, most commonly the target branch.
                - name: Checkout base branch
                  uses: actions/checkout@v4
                  with:
                    ref: '${{ github.event.pull_request.base.ref }}'

                - name: Run Infracost
                  run: |
                    infracost breakdown --path=./terraform --format=json --out-file=/tmp/infracost-base.json 

                # Checkout the current PR branch so we can create a diff.
                - name: Checkout PR branch
                  uses: actions/checkout@v4
                - name: Generate Infracost diff
                  run: |
                    infracost breakdown --path=./terraform --format=table --out-file=/tmp/infracost-new.txt \
                                        --terraform-var "ami_id=ami-005fc0f236362e99f" \
                                        --terraform-var "instance_type=t2.large" \
                                        --terraform-var "volume_size=20"         
                    infracost diff  --path=./terraform \
                                    --format=json \
                                    --compare-to=/tmp/infracost-base.json \
                                    --out-file=/tmp/infracost.json \
                                    --terraform-var "ami_id=ami-005fc0f236362e99f" \
                                    --terraform-var "instance_type=t2.large" \
                                    --terraform-var "volume_size=20"

                - name: Post Infracost Comment
                  run: |
                    infracost comment github --path=/tmp/infracost.json \
                                             --repo=$GITHUB_REPOSITORY \
                                             --github-token=${{ github.token }} \
                                             --pull-request=${{ github.event.pull_request.number }} \
                                             --behavior=update
                - name: Update PR Comment
                  uses: actions/github-script@v6
                  if: github.event_name == 'pull_request'
                  env:
                    PLAN: ${{ steps.plan.outcome }}
                  with:
                    github-token: ${{ secrets.GITHUB_TOKEN }}
                    script: |
                      const fs = require('fs');
                      const plan = fs.readFileSync('/tmp/tfplan.txt', 'utf8');
                      const infracost = fs.readFileSync('/tmp/infracost-new.txt', 'utf8');

                      const output = `#### Terraform Plan 📖\`${{ steps.plan.outcome }}\`

                      <details><summary>Show Plan</summary>

                      \`\`\`hcl
                      ${plan}
                      \`\`\`

                      </details>

                      #### New Infracost Breakdown 💰
                      <details><summary>Show Breakdown</summary>

                      \`\`\`sh
                      ${infracost}
                      \`\`\`

                      </details>

                      *Pushed by: @${{ github.actor }}, Action: \`${{ github.event_name }}\`*`;

                      github.rest.issues.createComment({
                        issue_number: context.issue.number,
                        owner: context.repo.owner,
                        repo: context.repo.repo,
                        body: output
                      })

terraform-apply.yml: This workflow automates the setup of resources.
1. Triggers: It is triggered by a push to infra_main or manually via the dispatch option.
2. Terraform: Sets up networking and server resources (VPC, EC2, Domain A records).
3. Artifacts: The public IP and dynamically generated Ansible inventory file are uploaded as reusable artifacts for use in specific workflows.
4. Post-Trigger: Activates the "Ansible Monitoring" workflow.

Note: The token used for this workflow is a PAT token generated from GitHub, not the default GITHUB_TOKEN.

    name: Terraform Apply

    on:
      workflow_dispatch:
        inputs:
          operation:
            description: 'Choose the Terraform operation'
            required: true
            default: 'apply'
            type: choice
            options:
              - apply
              - destroy
      push:
        branches:
          - 'infra_main'

    env:
      AWS_ACCESS_KEY: ${{ secrets.AWS_ACCESS_KEY }}
      AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
      TF_VAR_aws_region: ${{ vars.TF_AWS_REGION }}
      TF_VAR_ami_id: ${{ vars.TF_AMI_ID }}
      TF_VAR_instance_type: ${{ vars.TF_INSTANCE_TYPE }}
      TF_VAR_volume_size: ${{ vars.TF_VOLUME_SIZE }}
      TF_VAR_key_pair_name: ${{ vars.TF_KEY_PAIR_NAME }}
      TF_VAR_private_key: ${{ secrets.PRIVATE_KEY }}
      TF_VAR_domain_name: ${{ vars.TF_DOMAIN_NAME }}
      TF_VAR_frontend_domain: ${{ vars.TF_FRONTEND_DOMAIN }}
      TF_VAR_db_domain: ${{ vars.TF_DB_DOMAIN }}
      TF_VAR_traefik_domain: ${{ vars.TF_TRAEFIK_DOMAIN }}
      TF_VAR_cert_email: ${{ secrets.TF_CERT_EMAIL }}
      TF_VAR_private_key_path: ${{ vars.TF_KEY_PAIR_NAME }}.pem
      POSTGRES_PASSWORD: ${{ secrets.POSTGRES_PASSWORD }}

    jobs:
      build-infra:
        name: terraform-ci-cd
        runs-on: ubuntu-latest
        steps:
          - name: Checkout
            uses: actions/checkout@v2

          - name: Write Private Key to File
            run: |
              echo "${{ secrets.PRIVATE_KEY }}" > ${{ vars.TF_KEY_PAIR_NAME }}.pem
              chmod 600 ${{ vars.TF_KEY_PAIR_NAME }}.pem
            working-directory: ./terraform

          - name: Set up Terraform
            uses: hashicorp/setup-terraform@v2

          - name: Terraform Init
            id: init
            run: terraform init
            working-directory: ./terraform

          - name: Terraform Operation
            id: terraform-operation
            run: |
              if [ "${{ github.event.inputs.operation }}" = "destroy" ]; then
                terraform destroy --auto-approve
              else
                terraform apply --auto-approve
              fi
            working-directory: ./terraform

          - name: Upload Ansible Inventory
            if: steps.terraform-operation.outcome == 'success' && github.event.inputs.operation != 'destroy'
            uses: actions/upload-artifact@v4
            with:
              name: ansible_inventory
              path: ./terraform/inventory.ini

          - name: Save Public IP
            if: steps.terraform-operation.outcome == 'success' && github.event.inputs.operation != 'destroy' 
            run: |
              PUBLIC_IP=$(terraform output instance_public_ip | sed 's/"//g')
              echo "$PUBLIC_IP" > public_ip_env.txt
              cat public_ip_env.txt
            working-directory: ./terraform

          - name: Upload Public_IP
            if: steps.terraform-operation.outcome == 'success' && github.event.inputs.operation != 'destroy'
            uses: actions/upload-artifact@v4
            with:
              name: Public_IP
              path: ./terraform/public_ip_env.txt

          - name: Invoke workflow without inputs
            if: steps.terraform-operation.outcome == 'success' && github.event.inputs.operation != 'destroy'
            uses: benc-uk/workflow-dispatch@v1
            with:
              workflow: Ansible Monitoring
              token: "${{ secrets.TOKEN }}"

ansible-monitoring.yml: Deploys the monitoring stack (Prometheus, Grafana, Loki, Promtail, cAdvisor) using Ansible.
1. This is triggered by the terraform-apply.yml workflow.
2. This workflow retrieves the latest terraform-apply.yml run ID to download the Ansible inventory artifact.
3. It runs the Ansible playbook to set up the server and deploy the monitoring stack, which is available on the respective custom domains right after deployment.

    name: Ansible Monitoring

    on:
      workflow_dispatch:

    jobs:
      monitoring-stack-deploy:
        runs-on: ubuntu-latest
        steps:
          - name: Checkout
            uses: actions/checkout@v3

          - name: Get Workflow Run ID
            id: get-run-id
            run: |
              RUN_ID=$(curl -s \
                -H "Authorization: Bearer ${{ secrets.TOKEN }}" \
                -H "Accept: application/vnd.github+json" \
                "https://api.github.com/repos/${{ github.repository }}/actions/workflows/terraform-apply.yml/runs?branch=infra_main&per_page=1" \
                | jq -r '.workflow_runs[0].id')
              echo "run_id=$RUN_ID" >> $GITHUB_OUTPUT
              echo "$RUN_ID"

          - name: Write Private Key to File
            run: |
              echo "${{ secrets.PRIVATE_KEY }}" > ${{ vars.TF_KEY_PAIR_NAME }}.pem
              chmod 600 ${{ vars.TF_KEY_PAIR_NAME }}.pem

          - name: Download Ansible Inventory
            uses: actions/download-artifact@v4
            with:
              name: ansible_inventory
              github-token: ${{ secrets.TOKEN }} 
              run-id: ${{ steps.get-run-id.outputs.run_id }}

          - name: Verify Ansible Inventory
            run: |
              cat inventory.ini

          - name: "Install Ansible"
            uses: alex-oleshkevich/setup-ansible@v1.0.1
            with:
              version: "9.3.0"

          - name: Run Ansible Playbook
            run: |
              ANSIBLE_HOST_KEY_CHECKING=False ansible-playbook -i inventory.ini ./ansible/playbook.yml \
                  --extra-vars "frontend_domain=${{ vars.TF_FRONTEND_DOMAIN }} \
                    traefik_domain=${{ vars.TF_TRAEFIK_DOMAIN }} \
                    cert_email=${{ secrets.TF_CERT_EMAIL }}"

Application Pipelines

ci-frontend.yml: This process builds, tests, and pushes the frontend Docker image to Docker Hub, then updates compose.yml.
1. It tags the image with latest and a commit-based tag, which includes the workflow run number and a short commit-sha (e.g., maestrops/frontend:4-0a3087a). This helps track changes and ensures each image is uniquely identified and linked to the specific commit and workflow run.
2. It updates the compose.yml file with the new Docker image tag, commits the changes, and pushes them to the branch.

        name: Frontend CI Pipeline

        on:
          workflow_dispatch:
          push:
            branches:
              - 'integration'
            paths:
              - 'frontend/**'
              - '!frontend/*.md'

        jobs:
          build-and-test:
            name: Build & Test
            runs-on: ubuntu-latest
            steps:
              - name: Checkout code
                uses: actions/checkout@v4

              - name: Set up Node.js
                uses: actions/setup-node@v3
                with:
                  node-version: '18' 

              - name: Cache dependencies
                uses: actions/cache@v3
                with:
                  path: |
                    **/node_modules
                  key: ${{ runner.os }}-${{ hashFiles('**/package-lock.json') }}
                  restore-keys: |
                    ${{ runner.os }}-

              - name: Install dependencies
                run: |
                  cd frontend
                  npm install

              - name: Lint code
                run: |
                  cd frontend
                  npm run lint | tee /dev/null

              - name: Build
                run: |
                  cd frontend
                  npm run build

          build-push-image:
            name: Build & Push Image
            runs-on: ubuntu-latest
            needs: build-and-test
            env:
              GIT_USER_NAME: DrInTech
              GIT_USER_EMAIL: ${{ secrets.TF_CERT_EMAIL }}
            steps:
              - name: Checkout code
                uses: actions/checkout@v4

              - name: Set tags as environment variables
                run: |
                  COMMIT_SHA=$(git rev-parse --short HEAD)
                  echo "IMAGE_NAME=${{ secrets.DOCKERHUB_USERNAME }}/frontend" >> $GITHUB_ENV
                  echo "IMAGE_TAG_1=latest" >> $GITHUB_ENV
                  echo "IMAGE_TAG_2=${{ github.run_number }}-$COMMIT_SHA" >> $GITHUB_ENV

              - name: Set up Docker Buildx
                uses: docker/setup-buildx-action@v2

              - name: Login to Docker Hub
                uses: docker/login-action@v2
                with:
                  username: ${{ secrets.DOCKERHUB_USERNAME }}
                  password: ${{ secrets.DOCKERHUB_TOKEN }}  

              - name: Build and push
                uses: docker/build-push-action@v4
                with:
                  context: "{{defaultContext}}:frontend"
                  push: true
                  tags: ${{ env.IMAGE_NAME }}:${{ env.IMAGE_TAG_1 }}, ${{ env.IMAGE_NAME }}:${{ env.IMAGE_TAG_2 }}

          update-deployment:
            name: Update Deployment File
            runs-on: ubuntu-latest
            needs: build-push-image
            steps:
              - name: Checkout Repository
                uses: actions/checkout@v4

              - name: Set environment variables
                run: |
                  COMMIT_SHA=$(git rev-parse --short HEAD)
                  echo "IMAGE_NAME=${{ secrets.DOCKERHUB_USERNAME }}/frontend" >> $GITHUB_ENV
                  echo "IMAGE_TAG_2=${{ github.run_number }}-$COMMIT_SHA" >> $GITHUB_ENV

              - name: Update Deployment YAML
                run: |
                  sed -i "s|image: ${{ env.IMAGE_NAME }}:.*|image: ${{ env.IMAGE_NAME }}:${{ env.IMAGE_TAG_2 }}|" compose.yml

              - name: Commit and Push Changes
                run: |
                  BRANCH_NAME=${{ github.ref_name }}
                  git config --global user.email "${{ env.GIT_USER_EMAIL }}"
                  git config --global user.name "${{ env.GIT_USER_NAME }}"
                  git add compose.yml
                  git commit -m "Update Docker Image Tag to ${{ env.IMAGE_TAG_2 }}"
                  git pull --rebase origin $BRANCH_NAME
                  git push origin $BRANCH_NAME
                env:
                  GITHUB_TOKEN: ${{ secrets.TOKEN }}

cd-frontend.yml: Deploys the frontend container in the cloud. This GitHub Actions pipeline automates the deployment of the frontend application.
1. Triggered by a push to the deployment branch or manually via workflow_dispatch.
2. Retrieves the most recent Terraform apply run ID from the GitHub API to retrieve and download the Public_IP artifact from the terraform-apply.yml workflow.
3. Downloads the Public_IP artifact from a previous workflow.
4. Copy the necessary files and deploys the frontend application.

    name: Frontend CD Pipeline

    on:
      push:
        branches:
          - deployment
        paths:
          - 'frontend/**'
          - '!frontend/*.md'
      workflow_dispatch:

    jobs:
      deploy:
        runs-on: ubuntu-latest
        steps:
          - name: Checkout code
            uses: actions/checkout@v2

          - name: Prepare Frontend Env
            run: |
              # Frontend env file
              echo "VITE_API_URL=https://test.drintech.online" > frontend.env

          - name: Get terraform-apply.yml Run ID
            id: get-run-id
            run: |
              RUN_ID=$(curl -s \
                -H "Authorization: Bearer ${{ secrets.TOKEN }}" \
                -H "Accept: application/vnd.github+json" \
                "https://api.github.com/repos/${{ github.repository }}/actions/workflows/terraform-apply.yml/runs?branch=infra_main&per_page=1" \
                | jq -r '.workflow_runs[0].id')
              echo "run_id=$RUN_ID" >> $GITHUB_OUTPUT
              echo "$RUN_ID"

          - name: Download Public_IP File
            uses: actions/download-artifact@v4
            with:
              name: Public_IP
              github-token: ${{ secrets.TOKEN }} 
              run-id: ${{ steps.get-run-id.outputs.run_id }}

          - name: Read public IP
            id: read_ip
            run: |
              PUBLIC_IP=$(grep -oP '\b(?:[0-9]{1,3}\.){3}[0-9]{1,3}\b' public_ip_env.txt | head -n 1)
              echo "PUBLIC_IP=$PUBLIC_IP" >> $GITHUB_ENV

          - name: Copy files to Server
            uses: appleboy/scp-action@master
            with:
              host: ${{ env.PUBLIC_IP }}
              username: ${{ vars.EC2_USER }}
              key: ${{ secrets.PRIVATE_KEY }}
              source: "frontend.env, compose.yml"
              target: "~/"
              overwrite: true

          - name: Use SSH Action
            uses: appleboy/ssh-action@v1.2.0
            with:
              host: ${{ env.PUBLIC_IP }}
              username: ${{ vars.EC2_USER }}
              key: ${{ secrets.PRIVATE_KEY }}
              script: |
                mv frontend.env frontend/.env
                touch backend/.env 
                # docker compose down frontend
                docker compose up -d --no-deps --force-recreate frontend
                rm frontend/.env backend/.env

ci-backend.yml: Builds, tests, and pushes the backend Docker image to Docker Hub, then updates compose.yml.
1. Tags the image with latest and a commit-based tag, which includes the workflow run number and short commit SHA (e.g., maestrops/backend:4-0a3087a). This helps track changes and ensures each image is uniquely identified and linked to the specific commit and workflow run.
2. Updates the compose.yml file with the new Docker image tag, then commits and pushes the changes to the branch.

    name: Backend CI Pipeline

    on:
      workflow_dispatch:
      push:
        branches:
          - 'integration'
        paths:
          - 'backend/**'
          - '!backend/*.md'

    jobs:
      build-and-test:
        name: Build & Test
        runs-on: ubuntu-latest
        services:
          db:
            image: postgres:13
            env:
              POSTGRES_USER: app
              POSTGRES_PASSWORD: changethis123
              POSTGRES_DB: app
              POSTGRES_HOST: localhost
            ports:
              - 5432:5432
            options: >-
              --health-cmd pg_isready
              --health-interval 10s
              --health-timeout 5s
              --health-retries 5

        steps:
          - name: Checkout code
            uses: actions/checkout@v4

          - name: Set up Python
            uses: actions/setup-python@v4
            with:
              python-version: "3.10"

          - name: Cache dependencies
            uses: actions/cache@v3
            with:
              path: ~/.cache/pip
              key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt') }}
              restore-keys: |
                ${{ runner.os }}-pip-

          - name: Install dependencies
            run: |
              cd backend
              curl -sSL https://install.python-poetry.org | python3 -
              poetry install

          - name: Copy env file
            run: |
              cd backend
              cp .env.sample .env

          - name: Run app
            run: |
              cd backend
              poetry run bash ./prestart.sh
              poetry run uvicorn app.main:app --host 0.0.0.0 --port 8000 &

          - name: Run tests
            run: |
              cd backend
              poetry run pytest | tee /dev/null


      build-push-image:
        name: Build & Push Image
        runs-on: ubuntu-latest
        needs: build-and-test
        steps:
          - name: Checkout code
            uses: actions/checkout@v2

          - name: Set environment variables
            run: |
              COMMIT_SHA=$(git rev-parse --short HEAD)
              echo "IMAGE_NAME=${{ secrets.DOCKERHUB_USERNAME }}/backend" >> $GITHUB_ENV
              echo "IMAGE_TAG_1=latest" >> $GITHUB_ENV
              echo "IMAGE_TAG_2=${{ github.run_number }}-$COMMIT_SHA" >> $GITHUB_ENV

          - name: Set up Docker Buildx
            uses: docker/setup-buildx-action@v2

          - name: Login to Docker Hub
            uses: docker/login-action@v2
            with:
              username: ${{ secrets.DOCKERHUB_USERNAME }}
              password: ${{ secrets.DOCKERHUB_TOKEN }}

          - name: Build and push
            uses: docker/build-push-action@v4
            with:
              context: "{{defaultContext}}:backend"
              push: true
              tags: ${{ env.IMAGE_NAME }}:${{ env.IMAGE_TAG_1 }}, ${{ env.IMAGE_NAME }}:${{ env.IMAGE_TAG_2 }}


      update-deployment:
        name: Update Deployment File
        runs-on: ubuntu-latest
        needs: build-push-image
        env:
          GIT_USER_NAME: DrInTech
          GIT_USER_EMAIL: ${{ secrets.TF_CERT_EMAIL }}
        steps:
          - name: Checkout Repository
            uses: actions/checkout@v4

          - name: Set environment variables
            run: |
              COMMIT_SHA=$(git rev-parse --short HEAD)
              echo "IMAGE_NAME=${{ secrets.DOCKERHUB_USERNAME }}/backend" >> $GITHUB_ENV
              echo "IMAGE_TAG_2=${{ github.run_number }}-$COMMIT_SHA" >> $GITHUB_ENV

          - name: Update Deployment YAML
            run: |
              sed -i "s|image: ${{ env.IMAGE_NAME }}:.*|image: ${{ env.IMAGE_NAME }}:${{ env.IMAGE_TAG_2 }}|" compose.yml

          - name: Commit and Push Changes
            run: |
              BRANCH_NAME=${{ github.ref_name }}
              git config --global user.email "${{ env.GIT_USER_EMAIL }}"
              git config --global user.name "${{ env.GIT_USER_NAME }}"
              git add compose.yml
              git commit -m "Update Docker Image Tag to ${{ env.IMAGE_TAG_2 }}"
              git pull --rebase origin $BRANCH_NAME
              git push origin $BRANCH_NAME
            env:
              GITHUB_TOKEN: ${{ secrets.TOKEN }}

cd-backend.yml:Deploys the backend, PostgreSQL, and Adminer containers.
- Sets up environment files for the backend and PostgreSQL, including sensitive secrets.
- Fetches the latest Terraform run ID and downloads the public IP.
- Copies configuration files (backend.env, postgres.env, compose.yml) to the server using SCP.
- Sets up env files and deploys the backend services (backend, db, adminer).

    name: Backend CD Pipeline

    on:
      push:
        branches:
          - deployment
        paths:
          - 'backend/**'
          - '!backend/*.md'
      workflow_dispatch:

    jobs:
      deploy:
        runs-on: ubuntu-latest
        steps:
          - name: Checkout code
            uses: actions/checkout@v3

          - name: Set backend env file
            run: |
              # Insensitive Variables 
              echo "DOMAIN=localhost" >> backend.env
              echo "ENVIRONMENT=local" >> backend.env
              echo "PROJECT_NAME=\"Full Stack FastAPI Project\"" >> backend.env
              echo "STACK_NAME=full-stack-fastapi-project" >> backend.env
              echo "BACKEND_CORS_ORIGINS=\"http://localhost,http://localhost:5173,https://localhost,https://localhost:5173\"" >> backend.env
              echo "SMTP_TLS=true" >> backend.env
              echo "SMTP_SSL=false" >> backend.env
              echo "SMTP_PORT=587" >> backend.env
              echo "EMAILS_FROM_EMAIL=info@example.com" >> backend.env
              echo "POSTGRES_SERVER=db" >> backend.env
              echo "POSTGRES_USER=app" >> backend.env
              echo "POSTGRES_PORT=5432" >> backend.env
              echo "POSTGRES_DB=app" >> backend.env
              echo "USERS_OPEN_REGISTRATION=True" >> backend.env

              # Sensitive Variables
              echo "SECRET_KEY=${{ secrets.SECRET_KEY }}" >> backend.env
              echo "FIRST_SUPERUSER=${{ secrets.FIRST_SUPERUSER }}" >> backend.env
              echo "FIRST_SUPERUSER_PASSWORD=${{ secrets.FIRST_SUPERUSER_PASSWORD }}" >> backend.env
              echo "POSTGRES_PASSWORD=${{ secrets.POSTGRES_PASSWORD }}" >> backend.env

          - name: Prepare Frontend and PostgreSQL Env
            run: |
              # PostgreSQL env file
              echo "POSTGRES_USER=app" >> postgres.env
              echo "POSTGRES_DB=app" >> postgres.env
              echo "POSTGRES_PASSWORD=${{ secrets.POSTGRES_PASSWORD }}" >> postgres.env


          - name: Get terraform-apply.yml Run ID
            id: get-run-id
            run: |
              RUN_ID=$(curl -s \
                -H "Authorization: Bearer ${{ secrets.TOKEN }}" \
                -H "Accept: application/vnd.github+json" \
                "https://api.github.com/repos/${{ github.repository }}/actions/workflows/terraform-apply.yml/runs?branch=infra_main&per_page=1" \
                | jq -r '.workflow_runs[0].id')
              echo "run_id=$RUN_ID" >> $GITHUB_OUTPUT
              echo "$RUN_ID"

          - name: Download Public_IP File
            uses: actions/download-artifact@v4
            with:
              name: Public_IP
              github-token: ${{ secrets.TOKEN }} 
              run-id: ${{ steps.get-run-id.outputs.run_id }}

          - name: Read public IP
            id: read_ip
            run: |
              PUBLIC_IP=$(grep -oP '\b(?:[0-9]{1,3}\.){3}[0-9]{1,3}\b' public_ip_env.txt | head -n 1)
              echo "PUBLIC_IP=$PUBLIC_IP" >> $GITHUB_ENV

          - name: Copy files to Server
            uses: appleboy/scp-action@v0.1.7
            with:
              host: ${{ env.PUBLIC_IP }}
              username: ${{ vars.EC2_USER }}
              key: ${{ secrets.PRIVATE_KEY }}
              source: "backend.env, postgres.env, compose.yml"
              target: "~/"
              overwrite: true

          - name: Use SSH Action
            uses: appleboy/ssh-action@v1.2.0
            with:
              host: ${{ env.PUBLIC_IP }}
              username: ${{ vars.EC2_USER }}
              key: ${{ secrets.PRIVATE_KEY }}
              script: |
                mv backend.env backend/.env
                mv postgres.env .env && touch frontend/.env
                # docker compose down backend db adminer
                docker compose up -d --no-deps --force-recreate backend db adminer
                rm frontend/.env backend/.env .env

The workflow checks if any changes have been made to the backend folder, which holds the source code for the backend application, before starting. If no changes are found, or if only the README file in the backend folder is modified, the pipeline will not start.

How it works

GitOps uses a smart branching strategy and pull requests to allow code reviews, status checks, and discussions. This approach makes sure that any changes to the infrastructure are carefully reviewed before being deployed to the live environment.

For example, when a developer pushes a change to the integration (development) branch, the CI system builds, tests, and creates a Docker image of the application. It updates the deployment file with the application version. The Docker image is tagged with the workflow run number and a short commit SHA for traceability. A pull request is then made from the integration branch to the deployment (main) branch. Code reviews, suggestions, and successful status checks are required before the merge is allowed.

Once the pull request is merged into the deployment branch, it triggers the CD pipeline to deploy the new application version to the cloud infrastructure. This process is also applied to the infrastructure workflow.

Infrastructure CI/CD

Branching Strategy

infra_features: This is the development branch used for writing and testing Terraform configurations.
infra_main: This is the production branch used for provisioning infrastructure.

Pipeline Workflow

Push to infra_features:
- Triggers terraform-validate.yml to check configuration correctness.
Pull Request to infra_main:
- Runs terraform-plan.yml.
- Provides an infrastructure plan and cost breakdown using InfraCost.
- Runs status checks and requires plan, cost, and code reviews.
Merge to infra_main:
- Triggers terraform-apply.yml to provision resources.
- On success, triggers ansible-monitoring.yml to set up the monitoring stack.

Application CI/CD

Branching Strategy

integration: Used for continuous integration and building Docker images.
deployment: Used for continuous deployment to the cloud environment.

Pipeline Workflow

Push to integration:
- ci-frontend.yml: Builds, tests, and pushes the frontend image, then updates compose.yml.
- ci-backend.yml: Builds, tests, and pushes the backend image, then updates compose.yml.
Pull request from integration to deployment: This runs status checks and requires code reviews.
Merge from integration to deployment:
- cd-frontend.yml: Deploys the updated frontend container.
- cd-backend.yml: Deploys the backend, PostgreSQL, and Adminer containers.

Setting Repository Variables and Secrets

We have declared many secrets and variables in our workflows. Some non-sensitive variables are managed within the workflow since this project operates in a single environment. Below are the repository secrets and variables you need to set.

GitOps Workflow Security

Finally, we’ll set the status checks and branch protection rules to complete our GitOps implementation. These protections ensure that no one can push changes directly to the live infrastructure except through an approved pull request, which requires successful status checks and code reviews.

To set up branch protection rules, follow these steps:

Go to the settings tab.
Click on the branches option in the left sidebar, then click 'add rule'.
Enter the name of the branch, choose the options provided below, and click 'save'.
1. Repeat these steps for the infra_main branch.

How to get started

Clone the repository

 git clone https://github.com/DrInTech22/full-stack-gitops.git

Set the necessary repository secrets and variables as demonstrated here.
Make a change to the infra_features branch workflow configuration (perhaps add a comment in one of the files), commit it, and push. Observe the terraform-validate.yml workflow in the actions tab.
1. Create a pull request from the infra_features branch to the infra_main branch. Make sure the repository variable TF_INSTANCE_TYPE is set to t3.medium and TF_VOLUME_SIZE is 15. This is the instance we plan to launch in our infrastructure. Then, the configuration -var="instance_type=t2.large" -var="volume_size=20" in the terraform-plan.yml will be used to compare the cost differences between our live infrastructure and the incoming cost plan in the PR branch (infra_features).
  - Infracost provides insights into cost savings and optimizations

The checks have passed, and a review is required, but since I'm the only one working on this project, I'll bypass the review and proceed to merge. However, in an active environment, ensure you remove the bypass rules for administrators.

This merge will trigger the Terraform Apply workflow. Upon successful completion, the Terraform Apply workflow will produce two artifacts (Public_IP and inventory files) and trigger the Ansible Monitoring workflow.

The Ansible Monitoring workflow sets up the server and deploys the monitoring stack, which are immediately accessible on our custom domains.

Make changes to the integration branch, particularly in the backend folder and push. This will trigger the backend CI.

Check the compose file, and you'll see that the tag of the backend image has been automatically updated to include this workflow run number and the commit that initiated it.
1. Create a pull request from integration to deployment branch. We’ll bypass the branch protection as we don’t have a reviewer and merge the pull request. This will trigger the backend CD workflow.
2. Our backend, posgresql and adminer containers are up and are accessible on the custom domains.
  1. Next, make changes to the frontend folder on the integration branch, then commit and push them. Notice that the image tag matches the workflow number and commit SHA.
    1. Next, merge the pull request which triggers the frontend CD pipeline. After successful deployment, your frontend application is immediately accessible on its custom domain.

Note: The dynamic accessibility of the applications on the custom domain after deployment is due to the powerful static configurations of Traefik, the reverse proxy manager.

Conclusion

We have learnt how operations can quickly become disorganized without GitOps and how GitOps streamlines infrastructure provisioning and application deployment. The project demonstrated how to use effective branching strategies, robust CI/CD workflows, and cost estimation tools to ensure a seamless operational workflow in your infrastructure.

Cost Optimization and Automation: A Full Stack GitOps Workflow

Table of contents

Introduction

Prerequisites

Objectives

Technologies

Infrastructure Tools

Containerization and Deployment

Monitoring and Logging Stack

CI/CD

Project Architecture

Project Structure

Pipeline Configuration Files

Infrastructure Pipelines

Application Pipelines

How it works

Infrastructure CI/CD

Branching Strategy

Pipeline Workflow

Application CI/CD

Branching Strategy

Pipeline Workflow

Setting Repository Variables and Secrets

GitOps Workflow Security

How to get started

Conclusion