Skip to main content
HashnodeDrInTech

Command Palette

Search for a command to run...

How to Automate Full Stack Deployments with Terraform and Ansible

Updated
•14 min read
How to Automate Full Stack Deployments with Terraform and Ansible
O
Okesanya Samuel

Welcome to my blog! My name is Okesanya Odunayo, and I'm a passionate cloud practitioner, DevOps enthusiast, and health advocate. I believe that sharing knowledge and insights is essential for driving innovation and advancing the industry as a whole. So if you're looking to learn more about the exciting world of cloud computing and DevOps, you've come to the right place.

Introduction

In the previous project, we explored how to containerize and monitor a full-stack application using advanced tools. While that method offered great flexibility and scalability, managing deployments manually can be time-consuming and error-prone, especially in dynamic environments.

In this article, we're going to kick things up a notch by diving into Terraform and Ansible. We'll automate everything from our previous project, like setting up the server, managing DNS records, configuring the server, and deploying the applications. Plus, we'll dynamically set up Grafana dashboards, and use Traefik for direct reverse proxy and SSL certificate generation. It's all about full automation. Let's jump in!

Objectives

  • Provision Infrastructure: Use Terraform to set up the infrastructure.

  • Automate Application Deployment: Use Ansible to set up the environment and deploy containerized services.

  • Set Up Monitoring: Deploy Prometheus and Grafana to monitor application health and metrics with dynamic dashboards.

  • Automate Routing: Set up routing between services with Traefik for secure and seamless service communication.

Architecture

Project Structure

full-stack-automation/
├── ansible/
│   ├── roles/
│   ├── compose.monitoring.yml.j2
│   ├── compose.yml.j2
│   └── playbook.yml
├── backend/
│   ├── .env (should exist only on server or local machine)
│   └── .env.sample
├── frontend/
│   ├── .env (should exist only on server or local machine)
│   └── .env.sample
├── monitoring/
│   ├── dashboards/
│   ├── dashboard-providers.yml
│   ├── loki-config.yml
│   ├── loki-datasource.yml
│   ├── prometheus-datasource.yml
│   ├── prometheus.yml
│   └── promtail-config.yml
├── terraform/
│   ├── ansible.tf
│   ├── backend.tf
│   ├── ec2.tf
│   ├── dns.tf
│   ├── main.tf
│   ├── output.tf
│   ├── terraform.tfvars (should exist only on server or local machine)
│   ├── variables.tf
│   └── vpc.tf
├── .gitignore
└── POSTGRES_PASSWORD.txt (should exist only on server or local machine)

Find the link to the project repository here.

Deployed Services

  • Application Stack:

    • Frontend: React Application (containerized)

    • Backend: FastAPI Service (containerized)

    • Database: PostgreSQL (containerized)

    • Reverse Proxy: Traefik for routing

  • Monitoring Stack:

    • Prometheus: For metrics collection

    • Grafana: For data visualization and dashboard creation

    • cAdvisor: For container-level metrics

    • Loki: For log aggregation

    • Promtail: For log collection

Configurations

Let’s walk through the project configurations.

Terraform: Infrastructure provisioning

In this project, we’ll use Terraform as an Infrastructure as Code (IAC) tool. Terraform will automate the setup of the infrastructure. It will configure the VPC, EC2 server, DNS records, and trigger the Ansible playbook.

Config Files:

  • main.tf: defines and sets up the AWS provider.

      provider "aws" {
  region = var.aws_region
  }

  • backend.tf: This Terraform configuration sets up an S3 backend for state storage and locking to ensure consistent provisioning. It also specifies the required versions of Terraform and the AWS provider.

      terraform {
  backend "s3" {
      bucket         = "ansible-terraform01"
      region         = "us-east-1"
      key            = "ansible-terraform01/terraform.tfstate"
      dynamodb_table = "ansible-terraform01-lock"
      encrypt        = true
  }
  required_version = ">=0.13.0"
  required_providers {
      aws = {
      version = ">= 2.7.0"
      source  = "hashicorp/aws"
      }
  }
  }

  • vpc.tf: Creates AWS network infrastructure, including a VPC, subnet, internet gateway, route table, and security group.

      resource "aws_vpc" "vpc" {
  cidr_block = "10.0.0.0/16"
  enable_dns_support   = true
  enable_dns_hostnames = true
  tags = {
      Name = var.vpc-name
  }
  }

  resource "aws_subnet" "public_subnet" {
  vpc_id                  = aws_vpc.vpc.id
  cidr_block              = "10.0.1.0/24"
  map_public_ip_on_launch = true
  tags = {
      Name = var.subnet-name
  }
  }

  resource "aws_internet_gateway" "igw" {
  vpc_id = aws_vpc.vpc.id
  tags = {
      Name = var.igw-name
  }
  }

  resource "aws_route_table" "rt" {
  vpc_id = aws_vpc.vpc.id

  route {
      cidr_block = "0.0.0.0/0"
      gateway_id = aws_internet_gateway.igw.id
  }

  tags = {
      Name = var.rt-name
  }
  }

  resource "aws_route_table_association" "rt_association" {
  subnet_id      = aws_subnet.public_subnet.id
  route_table_id = aws_route_table.rt.id
  }

  resource "aws_security_group" "sg" {
  vpc_id = aws_vpc.vpc.id

  ingress = [
      for port in [22, 8080, 5173, 8000, 80, 443] : {
      description      = "TLS from VPC"
      from_port        = port
      to_port          = port
      protocol         = "tcp"
      ipv6_cidr_blocks = ["::/0"]
      self             = false
      prefix_list_ids  = []
      security_groups  = []
      cidr_blocks      = ["0.0.0.0/0"]
      }
  ]
  egress {
      from_port   = 0
      to_port     = 0
      protocol    = "-1"
      cidr_blocks = ["0.0.0.0/0"]
  }

  tags = {
      Name = var.sg-name
  }
  }

  • ec2.tf: Provisions an EC2 instance in the public subnet.

      resource "aws_instance" "ec2" {
  ami           = var.ami_id
  instance_type = var.instance_type
  key_name      = var.key_pair_name
  subnet_id     = aws_subnet.public_subnet.id
  vpc_security_group_ids = [aws_security_group.sg.id]
  tags = {
      Name = var.ec2_name
  }

  provisioner "local-exec" {
      command = "echo 'Instance provisioned: ${self.public_ip}'"
  }
  }

  • dns.tf: sets up DNS records for the domains.

      # Fetch the existing hosted zone
  data "aws_route53_zone" "domain_zone" {
    name         = var.domain_name
    private_zone = false
  }

  # Create DNS records for main and www subdomains
  resource "aws_route53_record" "frontend_record" {
    zone_id = data.aws_route53_zone.domain_zone.zone_id
    name    = var.frontend_domain
    type    = "A"
    ttl     = 300
    records = [aws_instance.ec2.public_ip]
  }

  resource "aws_route53_record" "www_frontend_record" {
    zone_id = data.aws_route53_zone.domain_zone.zone_id
    name    = "www.${var.frontend_domain}"
    type    = "A"
    ttl     = 300
    records = [aws_instance.ec2.public_ip]
  }

  resource "aws_route53_record" "db_record" {
    zone_id = data.aws_route53_zone.domain_zone.zone_id
    name    = var.db_domain
    type    = "A"
    ttl     = 300
    records = [aws_instance.ec2.public_ip]
  }

  resource "aws_route53_record" "www_db_record" {
    zone_id = data.aws_route53_zone.domain_zone.zone_id
    name    = "www.${var.db_domain}"
    type    = "A"
    ttl     = 300
    records = [aws_instance.ec2.public_ip]
  }

  resource "aws_route53_record" "traefik_record" {
    zone_id = data.aws_route53_zone.domain_zone.zone_id
    name    = var.traefik_domain
    type    = "A"
    ttl     = 300
    records = [aws_instance.ec2.public_ip]
  }

  resource "aws_route53_record" "www_traefik_record" {
    zone_id = data.aws_route53_zone.domain_zone.zone_id
    name    = "www.${var.traefik_domain}"
    type    = "A"
    ttl     = 300
    records = [aws_instance.ec2.public_ip]
  }

  • ansible.tf: Automatically generates a dynamic Ansible inventory file and triggers the Ansible playbook after DNS records have been created by terraform.

      # Create Ansible inventory file
  resource "local_file" "ansible_inventory" {
    filename = "../ansible/inventory.ini"
    content  = <<EOF
  [web_servers]
  ${aws_instance.ec2.public_ip} ansible_user=ubuntu ansible_ssh_private_key_file=${var.private_key_path}
  EOF
  }

  # Run the Ansible playbook
  resource "null_resource" "run_ansible" {
    depends_on = [
      aws_route53_record.frontend_record,
      aws_route53_record.www_frontend_record,
      aws_route53_record.db_record,
      aws_route53_record.www_db_record,
      aws_route53_record.traefik_record,
      aws_route53_record.www_traefik_record,
      local_file.ansible_inventory
    ]
    provisioner "local-exec" {
      command = "ANSIBLE_HOST_KEY_CHECKING=False ansible-playbook -i ../ansible/inventory.ini ../ansible/playbook.yml --extra-vars 'frontend_domain=${var.frontend_domain} db_domain=${var.db_domain} traefik_domain=${var.traefik_domain} cert_email=${var.cert_email}'"
    }
  }

    Optionally, you can use the ansible.tf.old file if you’re not managing your domain on Route53. It includes a custom script that ensures DNS propagation is done manually before triggering the Ansible playbook.
    Note: You can delegate your domain from your DNS provider to Route53 without transferring ownership.

  • outputs.tf: outputs the necessary information, such as the public IP of the EC2 instance, for access to the server.

      output "instance_public_ip" {
  description = "Public IP of the EC2 instance"
  value       = aws_instance.ec2.public_ip
  }

  • variables.tf: defines the structure of variables used in the Terraform configurations.

      variable "aws_region" {
  description = "AWS region to deploy resources"
  default     = "us-east-1"
  }

  variable "vpc-name" {
  description = "Name of the VPC"
  default     = "MainVPC"
  }

  variable "subnet-name" {
  description = "Name of the Subnet"
  default     = "MainSubnet"
  }

  variable "igw-name" {
  description = "Name of the Internet Gateway"
  default     = "MainIGW"
  }

  variable "rt-name" {
  description = "Name of the Route Table"
  default     = "MainRouteTable"
  }

  variable "sg-name" {
  description = "Name of the Security Group"
  default     = "MainSG"
  }

  variable "ami_id" {
  description = "AMI ID for the EC2 instance"
  type        = string
  }

  variable "instance_type" {
  description = "EC2 instance type"
  default     = "t2.micro"
  }

  variable "key_pair_name" {
  description = "Key pair name for SSH access"
  type        = string
  }

  variable "private_key_path" {
  description = "private key path for ansible ssh access"
  type        = string
  }

  variable "ec2_name" {
  description = "Name of the EC2 instance"
  default     = "MainEC2Instance"
  }

  variable "frontend_domain" {
  description = "Domain name for the frontend"
  type        = string
  }

  variable "db_domain" {
  description = "Domain name for the database admin (Adminer)"
  type        = string
  }

  variable "traefik_domain" {
  description = "Domain name for traefik Proxy Manager"
  type        = string
  }

  variable "cert_email" {
  description = "Email for the let's encrypt certificate"
  type        = string
  }
  variable "domain_name" {
    description = "the hosted zone domain name"
    type        = string
  }

  • terraform.tfvars: defines the values for the variables used in the Terraform configurations.

      aws_region     = "us-east-1"
  ami_id         = "ami-005fc0f236362e99f" # Replace with a valid AMI ID
  instance_type  = "t3.medium"
  key_pair_name  = "hello"
  private_key_path = "../../hello.pem" # relative path to terraform folder

  domain          = "drintech.online"
  frontend_domain = "cv1.drintech.online"
  db_domain       = "db.cv1.drintech.online"
  traefik_domain  = "traefik.cv1.drintech.online"
  cert_email      = "admin@example.com" # replace with a valid email

Ansible: Configuration management

Ansible installs the necessary software on the server, copies the required files, and automatically deploys the application and monitoring stack. The tasks are logically grouped into roles to promote modularity and maintainability. Below is the structure of the Ansible folder.

ansible/
├── roles/
├── copy_files/
│   └── tasks/
│       └── main.yml
├── docker_compose/
│   └── tasks/
│       └── main.yml
├── docker_setup/
│   └── tasks/
│       └── main.yml
├── file_structure/
│   └── tasks/
│       └── main.yml
├── loki_driver/
│   └── tasks/
│       └── main.yml
├── compose.monitoring.yml.j2
├── compose.yml.j2
├── inventory.ini
└── playbook.yml

  • playbook.yml: This is the main playbook that executes all tasks defined in the roles.

      ---
  - name: Setup Docker and Run Compose
    hosts: all
    become: true
    roles:
      - docker_setup
      - file_structure
      - copy_files
      - loki_driver
      - docker_compose

  • roles: contains the Ansible roles for each step of the deployment.

    • docker_setup: installs Docker and related plugins and sets up bridge networks for the application and monitoring stack.

        ---
  - name: Install prerequisites for Docker
    apt:
      name:
        - apt-transport-https
        - ca-certificates
        - curl
        - software-properties-common
      state: present

  - name: Add Docker GPG key
    shell: |
      curl -fsSL https://download.docker.com/linux/ubuntu/gpg | gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
    args:
      creates: /usr/share/keyrings/docker-archive-keyring.gpg

  - name: Add Docker repository
    shell: |
      echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" > /etc/apt/sources.list.d/docker.list

  - name: Install Docker and related plugins
    apt:
      name:
        - docker-ce
        - docker-ce-cli
        - containerd.io
        - docker-buildx-plugin
        - docker-compose-plugin
      state: present
      update_cache: yes

  - name: Ensure Docker service is running
    service:
      name: docker
      state: started
      enabled: true

  - name: Create app-network 
    community.docker.docker_network:
      name: app-network
      driver: bridge
    become: true

  - name: Create monitor-network
    community.docker.docker_network:
      name: monitor-network
      driver: bridge
    become: true

    • file_structure: creates the required folders and files.

        ---
  - name: Create required directories
    file:
      path: "{{ item }}"
      state: directory
      mode: '0755'
    loop:
      - /home/ubuntu/monitoring
      - /home/ubuntu/frontend
      - /home/ubuntu/backend
      - /home/ubuntu/traefik

  - name: Ensure acme.json file exists
    file:
      path: /home/ubuntu/traefik/acme.json
      state: touch
      owner: ubuntu
      group: ubuntu
      mode: '0600'

    • copy_files: copies the necessary files to the server.

        ---
  - name: Copy the main Docker Compose file
    template:
      src: compose.yml.j2
      dest: /home/ubuntu/compose.yml
      force: yes

  - name: Copy the monitoring Docker Compose file
    template:
      src: compose.monitoring.yml.j2
      dest: /home/ubuntu/compose.monitoring.yml
      force: yes

  - name: Copy frontend environment file
    template:
      src: ../frontend/.env
      dest: /home/ubuntu/frontend/.env

  - name: Copy backend environment file
    copy:
      src: ../backend/.env
      dest: /home/ubuntu/backend/.env

  - name: Copy monitoring configurations
    copy:
      src: ../monitoring/
      dest: /home/ubuntu/monitoring/
      force: yes

  - name: Copy postgres password file
    copy:
      src: ../POSTGRES_PASSWORD.txt
      dest: /home/ubuntu/POSTGRES_PASSWORD.txt
      mode: '0600'
      owner: ubuntu
      group: ubuntu
      force: yes

    • loki_driver: installs the Loki Docker driver plugin.

        ---
  - name: Install the Loki Docker driver plugin
    shell: |
      if ! docker plugin ls | grep -q "loki"; then
        docker plugin install grafana/loki-docker-driver:2.9.1 --alias loki --grant-all-permissions
      fi

  - name: Create /etc/docker/daemon.json with Loki configuration
    copy:
      dest: /etc/docker/daemon.json
      content: |
        {
          "log-driver": "loki",
          "log-opts": {
            "loki-url": "http://localhost:3100/loki/api/v1/push",
            "loki-batch-size": "400"
          }
        }
      force: yes

  - name: Restart Docker service
    shell: systemctl restart docker

    • docker_compose: deploys the application and monitoring stacks.

        ---
  - name: Run main Docker Compose
    shell: docker compose up -d
    args:
      chdir: /home/ubuntu

  • compose.monitoring.yml.j2: contains the templated configuration for the Docker Compose monitoring stack. Ansible dynamically injects the variable values during playbook execution.

services:
  prometheus:
    image: prom/prometheus
    expose:
      - "9090"
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--web.external-url=/prometheus'
    volumes:
      - ./monitoring/prometheus.yml:/etc/prometheus/prometheus.yml
    networks:
      - monitor-network
    depends_on:
      - cadvisor
    labels:
      - traefik.enable=true
      # HTTP Service
      - traefik.http.services.prometheus.loadbalancer.server.port=9090 # important
      # HTTP Router
      - traefik.http.routers.prometheus-http.rule=((Host(`{{ frontend_domain }}`) || Host(`www.{{ frontend_domain }}`)) && PathPrefix(`/prometheus`))
      - traefik.http.routers.prometheus-http.entrypoints=web
      # www to non-www redirect
      - traefik.http.routers.prometheus-https.middlewares=www-to-non-www
      # HTTPS Router
      - traefik.http.routers.prometheus-https.rule=((Host(`{{ frontend_domain }}`) || Host(`www.{{ frontend_domain }}`)) && PathPrefix(`/prometheus`))
      - traefik.http.routers.prometheus-https.entrypoints=websecure
      - traefik.http.routers.prometheus-https.tls.certresolver=letsencryptresolver
      - traefik.http.routers.prometheus-https.service=prometheus # optional

  grafana:
    image: grafana/grafana-oss
    expose:
      - "3000"
    labels:
      - traefik.enable=true
      # HTTP Service
      - traefik.http.services.grafana.loadbalancer.server.port=3000 # important
      # HTTP Router
      - traefik.http.routers.grafana-http.rule=((Host(`{{ frontend_domain }}`) || Host(`www.{{ frontend_domain }}`)) && PathPrefix(`/grafana`))
      - traefik.http.routers.grafana-http.entrypoints=web
      # HTTP to HTTPS redirect
      - traefik.http.routers.grafana-https.middlewares=www-to-non-www
      # HTTPS Router
      - traefik.http.routers.grafana-https.rule=((Host(`{{ frontend_domain }}`) || Host(`www.{{ frontend_domain }}`)) && PathPrefix(`/grafana`))
      - traefik.http.routers.grafana-https.entrypoints=websecure
      - traefik.http.routers.grafana-https.tls.certresolver=letsencryptresolver
      - traefik.http.routers.grafana-https.service=grafana # optional
    environment:
      - GF_SERVER_ROOT_URL=http://localhost:3000/grafana
      - GF_SERVER_SERVE_FROM_SUB_PATH=true
    volumes:
      - grafana:/var/lib/grafana
      - ./monitoring/loki-datasource.yml:/etc/grafana/provisioning/datasources/loki-datasource.yml
      - ./monitoring/prometheus-datasource.yml:/etc/grafana/provisioning/datasources/prometheus-datasource.yml
      - ./monitoring/dashboard-providers.yml:/etc/grafana/provisioning/dashboards/dashboard-providers.yml
      - ./monitoring/dashboards:/var/lib/grafana/dashboards # dashboard json files
    networks:
      - monitor-network
    depends_on:
      - loki

  cadvisor:
    image: gcr.io/cadvisor/cadvisor:v0.47.0
    expose:
      - "8081"
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:rw
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
    networks:
      - monitor-network
      - app-network
    depends_on:
      - backend
      - frontend
      - adminer
      - traefik
      - db

  loki:
    image: grafana/loki:latest
    ports:
      - 3100:3100
    networks:
      - monitor-network
    volumes:
      - ./monitoring/loki-config.yml:/etc/loki/loki-config.yaml

  promtail:
    image: grafana/promtail:latest
    networks:
      - monitor-network
    volumes:
      - ./monitoring/promtail-config.yml:/etc/promtail/promtail-config.yaml
    depends_on:
      - loki

volumes:
  grafana:

compose.yml.j2: contains the templated configuration for the Docker Compose application stack.

include:
  - compose.monitoring.yml
services:
  frontend:
    image: maestrops/frontend:latest
    env_file:
      - frontend/.env
    depends_on:
      - backend
    expose:
      - "5173"
    labels:
      - "traefik.enable=true"
      # HTTP Router
      - "traefik.http.routers.frontend-http.rule=(Host(`{{ frontend_domain }}`) || Host(`www.{{ frontend_domain }}`))"
      - "traefik.http.routers.frontend-http.entrypoints=web"
      - "traefik.http.services.frontend.loadbalancer.server.port=5173"
      # www to non-www redirect
      - "traefik.http.routers.frontend-https.middlewares=www-to-non-www"
      # HTTPS Router
      - "traefik.http.routers.frontend-https.rule=(Host(`{{ frontend_domain }}`) || Host(`www.{{ frontend_domain }}`))"
      - "traefik.http.routers.frontend-https.entrypoints=websecure"
      - "traefik.http.routers.frontend-https.tls.certresolver=letsencryptresolver"
      - "traefik.http.routers.frontend-https.service=frontend" # optional

    networks:
      - app-network

  backend:
    image: maestrops/backend:latest
    env_file:
      - backend/.env
    networks:
      - app-network
    expose:
      - "8000"
    labels:
      - traefik.enable=true
      # HTTP Router
      - "traefik.http.routers.backend-http.rule=((Host(`{{ frontend_domain }}`) || Host(`www.{{ frontend_domain }}`)) && (PathPrefix(`/api`) || PathPrefix(`/redoc`) || PathPrefix(`/docs`)))"
      - "traefik.http.routers.backend-http.entrypoints=web"
      - "traefik.http.services.backend-http.loadbalancer.server.port=8000"
      # www to non-www redirect
      - "traefik.http.routers.backend-https.middlewares=www-to-non-www"
      # HTTPS Router
      - "traefik.http.routers.backend-https.rule=(Host(`{{ frontend_domain }}`) && (PathPrefix(`/api`) || PathPrefix(`/redoc`) || PathPrefix(`/docs`)))"
      - "traefik.http.routers.backend-https.entrypoints=websecure"
      - "traefik.http.routers.backend-https.tls.certresolver=letsencryptresolver"

    depends_on:
      - db

  db:
    image: postgres:13
    environment:
      POSTGRES_PASSWORD_FILE: /run/secrets/postgres_password
      POSTGRES_USER: app
      POSTGRES_DB: app
    expose:
      - "5432"
    secrets:
      - postgres_password
    networks:
      - app-network

  adminer:
    image: adminer
    restart: always
    expose:
      - "8080"
    environment:
      ADMINER_DEFAULT_SERVER: db
    labels:
      - traefik.enable=true
      # HTTP Router
      - "traefik.http.routers.adminer-http.rule=Host(`{{ db_domain }}`) || Host(`www.{{ db_domain }}`)"
      - "traefik.http.routers.adminer-http.entrypoints=web"
      - "traefik.http.services.adminer.loadbalancer.server.port=8080"
      # www to non-www redirect
      - "traefik.http.routers.adminer-https.middlewares=www-to-non-www"
      # HTTPS Router
      - "traefik.http.routers.adminer-https.rule=Host(`{{ db_domain }}`) || Host(`www.{{ db_domain }}`)"
      - "traefik.http.routers.adminer-https.entrypoints=websecure"
      - "traefik.http.routers.adminer-https.tls.certresolver=letsencryptresolver"
    networks:
      - app-network

  traefik:
    image: traefik:v2.10.1
    restart: unless-stopped
    command:
      - "--entrypoints.web.address=:80"
      - "--entrypoints.web.http.redirections.entryPoint.to=websecure"
      - "--entrypoints.web.http.redirections.entryPoint.scheme=https"
      - "--entrypoints.websecure.address=:443"
      - "--providers.docker=true"
      - "--providers.docker.exposedByDefault=false"
      - "--api"
      - "--certificatesresolvers.letsencryptresolver.acme.email={{ cert_email }}"
      - "--certificatesresolvers.letsencryptresolver.acme.storage=/acme.json"
      - "--certificatesresolvers.letsencryptresolver.acme.tlschallenge=true"
      - "--accesslog=true"
      - "--log.level=ERROR"
    ports:
      - 80:80
      - 443:443
    expose:
      - "8080"
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./traefik/acme.json:/acme.json
    labels:
      - "traefik.enable=true"
      # HTTP Router
      - "traefik.http.routers.traefik-http.rule=Host(`{{ traefik_domain }}`) || Host(`www.{{ traefik_domain }}`)"
      - "traefik.http.routers.traefik-http.entrypoints=web"
      - "traefik.http.services.traefik-http.loadbalancer.server.port=8080"
      # www to non-www redirect
      - "traefik.http.routers.traefik-https.middlewares=www-to-non-www"
      # HTTP to HTTPS redirect
      - "traefik.http.middlewares.www-to-non-www.redirectregex.regex=^https?://www\\.(.+)"
      - "traefik.http.middlewares.www-to-non-www.redirectregex.replacement=https://$1"
      - "traefik.http.middlewares.www-to-non-www.redirectregex.permanent=true"
      # HTTPS Router
      - "traefik.http.routers.traefik-https.rule=Host(`{{ traefik_domain }}`) || Host(`www.{{ traefik_domain }}`)"
      - "traefik.http.routers.traefik-https.entrypoints=websecure"
      - "traefik.http.routers.traefik-https.service=api@internal"
      - "traefik.http.routers.traefik-https.tls.certresolver=letsencryptresolver"
    networks:
      - app-network
      - monitor-network

networks:
  app-network:
    external: true
  monitor-network:
    external: true
volumes:
  postgres_data:
secrets:
   postgres_password:
     file: ./POSTGRES_PASSWORD.txt

Dynamic Grafana Dashboard

This setup also automatically configures data sources in Grafana and loads the dashboard. The configurations are stored in the monitoring folder.

  • prometheus-datasource.yml: Configures the Prometheus data source on Grafana.

      apiVersion: 1
  datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090/prometheus

  • loki-datasource.yml: Configures the Loki data source on Grafana.

      apiVersion: 1
  datasources:
    - name: Loki
      type: loki
      access: proxy
      url: http://loki:3100
      jsonData:
        timeout: 60
        maxLines: 1000

  • dashboard-providers.yml: Configures the dashboard provisioning from the config files.

      apiVersion: 1
  providers:
    - name: 'default'
      orgId: 1
      folder: ''
      type: file
      disableDeletion: false
      updateIntervalSeconds: 10 #how often Grafana will scan for changed dashboards
      allowUiUpdates: true  
      editable: true
      options:
        path: /var/lib/grafana/dashboards

  • dashboards: This folder contains individual dashboard configurations in JSON format.

    • cadvisor.json (container dashboard)

    • containers.json (container dashboard)

    • logs.json (logs dashboard)

  • The options below applies the configuration to grafana to ensure dynamic dashboard loading.

        volumes:
      - grafana:/var/lib/grafana
      - ./monitoring/loki-datasource.yml:/etc/grafana/provisioning/datasources/loki-datasource.yml
      - ./monitoring/prometheus-datasource.yml:/etc/grafana/provisioning/datasources/prometheus-datasource.yml
      - ./monitoring/dashboard-providers.yml:/etc/grafana/provisioning/dashboards/dashboard-providers.yml
      - ./monitoring/dashboards:/var/lib/grafana/dashboards # dashboard json files

sensitive Files and environment variables

  • frontend/.env: Contains environment variables for the frontend. This file is set up to receive variable values defined in terraform.tfvars.

  • backend/.env: Contains environment variables for the backend.

  • POSTGRES_PASSWORD.txt: Holds the database password. This file is mounted as a secret and securely injected into the PostgreSQL container.

    Disclaimer: Please note that these environment files and sensitive data should only exist on your server for security reasons. They were included here only for demonstration and transparency. Make sure to add these files to your .gitignore file.

Project Setup

Prerequisites

  • Ensure you have Terraform and Ansible installed on your machine.

  • Set up AWS programmatic access.

  • Create an S3 bucket for managing the remote Terraform state file.

  • Create a DynamoDB table for state locking.

Step 1: Build Docker Images

  • clone the application repo

      git clone https://github.com/DrInTech22/full-stack-monitoring.git
  cd full-stack-monitoring

  • Build the Docker images for the frontend (React) and backend (FastAPI) applications.

    • docker build -t maestrops/frontend:latest .

    • docker build -t maestrops/backend:latest .

  • Login to docker on your terminal

    • docker login -u maestrops

  • Pushed the images to Docker Hub:

    • docker push maestrops/frontend:latest

    • docker push maestrops/backend:latest

  • remove the cloned repo

Step 2: Set up configuration files

  • clone this repo

      git clone https://github.com/DrInTech22/full-stack-automation.git
  cd full-stack-automation
    • Edit the terraform.tfvars, backend/.env, and POSTGRES_PASSWORD.txt file to match your configurations.

Step 3: Apply Terraform config.

  • run the command below

      cd terraform
  terraform init
  terraform plan
  terraform apply --auto-approve

Step 4: Access your applications and monitoring dashboards

You can now sit back and let Terraform and Ansible handle everything for you.

After all resources have been successfully created:

  • Access all your applications at their respective domains.

  • Login to your grafana to access your dashboards.

Conclusion

We have learnt how to automate the entire deployment of an application using Terraform and Ansible. With a single command, "terraform apply," you can provision the server, set up DNS records, configure the server, and deploy the application and monitoring stacks with dynamic dashboards.

This reliable setup ensures consistent deployment, scalability, efficient resource management, and infrastructure reproducibility during unexpected incidents.

#terraform#ansible#aws#grafana#docker#docker-compose

Comments (2)

Join the discussion
K
KkInTech151y ago

Well done. Great details. Thanks a lot.

D
David1y ago

Very well written and detailed. Thank you.

1

More from this blog

DrInTech

14 posts

Welcome my blog! I am just a normal guy with interest to push out simple and easy-to-digest contents that impacts our world positively. Making the world a better place with one line of ink at a time