Deploy Full-Stack App with Docker and Nginx

Introduction

This is the first project from the November DevOps CV challenge, organized by the DevOps Dojo team. The challenge aims to give you real-world projects that will strengthen your portfolio and help you stand out when applying for jobs. I have embraced the challenge, and here is my project solution and implementation.

In this article, we’ll walk through how to deploy a full-stack application using Docker Compose and set up monitoring and logging to keep track of the application's health and gain useful insights from the logs.

Prerequisites

To follow along with this project, make sure you have:

Basic knowledge of Docker
Understanding of how to set up an EC2 server with the necessary security groups
Basic knowledge of Prometheus and Grafana

Challenge Overview

The tasks for the challenge are outlined below:

Dockerize a full-stack application with a FastAPI backend and a React frontend.
Set up and deploy monitoring and logging tools, including Prometheus, Grafana, Loki, Promtail, and cAdvisor.
Configure a reverse proxy for application routing.
Deploy the application to a cloud platform with the correct domain setup.
Submit a working repository with detailed documentation and screenshots of your deployed application.

Project Implementation

This article will briefly cover deployment but will focus more on monitoring. For a step-by-step guide on local and production deployment, click here.

Technologies Used:

Docker: For containerizing applications.
AWS EC2: Infrastructure for deploying applications.
PostgreSQL: Serves as the database.
cAdvisor: Exposes container metrics.
Promtail: Collects logs from different sources on the server and sends them to Loki.
Loki: Aggregates and stores logs received from Promtail.
Prometheus: Collects metrics from cAdvisor and sends them to Grafana.
Grafana: Visualizes metrics and logs.

Containerization

Let's start by cloning the repository.

 git clone https://github.com/DrInTech22/full-stack-monitoring.git
 cd full-stack-monitoring

Here are the Dockerfiles for the frontend (React) and the backend (FastAPI) applications.

Frontend Dockerfile

# Use the latest official Node.js image as a base
FROM node:latest

# Set the working directory
WORKDIR /app

# Copy the application files
COPY . .

# Install dependencies
RUN npm install

# Expose the port the development server runs on
EXPOSE 5173

# Run the development server
CMD ["npm", "run", "dev", "--", "--host"]

This Dockerfile is used to package the application into a reusable Docker image. The CMD command makes the application accessible over the internet by exposing it on a public interface.

Backend Dockerfile

# Use Python 3.10 image as base
FROM python:3.10

# Set the working directory in the container
WORKDIR /app

# Install curl and bash
RUN apt-get update -y && \
    apt-get install -y --no-install-recommends \
    curl \
    && rm -rf /var/lib/apt/lists/*

# Install Poetry
RUN curl -sSL https://install.python-poetry.org | python3 -

# Add Poetry to the PATH
ENV PATH="/root/.local/bin:$PATH"

# Copy the rest of the application code into the container
COPY . /app

# Install the dependencies using Poetry
RUN poetry install

# Make sure the prestart script is executable
RUN chmod +x ./prestart.sh

# Expose the port the app runs on
EXPOSE 8000

# Run the prestart script and start the server
CMD ["sh", "-c", "poetry run bash ./prestart.sh && poetry run uvicorn app.main:app --host 0.0.0.0 --port 8000 --reload"]

The Dockerfile above packages the backend application into a reusable docker image.

Docker Compose Setup & Monitoring Configurations

The monitoring folder contains configuration files for Promtail, Loki, and Prometheus.

promtail-config.yml

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
- job_name: system
  static_configs:
  - targets:
      - "localhost"
    labels:
      job: varlogs
      __path__: /var/log/*log
    labels:
      job: docker_logs
      __path__: /var/lib/docker/containers/*/*-json.log

The promtail configuration sets up a server to listen on port 9080, defines where to store log positions, specifies the Loki endpoint for log shipping, and configures log scraping from the server and Docker log files with appropriate labels.

loki-config.yml

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096

common:
  instance_addr: 127.0.0.1
  path_prefix: /tmp/loki
  storage:
    filesystem:
      chunks_directory: /tmp/loki/chunks
      rules_directory: /tmp/loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

query_range:
  results_cache:
    cache:
      embedded_cache:
        enabled: true
        max_size_mb: 100

schema_config:
  configs:
    - from: 2020-10-24
      store: tsdb
      object_store: filesystem
      schema: v12
      index:
        prefix: index_
        period: 24h

This configuration sets up Loki to run on specific ports, use in-memory storage, enable caching, and define the schema for storing and indexing log data.

prometheus.yml

global:
  scrape_interval: 15s
scrape_configs:
  - job_name: "prometheus"
    metrics_path: "/prometheus/metrics"
    static_configs:
      - targets: ["prometheus:9090"]
  - job_name: "cadvisor"
    static_configs:
      - targets: ["cadvisor:8080"]

This configuration sets up Prometheus to scrape metrics from two sources: the Prometheus instance itself and cAdvisor, at specified intervals and paths.

Also, the challenge requires us to use two docker compose files. This is to promote modularity and avoid cluttering a single compose file with all the services and configurations. A single command docker compose up -d will simultaneously deploy both compose files.

We have two compose files:

compose.yml: This is the base compose file for the application stack. It is responsible for deploying the frontend, backend, database, and reverse proxy manager.
compose.monitoring.yml: This is the additional compose file for the monitoring stack. It deploys prometheus, grafana, loki, promtail, and cadvisor containers on the server.

Application stack: Compose.yml

include:
  - compose.monitoring.yml
services:
  frontend:
    build:
      context: ./frontend
    env_file:
      - frontend/.env
    depends_on:
      - backend
    ports:
      - "5173:5173"
    networks:
      - frontend-network

  backend:
    build:
      context: ./backend
    env_file:
      - backend/.env
    networks:
      - frontend-network
      - backend-network
    depends_on:
      - db

  db:
    image: postgres:13
    environment:
      POSTGRES_PASSWORD_FILE: /run/secrets/postgres_password
      POSTGRES_USER: app
      POSTGRES_DB: app
    secrets:
      - postgres_password
    networks:
      - backend-network
    volumes:
      - postgres_data:/var/lib/postgresql/data

  adminer:
    image: adminer
    restart: always
    environment:
      ADMINER_DEFAULT_SERVER: db
    networks:
      - backend-network
  nginx:
    image: 'jc21/nginx-proxy-manager:2.10.4'
    ports:
      - '80:80'
      - '443:443'
      - '8090:81'
    environment:
      DB_SQLITE_FILE: "/data/database.sqlite"
    volumes:
      - data:/data
      - letsencrypt:/etc/letsencrypt
      #- ./nginx/nginx.conf:/data/nginx/custom/http_top.conf
    restart: always
    depends_on:
      - frontend
      - backend
      - adminer
      - prometheus
      - grafana
    networks:
      - frontend-network
      - backend-network

networks:
  frontend-network:
  backend-network:
volumes:
  postgres_data:
  data:
  letsencrypt:

secrets:
   postgres_password:
     file: ./POSTGRES_PASSWORD.txt

This setup includes the frontend and backend, each with their respective .env files. Adminer is used to access the PostgreSQL database through a user interface, and Nginx Proxy Manager (NPM) manages and routes traffic to the containers.

The PostgreSQL data has a volume for data persistence, and similarly, NPM uses a volume to store SSL certificates.

The database credentials are directly added to the container from a securely stored file on the server. This method helps us avoid using export to set PostgreSQL credentials as environment variables on the server, which could lead to credentials being exposed in logs, making them insecure.

To securely set the password run:

echo "changethis123" > POSTGRES_PASSWORD.txt

Monitoring stack: compose.monitoring.yml

services:
  prometheus:
    image: prom/prometheus
    ports:
      - "9090:9090"
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--web.external-url=/prometheus'
    volumes:
      - ./monitoring/prometheus.yml:/etc/prometheus/prometheus.yml
    networks:
      - frontend-network
    depends_on:
      - cadvisor

  grafana:
    image: grafana/grafana-oss
    expose:
      - "3000"
    ports:
      - "3000:3000"
    environment:
      - GF_SERVER_ROOT_URL=http://localhost:3000/grafana
      - GF_SERVER_SERVE_FROM_SUB_PATH=true
    volumes:
      - grafana:/var/lib/grafana
    networks:
      - frontend-network
    depends_on:
      - loki

  cadvisor:
    image: gcr.io/cadvisor/cadvisor:v0.47.0
    ports:
      - "8081:8080"
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:rw
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
    networks:
      - frontend-network
      - backend-network
    depends_on:
      - backend
      - frontend
      - adminer
      - db

  loki:
    image: grafana/loki:latest
    ports:
      - 3100:3100
    networks:
      - frontend-network
    volumes:
      - ./monitoring/loki-config.yml:/etc/loki/loki-config.yaml

  promtail:
    image: grafana/promtail:latest
    networks:
      - frontend-network
      - backend-network
    volumes:
      - ./monitoring/promtail-config.yml:/etc/promtail/promtail-config.yaml
    depends_on:
      - loki

volumes:
  grafana:

This will deploy the monitoring stack containers. Promtail, Prometheus, and Loki will use their respective configurations from the monitoring folder.

Install Loki Log Driver

Let's set up Docker to send container logs directly to Loki.

Install Loki log driver

  docker plugin install grafana/loki-docker-driver:2.9.1 --alias loki --grant-all-permissions

edit /etc/docker/daemon.json with the config below

  {
     "log-driver": "loki",
     "log-opts": {
       "loki-url": "http://localhost:3100/loki/api/v1/push",
       "loki-batch-size": "400"
     }
   }

Updating Env Files and Deploying the Applications

The database credentials in the backend .env file must match the PostgreSQL credentials.

# Postgres
POSTGRES_SERVER=db # postgresql container service name
POSTGRES_PORT=5432
# POSTGRES_DB=app
POSTGRES_USER=app
POSTGRES_PASSWORD=changethis123

To avoid CORS errors when accessing the frontend, include http://<ip_addr> and http://<ip_addr>:5173 in the frontend .env file. Then, update the URL in the frontend .env to http://<ip_addr>:8000.

You can now deploy the applications using docker compose up. The logs will help us see what's happening since we haven't fully configured the monitoring stack yet. All the applications are now accessible over HTTP, and you can easily log in using the SUPERUSER credentials.

Configuring SSL Certs for secure access

The next step is to ensure secure access to our applications over HTTPS. We'll configure SSL certificates for our domains using Let's Encrypt on the NPM UI.

Access the NPM UI at http://<public_IP>:8090.
Create SSL certificates for each domain as shown below.
Pay attention to the paths in the log where NPM stores the SSL certificates. These paths will be included in the nginx.conf file.
Open the nginx.conf file and make sure the domain names match your configured domain names. Then, add the SSL paths for each domain in their respective SSL server blocks.

Nginx.conf

# Configuration for cv1.drintech.online
server {
    listen 80;
    server_name cv1.drintech.online www.cv1.drintech.online;

    # Redirect all HTTP traffic to HTTPS
    location / {
        return 301 https://cv1.drintech.online$request_uri;
    }
}

server {
    listen 443 ssl;
    server_name cv1.drintech.online www.cv1.drintech.online;

    # Redirect www to non-www
    if ($host = 'www.cv1.drintech.online') {
        return 301 https://cv1.drintech.online$request_uri;
    }

    # Add your SSL certificate and key paths here
    ssl_certificate /etc/letsencrypt/live/npm-5/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/npm-5/privkey.pem;

    location /api {
        proxy_pass http://backend:8000/api;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }

    location /docs {
        proxy_pass http://backend:8000/docs;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }

    location /redoc {
        proxy_pass http://backend:8000/redoc;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }

    location /prometheus {
        proxy_pass http://prometheus:9090;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }

    location /grafana {
        proxy_pass http://grafana:3000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }

    location / {
        proxy_pass http://frontend:5173;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

# Configuration for db.cv1.drintech.online
server {
    listen 80;
    server_name db.cv1.drintech.online www.db.cv1.drintech.online;

    # Redirect all HTTP traffic to HTTPS
    location / {
        return 301 https://db.cv1.drintech.online$request_uri;
    }
}

server {
    listen 443 ssl;
    server_name db.cv1.drintech.online www.db.cv1.drintech.online;

    # Redirect www to non-www
    if ($host = 'www.db.cv1.drintech.online') {
        return 301 https://db.cv1.drintech.online$request_uri;
    }

    # Add your SSL certificate and key paths here
    ssl_certificate /etc/letsencrypt/live/npm-6/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/npm-6/privkey.pem;

    location / {
        proxy_pass http://adminer:8080;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

# Configuration for nginx.cv1.drintech.online
server {
    listen 80;
    server_name nginx.cv1.drintech.online www.nginx.cv1.drintech.online;

    # Redirect all HTTP traffic to HTTPS
    location / {
        return 301 https://nginx.cv1.drintech.online$request_uri;
    }
}

server {
    listen 443 ssl;
    server_name nginx.cv1.drintech.online www.nginx.cv1.drintech.online;

    # Redirect www to non-www
    if ($host = 'www.nginx.cv1.drintech.online') {
        return 301 https://nginx.cv1.drintech.online$request_uri;
    }

    # Add your SSL certificate and key paths here
    ssl_certificate /etc/letsencrypt/live/npm-7/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/npm-7/privkey.pem;

    location / {
        proxy_pass http://nginx:81;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

The nginx.conf file sets up the proxy host for sub-domains, enables www to non-www redirection and HTTP to HTTPS redirection, and routes traffic over HTTPS.

In the compose.yml, uncomment the nginx.conf mount.

  volumes:
  - data:/data
  - letsencrypt:/etc/letsencrypt
  # -./nginx/nginx.conf:/data/nginx/custom/http_top.conf

Update the API URL in the frontend .env file to https://cv1.drintech.online.
The next step is to recreate the NPM container so our changes take effect.
```
  docker compose up frontend nginx --force-recreate
```

Test the Deployment

Verify that each service is running and securely accessible.

FastAPI Backend: cv1.drintech.online/api
FastAPI Backend Docs: cv1.drintech.online/docs
FastAPI Backend Redoc: cv1.drintech.online/redoc
Prometheus: cv1.drintech.online/prometheus
Grafana: cv1.drintech.online/grafana
Node.js Frontend: cv1.drintech.online
Adminer: db.cv1.drintech.online
Nginx Proxy Manager: nginx.cv1.drintech.online

Test that HTTP to HTTPS redirection and www to non-www redirection are working:

Dashboard setup

Grafana is a powerful tool that helps us visualize metrics and logs, providing quick and useful insights into the health of our applications.

First, let's set up our data sources: Prometheus and Loki.

On the left sidebar, click on "Data Sources," search for Prometheus, and select it as a data source.
Enter the Prometheus service URL http://prometheus:9090/prometheus , leave other settings as default, and click "save and test”.
Do the same for the Loki service using the URL http://loki:3100.

We will set up two dashboards:

Containers dashboard: This shows the health of the containers based on resource usage like RAM, memory, CPU, and network traffic.
Logs dashboard: This includes visualizations for container logs and server logs.

Container Dashboard Setup

Follow these steps to set up the container dashboard:

Log in to Grafana.
In the upper right corner, click the ‘+’ icon and select "Import Dashboard."
Use the dashboard ID 19792 then click “load”.
Set your dashboard name, select Prometheus as the data source and click “import”
You will have a beautiful dashboard with several visualizations for various container metrics.

Logs Dashboard

We’ll build the logs dashboard from scratch. Get ready to write some Loki query language.

Start by creating a new dashboard. Click settings in the upper right corner, set the dashboard name - “Logs Dashboard”, go to the variables tab and create these variables:
- Name: container, Display Name: container, Variable Type: Query, Label Value: container_name
- Name: container_search, Display Name: search, Variable Type: textbox
- Name: severity, Display Name: severity, Variable Type: custom, Custom Values: info, warn, error
  
  Note: The commas are important in the custom values.
- Name: NodeLogs, Display Name: NodeLogs, Variable Type: Query, Label Value: filename
- Name: varlog_search, Display Name: NodeLog filter, Variable Type: textbox
- Save the dashboard, then create a new visualization.
- Insert the following Loki query in the code tab.

    {container_name="$container"} |~ `$container_search` | logfmt level | (level =~ `$severity` or level= "")

The query above sets up the visualization for container logs. It includes options to select which container logs to view, filter based on severity (info, warn, error), and an additional option to search through the filtered logs.

Let's create the Node Logs visualization for server logs.
Create another visualization using the Loki query below:
```
 {filename="$NodeLogs"} |~ `$varlog_search`
```
The query above allows you to choose any server logs you want and filter them using your preferred keywords.
Ensure the variables you created all follow the order below.
You should now have a fully functional logs dashboard. You can test to ensure the filters and search labels are working as expected.

Conclusion

I know this was a long journey, but we've learnt how to containerize and deploy a full-stack app using Docker Compose. We then set up a monitoring and logging system with tools like Loki, Prometheus, Promtail, Cadvisor, and Grafana. These tools assist with log aggregation, metrics collection, log shipping, resource analysis, and creating visual dashboards.

By integrating these tools, we've created a robust infrastructure that supports our app, providing real-time insights into its performance and health. This setup allows us to quickly identify and resolve issues, offering a strong foundation for maintaining and scaling our app.

Till I write again, Adios.

Resources

https://www.virtualizationhowto.com/2023/01/prometheus-node-exporter-cadvisor-grafana-install-and-configure/

https://medium.com/@akifmalik200/guide-to-setting-up-prometheus-grafana-cadvisor-and-alertmanager-with-docker-67ff1542f9ac

https://medium.com/@alexishevia/setting-up-traefik-4026bda980bf

https://abhiraj2001.medium.com/monitoring-docker-containers-with-grafana-loki-and-promtail-4302a9417c0d

https://medium.com/@netopschic/implementing-the-log-monitoring-stack-using-promtail-loki-and-grafana-using-docker-compose-bcb07d1a51aa

https://youtu.be/h_GGd7HfKQ8?si=_yfIqt1fIxWTUksJ

https://youtu.be/AtxQHiFBn7k?si=jvcYAcxzsHfvJRu-