Traffic Mirroring with NGINX Mirror Module

A few months ago, I decided to migrate one of our company services from PHP to Go. This decision came about because we lacked engineers to continue developing or supporting our existing PHP endpoints, while new endpoints were being developed in Go. Although I have substantial experience with Ruby, Go, and JavaScript, I wasn't eager to invest time in enhancing legacy endpoints written years ago.

The Migration Challenge

When dealing with high-traffic services, choosing the right migration strategy is crucial. While researching options, I discovered the NGINX mirror module in the official documentation: https://nginx.org/en/docs/http/ngx_http_mirror_module.html.

This module seemed like an ideal solution for my use case. To create a prototype, I used Docker Compose to organize all services in one place and implemented logging for basic requests.

Implementation Example

Here's a simplified version of my setup:

version: "3"

services:
  entry:
    image: nginx
    volumes:
     - ./v2/nginx.template:/etc/nginx/conf.d/default.conf
    links:
      - v1
      - v2_backend
    ports:
     - "80:80"
    environment:
     - NGINX_PORT=80
    command: /bin/bash -c "exec nginx -g 'daemon off;'"

  v1:
    image: nginx
    volumes:
     - ./v1/nginx.template:/etc/nginx/conf.d/default.conf
    links:
      - v1_backend
    expose:
     - "80"
    environment:
     - NGINX_PORT=80
    command: /bin/bash -c "exec nginx -g 'daemon off;'"

  v1_backend:
    image: example-image1:latest
    expose:
      - "9000"

  v2_backend:
    image: example-image2:latest
    expose:
      - "5000"

PHP Configuration (v1/nginx.template)

This is the NGINX configuration file for the PHP service using FastCGI:

upstream backend {
  server v1_backend:9000;
}

server {
    listen       80;
    server_name  _;
    root /var/www/html;

    location / {
        try_files $uri $uri/ @rewrite;
    }

    location @rewrite {
      rewrite ^/(.*)$ /index.php?q=$1 last;
    }

    error_page  404              /404.html;
    error_page   500 502 503 504  /50x.html;

    location ~ .php$ {
        try_files  $uri =404;

        fastcgi_pass   backend;
        fastcgi_index  index.php;
        fastcgi_param  SCRIPT_FILENAME /var/www/html$fastcgi_script_name;
        include        fastcgi_params;

        fastcgi_max_temp_file_size 0;

        fastcgi_buffers 32 256k;
    }
}

Go Configuration with Mirroring (v2/nginx.template)

This NGINX template serves our new Go API while mirroring traffic to it:

upstream v1.backend {
    server v1;
}

upstream v2.backend {
    server v2_backend:5000;
}

server {
    listen 80;
    server_name _;

    location / {
      mirror /mirror;
      mirror_request_body on;

      proxy_pass http://v1.backend;
    }

    location = /mirror {
      internal;

      proxy_pass http://v2.backend$request_uri;
      proxy_set_header X-SERVER-PORT $server_port;
      proxy_set_header X-SERVER-ADDR $server_addr;
      proxy_set_header X-Original-URI $request_uri;
      proxy_set_header HOST $http_host;
      proxy_set_header X-REAL-IP $remote_addr;
    }
}

How It Works

This example demonstrates a basic approach to achieve a smooth migration. The PHP application serves as the primary application, while the Go app functions as the V2 API.

The key configuration is mirror /mirror;, which instructs NGINX to process each request through the /mirror location and then forward it to the v2 backend. This allows us to verify traffic behavior through console logs.

Once the V2 application is functioning correctly, we can completely remove V1 and update the NGINX template to a new version without the mirroring configuration.

Conclusion

Using NGINX's mirror module provides an elegant solution for gradually migrating between services while ensuring continuity. This approach allows you to validate your new implementation under real traffic conditions before fully committing to the switch.

Thanks for reading!