Blue/green deploys for OONI API services on Hetzner#454
Draft
hellais wants to merge 3 commits into
Draft
Conversation
Terraform Run Output 🤖Format and Style 🖌
|
| Pusher | @hellais |
| Action | pull_request |
| Environment | dev |
| Workflow | .github/workflows/check_terraform.yml |
| Last updated | Fri, 10 Jul 2026 19:53:46 GMT |
Ansible Run Output 🤖Ansible Playbook Recap 🔍Ansible playbook output 📖
|
| Pusher | @hellais |
| Action | pull_request |
| Working Directory | |
| Workflow | .github/workflows/check_ansible.yml |
| Last updated | Fri, 10 Jul 2026 19:53:37 GMT |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
ooniapi_service_deployer now supports two deploy backends behind one new variable, deploy_mode:
All 6 services (reverseproxy, ooniprobe, oonirun, oonifindings, ooniauth, oonimeasurements) in both dev and prod are wired with deploy_mode = "ecs" today. Nothing about their live deploy path has changed. Flipping a service later is a one-line edit (deploy_mode = "blue_green") plus the port and host variables noted below.
The blue/green model
Each service runs as two Quadlet units on each of two hosts: {service}-a and {service}-b, bound to two fixed host ports (host_port_a, host_port_b, loopback-only). An nginx upstream block for the service lists both ports, with one marked down. A deploy:
reads /etc/ooniapi/{service}/active_slot on the host to find which slot is live (defaults to a if the file doesn't exist yet, which is how a host bootstraps its first deploy),
installs the other slot's Quadlet unit with the new image tag and restarts only that slot,
health-checks the new slot directly on its host port, bypassing nginx,
only if that passes: flips the nginx upstream so the new slot is up and the old one down, reloads nginx, and updates the marker file.
The currently-live slot is never touched until the new one has proven healthy. That's what makes it zero-downtime and safely rollback-able: a failed health check aborts before nginx ever sees the new slot.
This runs sequentially against deploy_host_primary then deploy_host_secondary. If either host fails, the whole CodeBuild job, and the pipeline with it, fails. The two hosts never end up silently serving different versions.
What Terraform manages now vs. what's still manual
Terraform-managed as of this change:
Per-slot Quadlet .container files and the nginx upstream conf, rendered from templates and uploaded to S3 (var.quadlet_units_bucket, once set).
A dedicated IAM role and policy for the deploy CodeBuild project, scoped to the SSH-key secret, the service's secrets, S3 read on its own {service_name}/* prefix, and reading the pipeline's build artifact.
A new aws_codebuild_project (ooniapi-{service}-deploy) running templates/buildspec_deploy.yml, wired as the CodePipeline Deploy stage's action when deploy_mode = "blue_green".
One aws_secretsmanager_secret per service (oonidevops/ooniapi/{service}/service_secrets), new this round, whose value is a JSON blob built from the same underlying SSM parameters already feeding that service's ECS task_secrets. Both deploy paths draw from one source of truth and can't drift.
One shared aws_secretsmanager_secret (oonidevops/ooniapi/deploy_ssh_key) per environment, new this round, for the CodeBuild deploy job's SSH private key. Terraform creates the container with a placeholder value and ignore_changes = [secret_string]. It never manages or sees the real key material.
env_vars on every deployer module call now mirrors that service's existing task_environment verbatim, so the Quadlet unit's cleartext environment matches what ECS already injects.
Still manual, out-of-band, unchanged from the original plan:
Generating the deploy user's SSH keypair and putting the private key into oonidevops/ooniapi/deploy_ssh_key (public key goes on the hosts' ~deploy/.ssh/authorized_keys).
One-time per-host, per-service bootstrap before the first blue/green deploy can run:
Still pending before any service can flip
Operational notes for later
Rotating the deploy SSH key: update the Secrets Manager secret value directly via console or CLI. Terraform won't touch it (ignore_changes). No apply needed.
Rotating a service secret, for example JWT_ENCRYPTION_KEY: update the underlying SSM parameter as today. The next terraform apply refreshes the consolidated Secrets Manager JSON automatically, and the next deploy pushes the new value to both hosts via podman secret create --replace.
Manual rollback: SSH in, flip the nginx upstream conf back to the previous slot's up/down state, run nginx -t && systemctl reload nginx, and rewrite active_slot. The previous slot's container is still running (it's only replaced on the next deploy), so rollback doesn't require restarting anything.
Terraform state note: the per-service secret JSON blobs now contain plaintext values (DB URLs, JWT keys, AWS creds) in Terraform state, sourced via data "aws_ssm_parameter"...value and data "aws_secretsmanager_secret_version". This is materially the same exposure the ECS path already has in practice, since those data sources refresh and cache full attribute values in state regardless of which fields your config references.
note: this is a genai generated PoC