Use the shared edge nginx instead of Musify's own
CI/CD / BackEnd (push) Successful in 1m27s
CI/CD / WebPlayer (push) Successful in 39s
CI/CD / Deploy (push) Skipped
CI/CD / BackEnd (pull_request) Successful in 1m16s
CI/CD / WebPlayer (pull_request) Successful in 39s
CI/CD / Deploy (pull_request) Skipped

Remove deploy/nginx and the nginx service from compose.prod.yml: the
shared edge in Infrastructure owns ports 80/443 and serves every
domain. Make the deploy job fail when zitadel-init or seaweedfs-init
fail, and document the runner and edge setup.
This commit is contained in:
2026-09-23 16:56:27 +02:00
parent 8970f29410
commit 5efd59826b
7 changed files with 67 additions and 259 deletions
+43 -75
View File
@@ -13,7 +13,7 @@ wherever you actually cloned it.
|---|---|
| `compose.yml` | Musify's own services, including the one-shot bootstrap jobs. No published ports, no domains, no environment-specific values. |
| `compose.dev.yml` | Development overlay: publishes Musify's ports to the host. |
| `compose.prod.yml` | Production overlay: adds the nginx edge and joins Infrastructure's `infra-net` network. |
| `compose.prod.yml` | Production overlay: joins Infrastructure's `infra-net` network. |
| `.env` / `.env.prod` | The values compose interpolates. Not versioned; templates in `.env.example` / `.env.prod.example`. |
The four applications (`api`, `worker`, `gateway`, `web-player`) sit behind
@@ -113,31 +113,26 @@ SeaweedFS is done from the `Infrastructure` repo instead.
## Production
On a server with a domain:
Musify doesn't run its own nginx: the shared edge in Infrastructure
(`edge/`) owns host ports 80/443, terminates TLS for every project's domains
and proxies to Musify's containers over `infra-net`. On a server with
domains:
1. Bring up the `Infrastructure` repo first, with its own `.env.prod` filled
in. It creates the `infra-net` network this stack joins.
2. From this repository's root:
2. Bring up its `edge/` (see Infrastructure's README) and add Musify's
vhosts and certificate there (`edge/conf.d/musify.conf`). The five
public hostnames (web, api, stream, auth, s3) are independent: they
don't need to share a root domain.
3. From this repository's root:
```sh
cp deploy/.env.prod.example deploy/.env.prod
```
Fill it in:
1. Replace every `*.example.com` with your actual domains and every
`CHANGE_ME` with a random value (`openssl rand -hex 24`). Each of
`WEB_DOMAIN`/`API_DOMAIN`/`STREAM_DOMAIN`/`AUTH_DOMAIN`/`S3_DOMAIN` is
independent — they don't need to share a root domain (e.g.
`musify.example.com` + `musifyapi.example.com`, or sibling subdomains of
a domain whose DNS provider doesn't allow multi-level subdomains).
`chmod 600 deploy/.env.prod`.
2. Drop **`deploy/nginx/certs/origin.pem` and `origin.key`**, the origin
certificate nginx serves. The stack assumes Cloudflare in *Full (strict)*
mode in front of it, so a Cloudflare Origin Certificate is enough; any
certificate valid for the five hostnames works.
3. Point the five `*_DOMAIN` values at the server, proxied and
TLS-terminated the same way.
Replace every `*.example.com` with your domains (they must match the
edge's vhosts) and every `CHANGE_ME` with a random value
(`openssl rand -hex 24`). `chmod 600 deploy/.env.prod`.
Then:
@@ -146,68 +141,45 @@ docker compose --env-file deploy/.env.prod -f deploy/compose.yml -f deploy/compo
docker compose --env-file deploy/.env.prod -f deploy/compose.yml -f deploy/compose.prod.yml --profile apps up -d --build
```
The first command starts Musify's bootstrap jobs and the nginx edge; the
second builds and starts the applications, already wired to the OIDC client
ids the first one produced. Both are idempotent, so running them again
deploys a new version (`... --profile apps up -d --build` alone is enough for
an update once the stack already exists).
The first command starts Musify's bootstrap jobs (`zitadel-init` and
`seaweedfs-init` reach Zitadel/SeaweedFS through the public `auth.`/`s3.`
URLs, so the edge has to be serving them by then); the second builds and
starts the applications, already wired to the OIDC client ids the first one
produced. Both are idempotent, so running them again deploys a new version
(`... --profile apps up -d --build` alone is enough for an update once the
stack already exists).
Only nginx publishes ports (80/443). Every other Musify service, and all of
Infrastructure's, is reachable only on the internal `infra-net` network.
Nothing in this stack publishes a port; everything is reachable only on the
internal `infra-net` network, through the edge.
### Continuous deployment
`.gitea/workflows/ci-cd.yml` runs the backend and web-player checks on every
push/PR, and on a successful push to `master` deploys automatically. The
deploy job runs on `gitea-runner`, the existing self-hosted Gitea Actions
runner on the production host itself (`gitea.ikerdemo.cat`, in
`~/gitea/docker-compose.yml`), which also carries a `production:host`
label registered directly in `~/gitea/runner-data/.runner` (act_runner
reads `labels` from that file on every start, not from
`GITEA_RUNNER_LABELS`, once a runner is already registered). Running in
native mode means the label's steps execute inside the runner container
itself rather than spinning up a fresh one — that container has
`/var/run/docker.sock` mounted, so `docker`/`docker compose` there still
control the host's real Docker daemon (sibling containers, "Docker outside
of Docker"). Its compose service also bind-mounts `/home/ubuntu/deploys`
into itself at the *same* path, which matters because compose's own
relative bind mounts (`./keys`, `./nginx/...`) are resolved by the host
daemon against that literal path — mounting it anywhere else would silently
bind the wrong (empty) directory.
deploy job runs on `gitea-runner`, the self-hosted Gitea Actions runner on
the production host (`~/gitea/docker-compose.yml`), configured by
`~/gitea/runner-data/config.yaml` (`CONFIG_FILE`): job containers are pinned
to the `gitea_gitea` network so `actions/checkout` can resolve `gitea`,
`ubuntu-latest` is `node:22` (JS actions need Node inside the job
container), and a `production:host` label runs steps natively.
Native mode means the label's steps execute inside the runner container
itself rather than a fresh one. That container has `/var/run/docker.sock`
mounted, so `docker`/`docker compose` there still control the host's real
Docker daemon (sibling containers, "Docker outside of Docker"). Its compose
service also bind-mounts `/home/ubuntu/deploys` into itself at the *same*
path, which matters because compose's own relative bind mounts (`./keys`)
are resolved by the host daemon against that literal path; mounting it
anywhere else would silently bind the wrong (empty) directory.
`DEPLOY_PATH` (hardcoded in the workflow as `/home/ubuntu/deploys/musify`)
is a persistent clone on that host, reused instead of a fresh checkout each
run since `deploy/.env.prod` is untracked and only lives there.
`/home/ubuntu/deploys/SharedServices` is the sibling clone of the
`Infrastructure`/`SharedServices` repo, already running (`infra-net`
created); its `.env.prod` shares the Postgres/RabbitMQ/S3 credentials with
Musify's own `deploy/.env.prod` by design (see that repo's README).
**Pending DNS on the registered domains:** `deploy/.env.prod`'s five
`*_DOMAIN` values are set (`musify.ikerdemo.cat`, `musifyapi.ikerdemo.cat`,
`musifystream.ikerdemo.cat`, `auth.ikerdemo.cat`, `s3.ikerdemo.cat` —
siblings, not `api.musify.ikerdemo.cat`, because the DNS provider here
doesn't allow multi-level subdomains), but no DNS record points any of them
at the server yet, so:
- `zitadel-init` and `seaweedfs-init` (they verify themselves through the
public `auth.`/`s3.` URLs) fail every deploy until those two resolve and
are actually reachable — the workflow tolerates this on purpose, see its
comments. Until then, `api`/`worker` build and start but can't do
anything OIDC- or storage-dependent (`AUTH_CLIENT_ID`/`WEB_CLIENT_ID`/
`WEB_CLIENT_SECRET` in `.env.prod` stay blank); `gateway`/`web-player`
are unaffected.
- The `nginx` service from `compose.prod.yml` is intentionally **not**
brought up by the deploy job: it would try to publish host ports 80/443,
which `gitea-nginx` already owns for `gitea.ikerdemo.cat` (certbot). Once
the five DNS records above exist, merge Musify's vhosts
(`deploy/nginx/templates`) into `~/gitea/nginx/conf.d` instead — same
nginx, additional `server_name` blocks proxying to Musify's containers
over `infra-net` — rather than running a second nginx that would conflict
on those ports. That also needs a certificate valid for the 5 hostnames
(the existing setup uses certbot/Let's Encrypt, not the Cloudflare Origin
cert `compose.prod.yml`'s nginx assumes — reconcile whichever way when
that's set up).
`Infrastructure` repo (shared services + `edge/`), already running; its
`.env.prod` shares the Postgres/RabbitMQ/S3 credentials with Musify's own
`deploy/.env.prod` by design (see that repo's README). The edge is deployed
from that repo, not from this pipeline.
---
@@ -218,13 +190,9 @@ deploy/
├─ compose.yml # Musify's own services: wait-for-postgres, keys-init,
│ # migrate, seaweedfs-init, zitadel-init, api/worker/gateway/web-player
├─ compose.dev.yml # dev overlay: host ports
├─ compose.prod.yml # prod overlay: nginx edge, joins infra-net
├─ compose.prod.yml # prod overlay: joins infra-net
├─ .env.example / .env.prod.example
├─ keys/ # RS256 stream-ticket key pair (generated)
└─ nginx/
├─ templates/default.conf.template # vhosts, rendered with the domain by envsubst
├─ snippets/proxy.conf # proxy headers shared by every vhost
└─ certs/ # origin.pem + origin.key (not versioned)
└─ keys/ # RS256 stream-ticket key pair (generated)
```
The application images are built from `backend/Musify.*.Dockerfile` (build