From 63018a7c72ff9399321138bb8e1d1e32a319cdd7 Mon Sep 17 00:00:00 2001 From: XShine555 Date: Mon, 21 Sep 2026 22:09:08 +0200 Subject: [PATCH] Move shared infrastructure to a separate Infrastructure repo Postgres, Zitadel, SeaweedFS, RabbitMQ and Jaeger no longer run as part of this stack. They're shared services (used by Musify, Ping and docker-manager) that now live in a separate Infrastructure repository and must already be running. deploy/compose.yml keeps only what's specific to Musify: its own database connection, its own OIDC app and storage bucket (created against the shared Zitadel/SeaweedFS via zitadel-init/seaweedfs-init), and the application containers. A new wait-for-postgres one-shot replaces the in-file Postgres health check, which no longer works now that Postgres lives in a different compose project. compose.prod.yml joins Infrastructure's infra-net network instead of running its own copies. Docs and READMEs updated to match. --- README.md | 11 +- deploy/.env.example | 53 ++++----- deploy/.env.prod.example | 34 +++--- deploy/README.md | 162 +++++++++++++++------------- deploy/compose.dev.yml | 66 +++--------- deploy/compose.prod.yml | 24 ++--- deploy/compose.yml | 228 +++++++++++---------------------------- docs/README.md | 9 +- docs/architecture.md | 8 +- docs/development.md | 31 +++--- docs/projects.md | 4 +- 11 files changed, 261 insertions(+), 369 deletions(-) diff --git a/README.md b/README.md index 60fb8fc..3ea6e68 100644 --- a/README.md +++ b/README.md @@ -29,9 +29,11 @@ dedicated streaming gateway. **Web client**: SvelteKit 2 on Svelte 5 (runes), TypeScript, Tailwind 4. The session lives on the server, so the backend access token never reaches the browser. -**Infrastructure**: the whole stack runs from Docker Compose (dev and -production overlays), with Postgres, RabbitMQ, SeaweedFS, Zitadel and Jaeger -alongside the apps. +**Infrastructure**: Musify's own apps run from Docker Compose (dev and +production overlays). Postgres, RabbitMQ, SeaweedFS, Zitadel and Jaeger are +shared infrastructure that lives in a separate `Infrastructure` git +repository, used by Musify and by other projects that need the same OIDC +provider and object storage. ## Project layout @@ -42,6 +44,9 @@ deploy/ Docker Compose stacks for dev and production docs/ architecture notes, one file per area ``` +Postgres, Zitadel, SeaweedFS, RabbitMQ and Jaeger are not part of this +layout. They live in the separate `Infrastructure` repository instead. + ## Digging deeper [docs/](docs/README.md) covers how each piece works and why: architecture, diff --git a/deploy/.env.example b/deploy/.env.example index a5c82e3..160bfff 100644 --- a/deploy/.env.example +++ b/deploy/.env.example @@ -1,39 +1,39 @@ # Development configuration for deploy/compose.yml + compose.dev.yml. -# Copy to deploy/.env and adjust. Not versioned. +# Copy to deploy/.env and adjust. # # cp deploy/.env.example deploy/.env +# +# Requires ../../../Infrastructure running first (Postgres, Zitadel, +# SeaweedFS, RabbitMQ, Jaeger). See ../../../Infrastructure/README.md. +# Everything marked "must match Infrastructure's .env" below has to be the +# same value as the one in that file. DOTNET_ENVIRONMENT=Development # --- Public URLs ------------------------------------------------------------- # How each service is reached from *outside* Docker. `host.docker.internal` -# resolves to the same address from the browser, from apps running on the host -# and from inside the containers, which is what makes the OIDC issuer and the -# presigned S3 URLs valid in both run modes. +# resolves to the same address from the browser, from apps running on the +# host and from inside the containers, which is what makes the OIDC issuer +# and the presigned S3 URLs valid in both run modes. PUBLIC_WEB_URL=http://localhost:3000 PUBLIC_API_URL=http://localhost:5111 PUBLIC_STREAM_URL=http://localhost:8081 PUBLIC_AUTH_URL=http://host.docker.internal:8080 -# --- PostgreSQL (shared by Musify and Zitadel) ------------------------------- +# --- PostgreSQL (Infrastructure's shared instance, Musify's own database) --- +# POSTGRES_USER / POSTGRES_PASSWORD must match Infrastructure's .env. +POSTGRES_HOST=host.docker.internal POSTGRES_USER=postgres POSTGRES_PASSWORD=postgres POSTGRES_DB=musify_db -POSTGRES_HOST_PORT=59000 -# --- Zitadel (OIDC) ---------------------------------------------------------- -# The masterkey MUST be exactly 32 characters. The external domain/port form -# the token issuer, so they have to match PUBLIC_AUTH_URL. Changing them needs -# a re-init: docker compose -f deploy/compose.yml -f deploy/compose.dev.yml down -v -ZITADEL_MASTERKEY=MasterkeyNeedsToHave32Characters -ZITADEL_EXTERNAL_DOMAIN=host.docker.internal -ZITADEL_EXTERNAL_PORT=8080 -ZITADEL_EXTERNAL_SECURE=false -ZITADEL_DATABASE_PASSWORD=zitadel -# Console login: @zitadel. -ZITADEL_ADMIN_USERNAME=admin -ZITADEL_ADMIN_PASSWORD=Password1! -# Allows the OIDC apps to use plain-http redirect URIs (dev only). +# --- Zitadel (OIDC, shared instance) ----------------------------------------- +# Path to the admin service-account PAT Infrastructure's Zitadel writes on +# init. Adjust if your Infrastructure checkout lives somewhere else relative +# to this one. +ZITADEL_ADMIN_PAT_DIR=../../../Infrastructure/zitadel/.output +# Allows the OIDC apps this project creates to use plain-http redirect URIs +# (dev only). ZITADEL_DEV_MODE=true # OIDC clients: the zitadel-init one-shot service (see compose.yml) fills @@ -42,17 +42,25 @@ AUTH_CLIENT_ID= WEB_CLIENT_ID= WEB_CLIENT_SECRET= -# --- RabbitMQ (MassTransit) -------------------------------------------------- +# --- RabbitMQ (shared instance, MassTransit) -------------------------------- +# Must match Infrastructure's .env. +RABBITMQ_HOST=host.docker.internal RABBITMQ_DEFAULT_USER=guest RABBITMQ_DEFAULT_PASS=guest -# --- SeaweedFS (S3) ---------------------------------------------------------- +# --- SeaweedFS (S3, shared instance, Musify's own bucket) -------------------- +# S3_ACCESS_KEY / S3_SECRET_KEY must match Infrastructure's .env. +SEAWEEDFS_HOST=host.docker.internal S3_ACCESS_KEY=admin_access_key S3_SECRET_KEY=admin_secret_key S3_BUCKET=webapi-storage S3_PUBLIC_URL=http://host.docker.internal:8333 +S3_INTERNAL_URL=http://host.docker.internal:8333 S3_USE_HTTP=true +# --- Jaeger (shared instance, tracing) --------------------------------------- +JAEGER_HOST=host.docker.internal + # --- Web player -------------------------------------------------------------- # Key material for the encrypted session cookie: any long random string. SESSION_SECRET=dev-session-secret-change-me-0123456789abcdef @@ -61,6 +69,3 @@ AUTH_ALLOW_INSECURE_HTTP=true # --- Misc -------------------------------------------------------------------- AUTH_REQUIRE_HTTPS=false -PGADMIN_DEFAULT_EMAIL=admin@musify.dev -PGADMIN_DEFAULT_PASSWORD=admin -PGADMIN_HOST_PORT=5050 diff --git a/deploy/.env.prod.example b/deploy/.env.prod.example index b6befc8..08029c1 100644 --- a/deploy/.env.prod.example +++ b/deploy/.env.prod.example @@ -3,11 +3,14 @@ # # 1. replace every `example.com` with your domain, # 2. replace every CHANGE_ME with a strong random value -# (`openssl rand -hex 24`; the Zitadel masterkey needs exactly 32 chars), +# (`openssl rand -hex 24`), # 3. point these DNS records at the server, all proxied/TLS-terminated the # same way: example.com, www, api, auth, stream, s3. # -# Not versioned. Keep it readable only by the deploy user (chmod 600). +# Requires ../../../Infrastructure running first, on the same Docker host, +# with its own `infra-net` network already created. See +# ../../../Infrastructure/README.md. Not versioned. Keep it readable only by +# the deploy user (chmod 600). DOTNET_ENVIRONMENT=Production @@ -20,22 +23,15 @@ PUBLIC_STREAM_URL=https://stream.example.com PUBLIC_AUTH_URL=https://auth.example.com S3_PUBLIC_URL=https://s3.example.com -# --- PostgreSQL (shared by Musify and Zitadel) ------------------------------- +# --- PostgreSQL (Infrastructure's shared instance, Musify's own database) --- +# POSTGRES_USER / POSTGRES_PASSWORD must match Infrastructure's .env.prod. +POSTGRES_HOST=postgres POSTGRES_USER=postgres POSTGRES_PASSWORD=CHANGE_ME POSTGRES_DB=musify_db -# --- Zitadel (OIDC) ---------------------------------------------------------- -# The masterkey MUST be exactly 32 characters and can never be rotated in place. -ZITADEL_MASTERKEY=CHANGE_ME_32_CHARACTERS_EXACTLY_ -ZITADEL_EXTERNAL_DOMAIN=auth.example.com -ZITADEL_EXTERNAL_PORT=443 -ZITADEL_EXTERNAL_SECURE=true -ZITADEL_DATABASE_PASSWORD=CHANGE_ME -# Console login: @zitadel. -ZITADEL_ADMIN_USERNAME=admin -ZITADEL_ADMIN_PASSWORD=CHANGE_ME -# Keep OIDC apps strict about https redirect URIs in production. +# --- Zitadel (OIDC, shared instance) ----------------------------------------- +ZITADEL_ADMIN_PAT_DIR=../../../Infrastructure/zitadel/.output ZITADEL_DEV_MODE=false # OIDC clients: the zitadel-init one-shot service (see compose.yml) fills @@ -44,16 +40,22 @@ AUTH_CLIENT_ID= WEB_CLIENT_ID= WEB_CLIENT_SECRET= -# --- RabbitMQ (MassTransit) -------------------------------------------------- +# --- RabbitMQ (shared instance, MassTransit) -------------------------------- +RABBITMQ_HOST=rabbitmq RABBITMQ_DEFAULT_USER=musify RABBITMQ_DEFAULT_PASS=CHANGE_ME -# --- SeaweedFS (S3) ---------------------------------------------------------- +# --- SeaweedFS (S3, shared instance, Musify's own bucket) -------------------- +SEAWEEDFS_HOST=seaweedfs S3_ACCESS_KEY=CHANGE_ME S3_SECRET_KEY=CHANGE_ME S3_BUCKET=webapi-storage +S3_INTERNAL_URL=http://seaweedfs:8333 S3_USE_HTTP=false +# --- Jaeger (shared instance, tracing) --------------------------------------- +JAEGER_HOST=jaeger + # --- Web player -------------------------------------------------------------- # Key material for the encrypted session cookie. Rotating it logs everyone out. SESSION_SECRET=CHANGE_ME diff --git a/deploy/README.md b/deploy/README.md index 03e6b89..cf08e32 100644 --- a/deploy/README.md +++ b/deploy/README.md @@ -1,51 +1,65 @@ # Deploy -Everything needed to run Musify, in development and in production, from the -same set of compose files — no wrapper scripts, just `docker compose`. +Everything needed to run Musify itself, in development and in production, +from the same set of compose files, no wrapper scripts, just `docker +compose`. Postgres, Zitadel, SeaweedFS, RabbitMQ and Jaeger are **not** +started here: they're shared infrastructure that lives in the separate +`Infrastructure` repository and has to already be running (see its own +README). `../../Infrastructure` below is just an example path, assuming +that repo happens to be checked out as a sibling of this one. Adjust it to +wherever you actually cloned it. | File | Role | |---|---| -| `compose.yml` | Every service, including the one-shot bootstrap jobs. No published ports, no domains, no environment-specific values. | -| `compose.dev.yml` | Development overlay: publishes the ports to the host, adds pgAdmin. | -| `compose.prod.yml` | Production overlay: adds the nginx edge. | +| `compose.yml` | Musify's own services, including the one-shot bootstrap jobs. No published ports, no domains, no environment-specific values. | +| `compose.dev.yml` | Development overlay: publishes Musify's ports to the host. | +| `compose.prod.yml` | Production overlay: adds the nginx edge and joins Infrastructure's `infra-net` network. | | `.env` / `.env.prod` | The values compose interpolates. Not versioned; templates in `.env.example` / `.env.prod.example`. | -The four applications (`api`, `worker`, `gateway`, `web-player`) sit behind the -`apps` compose profile, so a plain `up` starts infrastructure only — the usual -development setup, with the apps running from the IDE. +The four applications (`api`, `worker`, `gateway`, `web-player`) sit behind +the `apps` compose profile, so a plain `up` starts only the one-shot +bootstrap jobs, the usual development setup, with the apps running from the +IDE. ### The one-shot bootstrap jobs -Three services in `compose.yml` run once and exit, so a plain `docker compose +Four services in `compose.yml` run once and exit, so a plain `docker compose up -d` leaves the stack fully ready with no manual steps: | Service | Does | |---|---| +| `wait-for-postgres` | Waits for Infrastructure's Postgres to accept connections. Replaces an in-file health-check `depends_on`, which only works within the same compose file. | | `keys-init` | Generates the RS256 stream-ticket key pair in `./keys`, if it is not there already. | | `migrate` | Applies the EF Core migrations (built from `backend/Musify.Migrator.Dockerfile`). | -| `zitadel-init` | Creates the Musify project and its two OIDC apps in Zitadel (or reuses them), then writes `AUTH_CLIENT_ID` / `WEB_CLIENT_ID` / `WEB_CLIENT_SECRET` into the env file. | +| `seaweedfs-init` | Creates Musify's own bucket on the shared SeaweedFS, if it doesn't exist yet. | +| `zitadel-init` | Creates the Musify project and its OIDC apps in the shared Zitadel (or reuses them), then writes `AUTH_CLIENT_ID` / `WEB_CLIENT_ID` / `WEB_CLIENT_SECRET` / `NATIVE_CLIENT_ID` into the env file. | -`api` and `worker` wait for `migrate`; `api` and `gateway` wait for -`keys-init` (`depends_on: condition: service_completed_successfully`). They do -**not** wait for `zitadel-init` — compose interpolates `${AUTH_CLIENT_ID}` once, -when a command starts, so a container that starts in the *same* `up` as -`zitadel-init` would still get the empty value it had before provisioning ran. -That's why bringing up the apps is a second command: by the time you run it, +`api` and `worker` wait for `migrate`; `api` waits for `keys-init` +(`depends_on: condition: service_completed_successfully`). Nothing waits on +`zitadel-init`, `seaweedfs-init` or the shared infrastructure services +directly. Compose interpolates `${AUTH_CLIENT_ID}` once, when a command +starts, so a container that started in the *same* `up` as `zitadel-init` +would still get the empty value it had before provisioning ran. That's why +bringing up the apps is a second command: by the time you run it, `zitadel-init` has already finished and the env file has real values. --- ## Development +Start the separate `Infrastructure` repository first, from wherever you +cloned it (see its own README for the exact command). Then, from this +repository's root: + ```sh docker compose -f deploy/compose.yml -f deploy/compose.dev.yml up -d docker compose -f deploy/compose.yml -f deploy/compose.dev.yml --profile apps up -d --build ``` -The first command starts the infrastructure and runs the bootstrap jobs -(keys, migrations, Zitadel project/apps). The second builds and runs `api`, -`worker`, `gateway` and `web-player` in Docker too — skip it to run the apps -from the IDE instead (the usual dev loop): +The first command starts Musify's own bootstrap jobs (Postgres wait, keys, +migrations, the bucket, the Zitadel project/apps). The second builds and runs +`api`, `worker`, `gateway` and `web-player` in Docker too. Skip it to run the +apps from the IDE instead (the usual dev loop): ```sh dotnet run --project backend/Hosts/Musify.Api # :5111 @@ -55,53 +69,55 @@ npm --prefix web-player run dev # :5173 ``` When running the apps from the IDE, `Musify.Api` reads `Authentication:*` -from user-secrets and `web-player` reads its own `web-player/.env` — neither -looks at `deploy/.env`. Copy the three values `zitadel-init` wrote -(`AUTH_CLIENT_ID`, `WEB_CLIENT_ID`, `WEB_CLIENT_SECRET`) into -`dotnet user-secrets set Authentication:ClientId --project -backend/Hosts/Musify.Api` and into `web-player/.env` once; they stay valid -until you wipe the Zitadel volume. - -Add `--profile tools` to also start pgAdmin. +from user-secrets and `web-player` reads its own `web-player/.env`, neither +looks at `deploy/.env`. Copy the four values `zitadel-init` wrote +(`AUTH_CLIENT_ID`, `WEB_CLIENT_ID`, `WEB_CLIENT_SECRET`, `NATIVE_CLIENT_ID`) +into `dotnet user-secrets set Authentication:ClientId --project +backend/Hosts/Musify.Api` and into `web-player/.env` once. They stay valid +until Infrastructure's Zitadel volume is wiped. ### Ports | Service | Port | Used by | |---|---|---| -| PostgreSQL | `59000` → 5432 | API + Worker (EF Core) and Zitadel | -| Zitadel | `8080` | OIDC/OAuth2, console at `/ui/console` | -| RabbitMQ | `5672` / `15672` | MassTransit / management UI | -| SeaweedFS | `8333` / `8888` / `9333` | S3 / filer (→ gateway) / master | -| Jaeger | `16686` / `4317` / `4318` | traces UI / OTLP gRPC / OTLP HTTP | -| pgAdmin (`--profile tools`) | `5050` | Postgres UI | -| API, gateway, web player (`--profile apps`) | `5111` / `8081` / `3000` | | +| Musify.Api | `5111` | `/scalar/v1`, `/openapi/v1.json` | +| Musify.StreamingGateway | `8081` | `/health`, `/media/...` | +| web-player | `5173` (`npm run dev`) or `3000` (container) | | -Host requirements: .NET 10 SDK, Node 22, Docker Desktop. (ffmpeg is only -needed on the host if you run the Worker outside Docker — inside Docker its -image already has it.) +Postgres, Zitadel, SeaweedFS, RabbitMQ and Jaeger ports come from +Infrastructure. See its own README for that table. + +Host requirements: .NET 10 SDK, Node 22, Docker Desktop. ffmpeg is only +needed on the host if you run the Worker outside Docker; inside Docker its +image already has it. ### Why `host.docker.internal` `PUBLIC_AUTH_URL` and `S3_PUBLIC_URL` point at `host.docker.internal`, which resolves to the same address from the browser, from apps running on the host and from inside the containers. That keeps the OIDC token issuer and the -presigned S3 URLs valid whichever way the apps are running. Zitadel routes by -the `Host` header, so the domain has to be identical everywhere; changing it -requires wiping the Postgres/Zitadel volumes and starting over (`docker -compose -f compose.yml -f compose.dev.yml down -v`, then `up -d` again). +presigned S3 URLs valid whichever way the apps are running, and is the same +address Infrastructure's own dev ports are published on. ### Resetting ```sh docker compose -f deploy/compose.yml -f deploy/compose.dev.yml down # stop, keep data -docker compose -f deploy/compose.yml -f deploy/compose.dev.yml down -v # stop, wipe volumes +docker compose -f deploy/compose.yml -f deploy/compose.dev.yml down -v # stop, wipe Musify's own volumes ``` +This only touches Musify's own containers and volumes. Resetting Zitadel or +SeaweedFS is done from the `Infrastructure` repo instead. + --- ## Production -On a server with a domain, from the repository root: +On a server with a domain: + +1. Bring up the `Infrastructure` repo first, with its own `.env.prod` filled + in. It creates the `infra-net` network this stack joins. +2. From this repository's root: ```sh cp deploy/.env.prod.example deploy/.env.prod @@ -110,31 +126,29 @@ cp deploy/.env.prod.example deploy/.env.prod Fill it in: 1. Replace every `example.com` with your domain and every `CHANGE_ME` with a - random value (`openssl rand -hex 24`; the Zitadel masterkey needs exactly - 32 characters). `chmod 600 deploy/.env.prod`. -2. Drop **`deploy/nginx/certs/origin.pem` and `origin.key`** — the origin + random value (`openssl rand -hex 24`). `chmod 600 deploy/.env.prod`. +2. Drop **`deploy/nginx/certs/origin.pem` and `origin.key`**, the origin certificate nginx serves. The stack assumes Cloudflare in *Full (strict)* mode in front of it, so a Cloudflare Origin Certificate is enough; any certificate valid for the five hostnames works. 3. Point `example.com`, `www`, `api`, `auth`, `stream` and `s3` at the server, - proxied/TLS-terminated the same way. + proxied and TLS-terminated the same way. -Then, same two commands as development, with the prod env file and overlay: +Then: ```sh docker compose --env-file deploy/.env.prod -f deploy/compose.yml -f deploy/compose.prod.yml up -d docker compose --env-file deploy/.env.prod -f deploy/compose.yml -f deploy/compose.prod.yml --profile apps up -d --build ``` -The first command starts the infrastructure, the nginx edge and the bootstrap -jobs; the second builds and starts the applications, already wired to the -OIDC client ids the first one produced. Both are idempotent — run them again -to deploy a new version (`docker compose ... --profile apps up -d --build` -alone is enough for an update once the stack already exists; it reuses the -infra containers and only rebuilds and restarts the apps). +The first command starts Musify's bootstrap jobs and the nginx edge; the +second builds and starts the applications, already wired to the OIDC client +ids the first one produced. Both are idempotent, so running them again +deploys a new version (`... --profile apps up -d --build` alone is enough for +an update once the stack already exists). -Only nginx publishes ports (80/443); everything else is reachable only on the -internal compose network. +Only nginx publishes ports (80/443). Every other Musify service, and all of +Infrastructure's, is reachable only on the internal `infra-net` network. --- @@ -142,17 +156,16 @@ internal compose network. ``` deploy/ -├─ compose.yml # all services, incl. keys-init / migrate / zitadel-init -├─ compose.dev.yml # dev overlay: host ports, pgAdmin -├─ compose.prod.yml # prod overlay: nginx edge +├─ compose.yml # Musify's own services: wait-for-postgres, keys-init, +│ # migrate, seaweedfs-init, zitadel-init, api/worker/gateway/web-player +├─ compose.dev.yml # dev overlay: host ports +├─ compose.prod.yml # prod overlay: nginx edge, joins infra-net ├─ .env.example / .env.prod.example ├─ keys/ # RS256 stream-ticket key pair (generated) -├─ nginx/ -│ ├─ templates/default.conf.template # vhosts, rendered with the domain by envsubst -│ ├─ snippets/proxy.conf # proxy headers shared by every vhost -│ └─ certs/ # origin.pem + origin.key (not versioned) -└─ zitadel/ - └─ .output/ # PAT written by Zitadel on init (not versioned) +└─ nginx/ + ├─ templates/default.conf.template # vhosts, rendered with the domain by envsubst + ├─ snippets/proxy.conf # proxy headers shared by every vhost + └─ certs/ # origin.pem + origin.key (not versioned) ``` The application images are built from `backend/Musify.*.Dockerfile` (build @@ -161,18 +174,15 @@ and `web-player/Dockerfile`. ## Notes -- **S3 credentials** are rendered into the SeaweedFS identity file from - `S3_ACCESS_KEY` / `S3_SECRET_KEY` when the container starts, so they cannot - drift from the ones the apps are given. +- **S3 credentials, the Postgres admin user, the RabbitMQ user and the + Zitadel masterkey** all live in Infrastructure's own `.env`/`.env.prod` now. + This project's env files only duplicate the values that have to match (see + the comments in `.env.example`). - **Zitadel** runs `start-from-init`, which is idempotent: the same command works on an empty volume and on an existing one. -- **`latest` Zitadel images** require the standalone Login UI v2 container, - which this stack does not run. `zitadel-init` turns that requirement off so - the built-in `/ui/login` is used. - **`AppSettings*.json` are PascalCase** in `Musify.Api` and `Musify.Worker`; on case-sensitive Linux the host looks for `appsettings*.json`, so their Dockerfiles create lowercase copies at publish time. -- **Re-running `zitadel-init` or `migrate`** is safe: the former reuses the - existing project/apps by name (regenerating the web app's secret only if it - is missing from the env file), the latter is a normal `dotnet ef database - update`, a no-op once the schema is current. +- **Re-running `zitadel-init`, `seaweedfs-init` or `migrate`** is safe: they + reuse the existing project/apps/bucket by name, and `migrate` is a normal + `dotnet ef database update`, a no-op once the schema is current. diff --git a/deploy/compose.dev.yml b/deploy/compose.dev.yml index 519c313..80e25d6 100644 --- a/deploy/compose.dev.yml +++ b/deploy/compose.dev.yml @@ -1,67 +1,36 @@ name: musify-dev -# Development overlay: publishes the infrastructure ports to the host so the -# .NET apps and the web player can run from the IDE, and adds pgAdmin. +# Development overlay: publishes Musify's own ports to the host so the .NET +# apps and the web player can run from the IDE. # # docker compose -f deploy/compose.yml -f deploy/compose.dev.yml up -d # docker compose -f deploy/compose.yml -f deploy/compose.dev.yml --profile apps up -d --build # -# `host.docker.internal` (see ZITADEL_EXTERNAL_DOMAIN / *_PUBLIC_URL in .env) -# resolves to the same address from the browser, from apps on the host and from -# inside the containers, so the OIDC issuer and the presigned S3 URLs are valid -# in both modes. `extra_hosts` gives Linux containers that same name. +# Postgres/Zitadel/SeaweedFS/RabbitMQ/Jaeger ports come from Infrastructure's +# own compose.dev.yml, not from here. See ../../../Infrastructure/README.md. +# `host.docker.internal` resolves to the same address from the browser, from +# apps on the host and from inside these containers, which is what makes the +# OIDC issuer and the presigned S3 URLs valid in both run modes. `extra_hosts` +# gives Linux containers that same name. x-host-alias: &host-alias - - "${ZITADEL_EXTERNAL_DOMAIN}:host-gateway" + - "host.docker.internal:host-gateway" services: - postgres: - ports: - - "${POSTGRES_HOST_PORT}:5432" + wait-for-postgres: + extra_hosts: *host-alias - zitadel: - ports: - - "${ZITADEL_EXTERNAL_PORT}:8080" - - rabbitmq: - ports: - - "5672:5672" # AMQP -> MassTransit - - "15672:15672" # management UI - - seaweedfs: - ports: - - "9333:9333" # master / admin UI - - "8888:8888" # filer -> StreamingGateway - - "8333:8333" # S3 API -> API, Worker and presigned uploads - - jaeger: - ports: - - "16686:16686" # UI - - "4317:4317" # OTLP gRPC - - "4318:4318" # OTLP HTTP + seaweedfs-init: + extra_hosts: *host-alias # zitadel-init (compose.yml) edits whichever env file is mounted here. zitadel-init: + extra_hosts: *host-alias volumes: - ./.env:/workspace/.env - # Optional Postgres UI: docker compose ... --profile tools up -d - pgadmin: - profiles: [tools] - image: dpage/pgadmin4:latest - container_name: musify-pgadmin - environment: - PGADMIN_DEFAULT_EMAIL: ${PGADMIN_DEFAULT_EMAIL} - PGADMIN_DEFAULT_PASSWORD: ${PGADMIN_DEFAULT_PASSWORD} - PGADMIN_CONFIG_SERVER_MODE: "False" - depends_on: - postgres: - condition: service_healthy - ports: - - "${PGADMIN_HOST_PORT}:80" - volumes: - - pgadmin_data:/var/lib/pgadmin - restart: unless-stopped + migrate: + extra_hosts: *host-alias api: ports: @@ -80,6 +49,3 @@ services: ports: - "3000:3000" extra_hosts: *host-alias - -volumes: - pgadmin_data: diff --git a/deploy/compose.prod.yml b/deploy/compose.prod.yml index 344b226..f4fd18d 100644 --- a/deploy/compose.prod.yml +++ b/deploy/compose.prod.yml @@ -1,16 +1,19 @@ name: musify-prod -# Production overlay: adds the nginx edge and the host-mounted secrets. -# Nothing but nginx publishes a port; every other service is reachable only on -# the internal `musify-prod_default` network. +# Production overlay: adds the nginx edge and joins Infrastructure's shared +# `infra-net` network so this project's containers can reach Postgres, +# Zitadel, SeaweedFS, RabbitMQ and Jaeger by container name. None of them +# run as part of this compose project anymore (see +# ../../../Infrastructure/README.md, which must already be up). Nothing but +# nginx publishes a port; every other service is reachable only on infra-net. # # docker compose --env-file deploy/.env.prod -f deploy/compose.yml -f deploy/compose.prod.yml up -d # docker compose --env-file deploy/.env.prod -f deploy/compose.yml -f deploy/compose.prod.yml --profile apps up -d --build -# -# The first command starts the infrastructure, the nginx edge and the -# one-shot bootstrap jobs (stream keys, migrations, Zitadel project/apps); the -# second builds and starts the applications, picking up the OIDC client ids -# the first command just wrote into deploy/.env.prod. See deploy/README.md. + +networks: + default: + name: infra-net + external: true services: # zitadel-init (compose.yml) edits whichever env file is mounted here. @@ -20,16 +23,13 @@ services: # Every vhost proxies through `set $upstream ...; proxy_pass $upstream;` # with the embedded Docker DNS resolver (see nginx/templates), so nginx - # does not need api/gateway/web-player to exist at its own startup — only + # does not need api/gateway/web-player to exist at its own startup, only # at request time. That is what lets it start in the FIRST command # (profile-gated services aren't part of that invocation yet) and still be # reachable at the auth domain for zitadel-init to provision through. nginx: image: nginx:1.27-alpine container_name: musify-nginx - depends_on: - zitadel: - condition: service_started environment: # The entrypoint renders /etc/nginx/templates/*.template with envsubst. # The filter keeps it from eating nginx's own $host / $remote_addr. diff --git a/deploy/compose.yml b/deploy/compose.yml index cdb5b6e..3a055aa 100644 --- a/deploy/compose.yml +++ b/deploy/compose.yml @@ -1,33 +1,32 @@ -# Musify — base stack, shared by development and production. +# Musify: base stack, shared by development and production. # -# This file publishes no host ports and hardcodes no domain: everything that -# differs between environments comes from the .env file (see .env.example) or -# from the overlay that is merged on top of it. +# Postgres, Zitadel, SeaweedFS, RabbitMQ and Jaeger are NOT part of this +# file. They're shared infrastructure that lives in ../../../Infrastructure +# and must already be running (see Infrastructure/README.md). This file only +# has what's specific to Musify: its own database, its own OIDC apps and +# storage bucket (created against the shared instances below), and its own +# apps. # # dev docker compose -f deploy/compose.yml -f deploy/compose.dev.yml up -d # prod docker compose --env-file deploy/.env.prod -f deploy/compose.yml -f deploy/compose.prod.yml up -d # -# The four application services sit behind the `apps` profile, so a plain `up` -# starts the infrastructure plus the one-shot bootstrap jobs (stream-ticket -# keys, database migrations, Zitadel project/apps) and stops there — the usual -# dev setup, with the apps running from the IDE. Add `--profile apps up -d -# --build` as a SECOND command to also build and run them in Docker: by then -# zitadel-init has already written the OIDC client ids into the env file, so -# the apps start with real values. See deploy/README.md. - -x-app-depends: &app-depends - postgres: - condition: service_healthy - rabbitmq: - condition: service_healthy - seaweedfs: - condition: service_healthy +# The four application services sit behind the `apps` profile, so a plain +# `up` runs only the one-shot bootstrap jobs (stream-ticket keys, database +# migrations, Zitadel project/apps, the storage bucket) and stops there. That's +# the usual dev setup, with the apps running from the IDE. Add `--profile apps +# up -d --build` as a SECOND command to also build and run them in Docker: by +# then zitadel-init has already written the OIDC client ids into the env +# file, so the apps start with real values. See deploy/README.md. # Backend settings that are identical for the API and the Worker. ASP.NET Core # maps `__` to `:`, so these override the matching AppSettings.json sections. +# POSTGRES_HOST / RABBITMQ_HOST / JAEGER_HOST differ between environments +# (host.docker.internal in dev, the container name over infra-net in prod, +# see .env.example / .env.prod.example), same as PUBLIC_AUTH_URL/S3_PUBLIC_URL +# already do. x-backend-env: &backend-env - Database__ConnectionString: "Host=postgres;Port=5432;Database=${POSTGRES_DB};Username=${POSTGRES_USER};Password=${POSTGRES_PASSWORD}" - MassTransit__Address: "rabbitmq://rabbitmq:5672" + Database__ConnectionString: "Host=${POSTGRES_HOST};Port=5432;Database=${POSTGRES_DB};Username=${POSTGRES_USER};Password=${POSTGRES_PASSWORD}" + MassTransit__Address: "rabbitmq://${RABBITMQ_HOST}:5672" MassTransit__Username: ${RABBITMQ_DEFAULT_USER} MassTransit__Password: ${RABBITMQ_DEFAULT_PASS} # The public S3 URL, not the internal one: the same address is used to sign @@ -38,128 +37,35 @@ x-backend-env: &backend-env InfrastructureStorage__ForcePathStyle: "true" InfrastructureStorage__UseHttp: ${S3_USE_HTTP} ApplicationStorage__Bucket: ${S3_BUCKET} - OpenTelemetry__OtlpEndpoint: "http://jaeger:4317" + OpenTelemetry__OtlpEndpoint: "http://${JAEGER_HOST}:4317" services: - # --- Infrastructure ------------------------------------------------------- + # --- Shared-infrastructure bootstrap --------------------------------------- - postgres: + # Waits for Infrastructure's Postgres to accept connections before `migrate` + # runs. This replaces a `depends_on: postgres: condition: service_healthy`: + # that only works within the same compose file, and Postgres now lives in a + # separate one (../../../Infrastructure). + wait-for-postgres: image: postgres:17-alpine - container_name: musify-postgres - environment: - POSTGRES_USER: ${POSTGRES_USER} - POSTGRES_PASSWORD: ${POSTGRES_PASSWORD} - POSTGRES_DB: ${POSTGRES_DB} - volumes: - - postgres_data:/var/lib/postgresql/data - healthcheck: - test: ["CMD-SHELL", "pg_isready -U ${POSTGRES_USER} -d ${POSTGRES_DB}"] - interval: 10s - timeout: 5s - retries: 10 - restart: unless-stopped - - # OIDC provider. `start-from-init` runs init + setup (both idempotent) and - # then starts, so the same command works on a cold and on a warm volume. - # It reuses the Postgres above: the admin user creates the `zitadel` database - # and role on first start. - zitadel: - image: ghcr.io/zitadel/zitadel:latest - container_name: musify-zitadel - command: start-from-init --masterkeyFromEnv --tlsMode disabled - environment: - ZITADEL_MASTERKEY: ${ZITADEL_MASTERKEY} - # The external domain/port forms the token issuer and must be identical - # for the browser, the apps and the containers (Zitadel routes by Host). - ZITADEL_EXTERNALSECURE: ${ZITADEL_EXTERNAL_SECURE} - ZITADEL_EXTERNALDOMAIN: ${ZITADEL_EXTERNAL_DOMAIN} - ZITADEL_EXTERNALPORT: ${ZITADEL_EXTERNAL_PORT} - ZITADEL_TLS_ENABLED: "false" - ZITADEL_DATABASE_POSTGRES_HOST: postgres - ZITADEL_DATABASE_POSTGRES_PORT: 5432 - ZITADEL_DATABASE_POSTGRES_DATABASE: zitadel - ZITADEL_DATABASE_POSTGRES_USER_USERNAME: zitadel - ZITADEL_DATABASE_POSTGRES_USER_PASSWORD: ${ZITADEL_DATABASE_PASSWORD} - ZITADEL_DATABASE_POSTGRES_USER_SSL_MODE: disable - ZITADEL_DATABASE_POSTGRES_ADMIN_USERNAME: ${POSTGRES_USER} - ZITADEL_DATABASE_POSTGRES_ADMIN_PASSWORD: ${POSTGRES_PASSWORD} - ZITADEL_DATABASE_POSTGRES_ADMIN_SSL_MODE: disable - # Initial console admin. Login: @zitadel. - ZITADEL_FIRSTINSTANCE_ORG_HUMAN_USERNAME: ${ZITADEL_ADMIN_USERNAME} - ZITADEL_FIRSTINSTANCE_ORG_HUMAN_PASSWORD: ${ZITADEL_ADMIN_PASSWORD} - ZITADEL_FIRSTINSTANCE_ORG_HUMAN_PASSWORDCHANGEREQUIRED: "false" - # Service account + PAT written on init; the provisioning script uses it - # to create the Musify project and its OIDC apps. - ZITADEL_FIRSTINSTANCE_ORG_MACHINE_MACHINE_USERNAME: zitadel-admin-sa - ZITADEL_FIRSTINSTANCE_ORG_MACHINE_MACHINE_NAME: Admin Service Account - ZITADEL_FIRSTINSTANCE_ORG_MACHINE_PAT_EXPIRATIONDATE: "2099-01-01T00:00:00Z" - ZITADEL_FIRSTINSTANCE_PATPATH: /output/admin-sa.pat - depends_on: - postgres: - condition: service_healthy - volumes: - - ./zitadel/.output:/output - restart: unless-stopped - - rabbitmq: - image: rabbitmq:4-management-alpine - container_name: musify-rabbitmq - environment: - RABBITMQ_DEFAULT_USER: ${RABBITMQ_DEFAULT_USER} - RABBITMQ_DEFAULT_PASS: ${RABBITMQ_DEFAULT_PASS} - volumes: - - rabbitmq_data:/var/lib/rabbitmq - healthcheck: - test: ["CMD", "rabbitmq-diagnostics", "-q", "ping"] - interval: 10s - timeout: 5s - retries: 10 - restart: unless-stopped - - # All-in-one object store: master + volume + filer + S3 gateway. - # The S3 identity file is rendered from the environment on start so the - # credentials can never drift from the ones handed to the apps. - seaweedfs: - image: chrislusf/seaweedfs:latest - container_name: musify-seaweedfs + container_name: musify-wait-for-postgres entrypoint: ["/bin/sh", "-c"] command: - | - cat > /tmp/s3.conf <<'JSON' - { - "identities": [ - { - "name": "musify", - "credentials": [{ "accessKey": "${S3_ACCESS_KEY}", "secretKey": "${S3_SECRET_KEY}" }], - "actions": ["Admin", "Read", "Write"] - } - ] - } - JSON - exec weed server \ - -ip.bind=0.0.0.0 \ - -dir=/data \ - -master.volumeSizeLimitMB=1024 \ - -volume.max=0 \ - -filer \ - -s3 \ - -s3.config=/tmp/s3.conf - volumes: - - seaweedfs_data:/data - healthcheck: - test: ["CMD", "wget", "-q", "-O", "-", "http://localhost:9333/cluster/healthz"] - interval: 10s - timeout: 5s - retries: 10 - restart: unless-stopped + set -e + deadline=$$(( $$(date +%s) + 60 )) + until pg_isready -h ${POSTGRES_HOST} -p 5432 -U ${POSTGRES_USER} >/dev/null 2>&1; do + [ "$$(date +%s)" -lt "$$deadline" ] || { echo "postgres did not respond at ${POSTGRES_HOST}:5432. Is Infrastructure running?" >&2; exit 1; } + sleep 2 + done + restart: "no" - # One-shot: SeaweedFS does not auto-create buckets. + # One-shot: SeaweedFS does not auto-create buckets. `aws s3 ls` doubles as + # the readiness probe, since seaweedfs-init no longer has an in-file + # `depends_on: seaweedfs: condition: service_healthy` to wait for. seaweedfs-init: image: amazon/aws-cli:latest container_name: musify-seaweedfs-init - depends_on: - seaweedfs: - condition: service_healthy environment: AWS_ACCESS_KEY_ID: ${S3_ACCESS_KEY} AWS_SECRET_ACCESS_KEY: ${S3_SECRET_KEY} @@ -167,13 +73,19 @@ services: entrypoint: ["/bin/sh", "-c"] command: - | - aws --endpoint-url http://seaweedfs:8333 s3 mb s3://${S3_BUCKET} 2>/dev/null || true - aws --endpoint-url http://seaweedfs:8333 s3 ls + set -e + deadline=$$(( $$(date +%s) + 60 )) + until aws --endpoint-url ${S3_PUBLIC_URL} s3 ls >/dev/null 2>&1; do + [ "$$(date +%s)" -lt "$$deadline" ] || { echo "seaweedfs did not respond at ${S3_PUBLIC_URL}. Is Infrastructure running?" >&2; exit 1; } + sleep 2 + done + aws --endpoint-url ${S3_PUBLIC_URL} s3 mb s3://${S3_BUCKET} 2>/dev/null || true + aws --endpoint-url ${S3_PUBLIC_URL} s3 ls restart: "no" # One-shot: generates the RS256 key pair used to sign (API) and verify # (Gateway) stream tickets, if it is not already in ./keys. No host tools - # required — openssl runs inside the container. + # required, since openssl runs inside the container. keys-init: image: alpine:3.20 container_name: musify-keys-init @@ -203,30 +115,27 @@ services: dockerfile: Musify.Migrator.Dockerfile container_name: musify-migrate depends_on: - postgres: - condition: service_healthy + wait-for-postgres: + condition: service_completed_successfully environment: - Database__ConnectionString: "Host=postgres;Port=5432;Database=${POSTGRES_DB};Username=${POSTGRES_USER};Password=${POSTGRES_PASSWORD}" + Database__ConnectionString: "Host=${POSTGRES_HOST};Port=5432;Database=${POSTGRES_DB};Username=${POSTGRES_USER};Password=${POSTGRES_PASSWORD}" restart: "no" - # One-shot: creates the Musify project and its two OIDC apps in Zitadel (or - # reuses them if they already exist), then writes AUTH_CLIENT_ID, - # WEB_CLIENT_ID and WEB_CLIENT_SECRET back into the env file mounted at - # /workspace/.env (see compose.dev.yml / compose.prod.yml) — so a later - # `--profile apps up` picks up real values instead of empty ones. + # One-shot: creates the Musify project and its two OIDC apps in the shared + # Zitadel (or reuses them if they already exist), then writes + # AUTH_CLIENT_ID, WEB_CLIENT_ID and WEB_CLIENT_SECRET back into the env file + # mounted at /workspace/.env (see compose.dev.yml / compose.prod.yml), so a + # later `--profile apps up` picks up real values instead of empty ones. zitadel-init: image: alpine:3.20 container_name: musify-zitadel-init - depends_on: - zitadel: - condition: service_started environment: PUBLIC_AUTH_URL: ${PUBLIC_AUTH_URL} PUBLIC_API_URL: ${PUBLIC_API_URL} PUBLIC_WEB_URL: ${PUBLIC_WEB_URL} ZITADEL_DEV_MODE: ${ZITADEL_DEV_MODE:-false} volumes: - - ./zitadel/.output:/output:ro + - ${ZITADEL_ADMIN_PAT_DIR}:/output:ro entrypoint: ["/bin/sh", "-c"] command: - | @@ -240,7 +149,7 @@ services: fi # The env file may have CRLF line endings (edited on Windows), so - # strip \r on read and drop it on write — otherwise a "blank" value + # strip \r on read and drop it on write, otherwise a "blank" value # is actually a lone \r, and `[ -z ... ]` lies. env_get() { grep -E "^$$2=" "$$1" 2>/dev/null | tail -n1 | cut -d= -f2- | tr -d '\r'; } env_set() { @@ -257,7 +166,7 @@ services: echo "==> waiting for the Zitadel service-account PAT" deadline=$$(( $$(date +%s) + 180 )) while [ ! -f /output/admin-sa.pat ]; do - [ "$$(date +%s)" -lt "$$deadline" ] || { echo "PAT not found, check 'docker logs musify-zitadel'" >&2; exit 1; } + [ "$$(date +%s)" -lt "$$deadline" ] || { echo "PAT not found under /output. Is Infrastructure's Zitadel running? Check 'docker logs infra-zitadel'." >&2; exit 1; } sleep 3 done PAT=$$(tr -d '\r\n' wired into $$ENV_FILE" restart: "no" - # OTLP collector + trace UI (MassTransit / EF Core / HTTP spans). - jaeger: - image: jaegertracing/all-in-one:latest - container_name: musify-jaeger - environment: - COLLECTOR_OTLP_ENABLED: "true" - restart: unless-stopped - # --- Applications (profile: apps) ----------------------------------------- api: @@ -404,7 +305,6 @@ services: dockerfile: Musify.Api.Dockerfile container_name: musify-api depends_on: - <<: *app-depends migrate: condition: service_completed_successfully keys-init: @@ -439,7 +339,6 @@ services: dockerfile: Musify.Worker.Dockerfile container_name: musify-worker depends_on: - <<: *app-depends migrate: condition: service_completed_successfully environment: @@ -458,15 +357,13 @@ services: dockerfile: Musify.StreamingGateway.Dockerfile container_name: musify-gateway depends_on: - seaweedfs: - condition: service_healthy keys-init: condition: service_completed_successfully environment: ASPNETCORE_ENVIRONMENT: ${DOTNET_ENVIRONMENT} ASPNETCORE_URLS: http://+:8081 StreamTicket__PublicKeyPath: /keys/stream_public.pem - ReverseProxy__Clusters__seaweed-filer__Destinations__filer__Address: "http://seaweedfs:8888/" + ReverseProxy__Clusters__seaweed-filer__Destinations__filer__Address: "http://${SEAWEEDFS_HOST}:8888/" ReverseProxy__Routes__media__Transforms__0__PathPattern: "/buckets/${S3_BUCKET}/{**rest}" Cors__AllowedOrigins__0: ${PUBLIC_WEB_URL} volumes: @@ -495,10 +392,5 @@ services: API_BASE_URL: http://api:5111 # Presigned PUTs are signed for the public S3 host; the container sends # them straight to SeaweedFS instead of going back out through the edge. - S3_UPLOAD_ENDPOINT: http://seaweedfs:8333 + S3_UPLOAD_ENDPOINT: ${S3_INTERNAL_URL} restart: unless-stopped - -volumes: - postgres_data: - rabbitmq_data: - seaweedfs_data: diff --git a/docs/README.md b/docs/README.md index 80547bf..772dc1c 100644 --- a/docs/README.md +++ b/docs/README.md @@ -34,6 +34,11 @@ this way**. Login: Client ◄──► Zitadel (:8080, OIDC) → Musify.Api validates the JWT ``` +PostgreSQL, Zitadel, SeaweedFS and RabbitMQ are not run by this repository. +They're shared infrastructure, provided by the separate `Infrastructure` +repository, that this stack connects to the same way any other project +does. + ## Stack - **.NET 10**, C#. Minimal APIs (`Musify.Api`), Worker Service (`Musify.Worker`), ASP.NET + YARP (`Musify.StreamingGateway`). @@ -48,4 +53,6 @@ this way**. A single repo: `backend/` (the six .NET projects), `web-player/` (the SvelteKit web client), `deploy/` (Docker Compose stacks and scripts), and -`docs/`. +`docs/`. The shared infrastructure (Postgres, Zitadel, SeaweedFS, RabbitMQ, +Jaeger) lives in the separate `Infrastructure` repository, a sibling of this +one, and is not part of this layout. diff --git a/docs/architecture.md b/docs/architecture.md index 5f04818..3af19ac 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -15,10 +15,10 @@ | **Domain/Application/Infrastructure** | Domain + use cases + shared infrastructure (EF, S3, MassTransit, ffmpeg) | Logic shared between the API and the Worker | | **StreamingGateway** | Reverse proxy (YARP) that validates the ticket and serves the audio (`.m4a`) from SeaweedFS | Scales with byte traffic independently of the API, and keeps SeaweedFS private behind it | | **Worker** | Consumes MassTransit events: transcodes audio to `.m4a` and generates thumbnails | Keeps CPU/IO-heavy work off the HTTP request path | -| **PostgreSQL** | Metadata (users, tracks, playlists, upload intents) | n/a | -| **RabbitMQ** | Event queue + transactional outbox | Decouples creation from processing | -| **SeaweedFS** | Object storage (S3 + filer) | Stores originals and derivatives (`.m4a` audio, thumbnails) | -| **Zitadel** | Identity (OIDC/OAuth2) | Login and JWT issuance | +| **PostgreSQL** *(shared, runs in the separate `Infrastructure` repo)* | Metadata (users, tracks, playlists, upload intents) | n/a | +| **RabbitMQ** *(shared, runs in Infrastructure)* | Event queue + transactional outbox | Decouples creation from processing | +| **SeaweedFS** *(shared, runs in Infrastructure)* | Object storage (S3 + filer) | Stores originals and derivatives (`.m4a` audio, thumbnails) | +| **Zitadel** *(shared, runs in Infrastructure)* | Identity (OIDC/OAuth2) | Login and JWT issuance | ## Main flows diff --git a/docs/development.md b/docs/development.md index 3a15444..3775245 100644 --- a/docs/development.md +++ b/docs/development.md @@ -5,15 +5,19 @@ and prod) live in [deploy/README.md](../deploy/README.md). ## Startup +Bring up the separate `Infrastructure` repository first (Postgres, Zitadel, +SeaweedFS, RabbitMQ, Jaeger, see its own README for the exact command), then: + ```sh docker compose -f deploy/compose.yml -f deploy/compose.dev.yml up -d ``` -That one command brings up the infrastructure and runs the one-shot bootstrap -jobs: generates the RS256 stream-ticket keys in `deploy/keys/`, applies the -EF Core migrations, and provisions Zitadel (project + OIDC apps), writing the -client ids into `deploy/.env`. Full details, including how to run the apps in -Docker with `--profile apps`, are in [deploy/README.md](../deploy/README.md). +That runs Musify's own one-shot bootstrap jobs: it generates the RS256 +stream-ticket keys in `deploy/keys/`, applies the EF Core migrations, creates +Musify's storage bucket, and provisions Zitadel (project + OIDC apps), +writing the client ids into `deploy/.env`. Full details, including how to run +the apps in Docker with `--profile apps`, are in +[deploy/README.md](../deploy/README.md). ## Applications @@ -31,16 +35,15 @@ npm --prefix web-player run dev # :5173 | Musify.Api | 5111 (`/scalar/v1`, `/openapi/v1.json`) | | Musify.StreamingGateway | 8081 (`/health`, `/media/...`) | | web-player | 5173 (`npm run dev`) or 3000 (container) | -| SeaweedFS S3 / filer / master | 8333 / 8888 / 9333 | -| Zitadel | 8080 | -| PostgreSQL | 59000 (→ 5432 in the container) | -| RabbitMQ AMQP / management | 5672 / 15672 | -| Jaeger UI / OTLP | 16686 / 4317 | -| pgAdmin (`-Tools`) | 5050 | -DB: `musify_db`, user `postgres`/`postgres`. S3 bucket: `webapi-storage` -(credentials `admin_access_key`/`admin_secret_key`). All of these come from -`deploy/.env`. +Postgres, Zitadel, SeaweedFS, RabbitMQ and Jaeger are not run by this repo. +Their ports are documented in the separate `Infrastructure` repository's own +README. + +DB: `musify_db`, user `postgres`/`postgres` on Infrastructure's shared +Postgres. S3 bucket: `webapi-storage` (credentials +`admin_access_key`/`admin_secret_key`, also Infrastructure's). All of these +come from `deploy/.env`. ## Migrations diff --git a/docs/projects.md b/docs/projects.md index 0891c33..2549e0e 100644 --- a/docs/projects.md +++ b/docs/projects.md @@ -69,6 +69,8 @@ real Postgres + SeaweedFS), and `Musify.StreamingGateway.Tests`. Details in ## Other folders -- **`deploy/`**: self-contained Docker Compose stacks (dev and prod). See +- **`deploy/`**: Musify's own Docker Compose stacks (dev and prod). The + shared infrastructure (Postgres, Zitadel, SeaweedFS, RabbitMQ, Jaeger) + lives in the separate `Infrastructure` repo instead. See [../deploy/README.md](../deploy/README.md). - **`docs/`**: this documentation.