From cf8ac706b6aafd9fbbf7d343cd972a2d2acea242 Mon Sep 17 00:00:00 2001 From: XShine555 Date: Wed, 9 Sep 2026 15:08:34 +0200 Subject: [PATCH] Replace deploy scripts with self-contained compose bootstrap jobs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Removes every wrapper script under deploy/ (up.ps1, up.sh, scripts/*, zitadel/provision.*) and folds their logic into three one-shot services in compose.yml, following the pattern already used by seaweedfs-init: keys-init generates the RS256 stream-ticket key pair migrate applies the EF Core migrations (backend/Musify.Migrator.Dockerfile) zitadel-init creates the Musify project + OIDC apps, writes the client ids back into the active env file Deploying is now two plain `docker compose` commands, no scripts to run or chmod: docker compose -f deploy/compose.yml -f deploy/compose.dev.yml up -d docker compose -f deploy/compose.yml -f deploy/compose.dev.yml --profile apps up -d --build (swap compose.dev.yml for compose.prod.yml + --env-file deploy/.env.prod for production). The first command brings up infra and lets the bootstrap jobs run to completion; the second builds and starts the apps, which by then read real OIDC values instead of empty ones — compose interpolates ${VAR} once per invocation, so this can't collapse into a single command without the apps reading their own config file at container start instead of via compose env (out of scope here). Also: - DatabaseDesignTimeFactory now also reads Database__ConnectionString from the environment, and DesignSettings.json is optional, so the new migrate container works without user-secrets or a config file excluded from the Docker build context by .dockerignore. - nginx's depends_on no longer lists the apps-profile services: it does not need them at its own startup (every vhost resolves its upstream at request time via the embedded Docker resolver), and requiring them made `compose.prod.yml up -d` (without --profile apps) an invalid project. - env_get/env_set in zitadel-init strip \r: the mounted env file can have CRLF line endings on Windows, which otherwise makes an empty-looking value non-empty to `[ -z ... ]`. .env.example/.env.prod.example are pinned to LF via .gitattributes for the same reason. Also fixed: rewriting the bind-mounted env file in place instead of renaming a temp file over it (rename onto a bind-mounted inode fails with "Resource busy"). - Removed compose.share.yml and zitadel/.output-share/ — a "temporary" personal overlay that had leaked a real Zitadel service-account PAT into git history (committed in 70968b3). That token should be rotated. - Removed deploy/.tmp/ from .gitignore: nothing mounts or creates it anymore (it never actually did, either — no compose service referenced it, only the now-removed up.ps1 did). Verified end-to-end with Docker: a clean `down -v` + fresh `up -d` + `--profile apps up -d --build` on the dev stack brings up all four one-shot jobs successfully and all four apps healthy (api/gateway return 200, worker starts its consumers with no errors, web-player responds). Also verified compose.prod.yml starts nginx cleanly with dummy certs and no apps running, confirming the depends_on fix. --- .gitattributes | 7 + .gitignore | 1 - README.md | 20 +- .../Persistence/DatabaseDesignTimeFactory.cs | 3 +- backend/Musify.Migrator.Dockerfile | 25 ++ deploy/.env.example | 11 +- deploy/.env.prod.example | 7 +- deploy/README.md | 148 +++++++----- deploy/compose.dev.yml | 5 + deploy/compose.prod.yml | 27 ++- deploy/compose.share.yml | 15 -- deploy/compose.yml | 216 +++++++++++++++++- deploy/scripts/common.ps1 | 26 --- deploy/scripts/lib.sh | 31 --- deploy/scripts/migrate.ps1 | 15 -- deploy/scripts/migrate.sh | 28 --- deploy/scripts/stream-keys.ps1 | 28 --- deploy/scripts/stream-keys.sh | 21 -- deploy/up.ps1 | 145 ------------ deploy/up.sh | 103 --------- deploy/zitadel/.output-share/admin-sa.pat | 1 - deploy/zitadel/provision.ps1 | 193 ---------------- deploy/zitadel/provision.sh | 145 ------------ docs/development.md | 36 +-- web-player/.env.example | 5 +- web-player/README.md | 8 +- 26 files changed, 408 insertions(+), 862 deletions(-) create mode 100644 backend/Musify.Migrator.Dockerfile delete mode 100644 deploy/compose.share.yml delete mode 100644 deploy/scripts/common.ps1 delete mode 100644 deploy/scripts/lib.sh delete mode 100644 deploy/scripts/migrate.ps1 delete mode 100644 deploy/scripts/migrate.sh delete mode 100644 deploy/scripts/stream-keys.ps1 delete mode 100644 deploy/scripts/stream-keys.sh delete mode 100644 deploy/up.ps1 delete mode 100644 deploy/up.sh delete mode 100644 deploy/zitadel/.output-share/admin-sa.pat delete mode 100644 deploy/zitadel/provision.ps1 delete mode 100644 deploy/zitadel/provision.sh diff --git a/.gitattributes b/.gitattributes index 9f3e779..da2c853 100644 --- a/.gitattributes +++ b/.gitattributes @@ -4,6 +4,13 @@ # Force LF for shell scripts *.sh text eol=lf +# Env templates get `cp`-ed straight into deploy/.env / deploy/.env.prod, +# which containers (e.g. zitadel-init in deploy/compose.yml) parse with +# plain shell text tools — keep them LF so a Windows checkout can't +# reintroduce CRLF into a value that then looks non-empty but isn't. +.env.example text eol=lf +.env.prod.example text eol=lf + # Treat common binaries as binary *.png binary *.jpg binary diff --git a/.gitignore b/.gitignore index 2d28ffc..151e9d4 100644 --- a/.gitignore +++ b/.gitignore @@ -27,4 +27,3 @@ deploy/keys/ deploy/secrets/ deploy/nginx/certs/ deploy/zitadel/.output/ -deploy/.tmp/ diff --git a/README.md b/README.md index 07424bb..7493a26 100644 --- a/README.md +++ b/README.md @@ -7,26 +7,32 @@ and stream them without the bytes ever going through the API. ``` backend/ .NET 10 — Domain / Application / Infrastructure / Api / Worker / StreamingGateway web-player/ SvelteKit 2 + Svelte 5 + Tailwind 4 web client -deploy/ Docker Compose stacks and the scripts that bootstrap them +deploy/ Docker Compose stacks — self-contained, no wrapper scripts docs/ What each piece does and why (in Spanish) ``` ## Run it -Development (Windows, Docker Desktop): +Development (Docker Desktop): -```powershell -./deploy/up.ps1 # infrastructure; the apps run from the IDE -./deploy/up.ps1 -Apps # everything in Docker +```sh +docker compose -f deploy/compose.yml -f deploy/compose.dev.yml up -d ``` +Infrastructure plus stream-ticket keys, database migrations and Zitadel OIDC +provisioning — all as one-shot jobs baked into the compose file. Apps run from +the IDE by default; add `--profile apps up -d --build` as a second command to +run them in Docker instead. + Production, on a server with a domain: ```sh -./deploy/up.sh # fill in deploy/.env.prod and the TLS certs when it asks +cp deploy/.env.prod.example deploy/.env.prod # fill in the domain and secrets +docker compose --env-file deploy/.env.prod -f deploy/compose.yml -f deploy/compose.prod.yml up -d +docker compose --env-file deploy/.env.prod -f deploy/compose.yml -f deploy/compose.prod.yml --profile apps up -d --build ``` -Both are idempotent and print what they did. See +Idempotent — run the same commands again to deploy an update. See [deploy/README.md](deploy/README.md) for the details, the port map and the production prerequisites. diff --git a/backend/Core/Musify.Infrastructure/Persistence/DatabaseDesignTimeFactory.cs b/backend/Core/Musify.Infrastructure/Persistence/DatabaseDesignTimeFactory.cs index 18f38a9..2b3cb7d 100644 --- a/backend/Core/Musify.Infrastructure/Persistence/DatabaseDesignTimeFactory.cs +++ b/backend/Core/Musify.Infrastructure/Persistence/DatabaseDesignTimeFactory.cs @@ -11,8 +11,9 @@ namespace Musify.Infrastructure.Persistence { var configuration = new ConfigurationBuilder() .SetBasePath(Directory.GetCurrentDirectory()) - .AddJsonFile("DesignSettings.json") + .AddJsonFile("DesignSettings.json", optional: true) .AddUserSecrets() + .AddEnvironmentVariables() .Build(); var databaseConfiguration = configuration diff --git a/backend/Musify.Migrator.Dockerfile b/backend/Musify.Migrator.Dockerfile new file mode 100644 index 0000000..64fba65 --- /dev/null +++ b/backend/Musify.Migrator.Dockerfile @@ -0,0 +1,25 @@ +# Build context: backend/ (docker compose -f deploy/compose.yml up migrate) +# +# Applies EF Core migrations, then exits. Musify.Infrastructure is both the +# migrations and the design-time startup project: it owns the +# IDesignTimeDbContextFactory and the EFCore.Design package. The connection +# string comes from the Database__ConnectionString env var (compose sets it), +# read via DatabaseDesignTimeFactory's AddEnvironmentVariables(). +FROM mcr.microsoft.com/dotnet/sdk:10.0 +WORKDIR /src + +COPY Directory.Build.props Directory.Packages.props nuget.config ./ +COPY Core/Musify.Domain/Musify.Domain.csproj Core/Musify.Domain/ +COPY Core/Musify.Application/Musify.Application.csproj Core/Musify.Application/ +COPY Core/Musify.Infrastructure/Musify.Infrastructure.csproj Core/Musify.Infrastructure/ +RUN dotnet restore Core/Musify.Infrastructure/Musify.Infrastructure.csproj + +COPY Core/Musify.Domain Core/Musify.Domain/ +COPY Core/Musify.Application Core/Musify.Application/ +COPY Core/Musify.Infrastructure Core/Musify.Infrastructure/ + +RUN dotnet tool install --global dotnet-ef --version 10.0.8 +ENV PATH="$PATH:/root/.dotnet/tools" + +WORKDIR /src/Core/Musify.Infrastructure +ENTRYPOINT ["dotnet", "ef", "database", "update", "--context", "Database"] diff --git a/deploy/.env.example b/deploy/.env.example index c329987..a5c82e3 100644 --- a/deploy/.env.example +++ b/deploy/.env.example @@ -1,5 +1,7 @@ # Development configuration for deploy/compose.yml + compose.dev.yml. -# Copy to deploy/.env (up.ps1 does it for you) and adjust. Not versioned. +# Copy to deploy/.env and adjust. Not versioned. +# +# cp deploy/.env.example deploy/.env DOTNET_ENVIRONMENT=Development @@ -22,7 +24,7 @@ POSTGRES_HOST_PORT=59000 # --- Zitadel (OIDC) ---------------------------------------------------------- # The masterkey MUST be exactly 32 characters. The external domain/port form # the token issuer, so they have to match PUBLIC_AUTH_URL. Changing them needs -# a re-init: ./up.ps1 -Destroy; ./up.ps1 +# a re-init: docker compose -f deploy/compose.yml -f deploy/compose.dev.yml down -v ZITADEL_MASTERKEY=MasterkeyNeedsToHave32Characters ZITADEL_EXTERNAL_DOMAIN=host.docker.internal ZITADEL_EXTERNAL_PORT=8080 @@ -31,8 +33,11 @@ ZITADEL_DATABASE_PASSWORD=zitadel # Console login: @zitadel. ZITADEL_ADMIN_USERNAME=admin ZITADEL_ADMIN_PASSWORD=Password1! +# Allows the OIDC apps to use plain-http redirect URIs (dev only). +ZITADEL_DEV_MODE=true -# OIDC clients, filled in by deploy/zitadel/provision.ps1. +# OIDC clients: the zitadel-init one-shot service (see compose.yml) fills +# these in automatically on `docker compose up -d`. AUTH_CLIENT_ID= WEB_CLIENT_ID= WEB_CLIENT_SECRET= diff --git a/deploy/.env.prod.example b/deploy/.env.prod.example index 6c603c3..7bcb394 100644 --- a/deploy/.env.prod.example +++ b/deploy/.env.prod.example @@ -1,5 +1,5 @@ # Production configuration for deploy/compose.yml + compose.prod.yml. -# Copy to deploy/.env.prod (up.sh does it for you), then: +# Copy to deploy/.env.prod, then: # # 1. replace every `example.com` with your domain, # 2. replace every CHANGE_ME with a strong random value @@ -35,8 +35,11 @@ ZITADEL_DATABASE_PASSWORD=CHANGE_ME # Console login: @zitadel. ZITADEL_ADMIN_USERNAME=admin ZITADEL_ADMIN_PASSWORD=CHANGE_ME +# Keep OIDC apps strict about https redirect URIs in production. +ZITADEL_DEV_MODE=false -# OIDC clients, filled in by deploy/zitadel/provision.sh. +# OIDC clients: the zitadel-init one-shot service (see compose.yml) fills +# these in automatically on `docker compose up -d`. AUTH_CLIENT_ID= WEB_CLIENT_ID= WEB_CLIENT_SECRET= diff --git a/deploy/README.md b/deploy/README.md index a809ef5..bcc2a5f 100644 --- a/deploy/README.md +++ b/deploy/README.md @@ -1,44 +1,68 @@ # Deploy Everything needed to run Musify, in development and in production, from the -same set of compose files. +same set of compose files — no wrapper scripts, just `docker compose`. | File | Role | |---|---| -| `compose.yml` | Every service. No published ports, no domains, no environment-specific values. | +| `compose.yml` | Every service, including the one-shot bootstrap jobs. No published ports, no domains, no environment-specific values. | | `compose.dev.yml` | Development overlay: publishes the ports to the host, adds pgAdmin. | | `compose.prod.yml` | Production overlay: adds the nginx edge and the mounted secrets. | | `.env` / `.env.prod` | The values compose interpolates. Not versioned; templates in `.env.example` / `.env.prod.example`. | -| `up.ps1` | Development orchestrator (Windows). | -| `up.sh` | Production orchestrator (the server). | The four applications (`api`, `worker`, `gateway`, `web-player`) sit behind the `apps` compose profile, so a plain `up` starts infrastructure only — the usual development setup, with the apps running from the IDE. +### The one-shot bootstrap jobs + +Three services in `compose.yml` run once and exit, so a plain `docker compose +up -d` leaves the stack fully ready with no manual steps: + +| Service | Does | +|---|---| +| `keys-init` | Generates the RS256 stream-ticket key pair in `./keys`, if it is not there already. | +| `migrate` | Applies the EF Core migrations (built from `backend/Musify.Migrator.Dockerfile`). | +| `zitadel-init` | Creates the Musify project and its two OIDC apps in Zitadel (or reuses them), then writes `AUTH_CLIENT_ID` / `WEB_CLIENT_ID` / `WEB_CLIENT_SECRET` into the env file. | + +`api` and `worker` wait for `migrate`; `api` and `gateway` wait for +`keys-init` (`depends_on: condition: service_completed_successfully`). They do +**not** wait for `zitadel-init` — compose interpolates `${AUTH_CLIENT_ID}` once, +when a command starts, so a container that starts in the *same* `up` as +`zitadel-init` would still get the empty value it had before provisioning ran. +That's why bringing up the apps is a second command: by the time you run it, +`zitadel-init` has already finished and the env file has real values. + --- ## Development -```powershell -./deploy/up.ps1 # infrastructure only, apps from the IDE -./deploy/up.ps1 -Apps # build and run the apps in Docker too +```sh +docker compose -f deploy/compose.yml -f deploy/compose.dev.yml up -d +docker compose -f deploy/compose.yml -f deploy/compose.dev.yml --profile apps up -d --build ``` -`up.ps1` does the whole bootstrap, with no manual steps afterwards: +The first command starts the infrastructure and runs the bootstrap jobs +(keys, migrations, Zitadel project/apps). The second builds and runs `api`, +`worker`, `gateway` and `web-player` in Docker too — skip it to run the apps +from the IDE instead (the usual dev loop): -1. Creates `deploy/.env` from `.env.example` if it is missing. -2. Generates the RS256 stream-ticket keys in `deploy/keys/` and the Worker - scratch directory `deploy/.tmp/`. -3. Starts the stack and waits for Postgres. -4. Applies the EF Core migrations. -5. Provisions Zitadel: project, API OIDC app and web player OIDC app, then - writes the client ids into `deploy/.env`, `Musify.Api` user-secrets and - `web-player/.env`. -6. Prints a summary with every URL and credential. +```sh +dotnet run --project backend/Hosts/Musify.Api # :5111 +dotnet run --project backend/Hosts/Musify.StreamingGateway # :8081 +dotnet run --project backend/Hosts/Musify.Worker # background +npm --prefix web-player run dev # :5173 +``` -Other switches: `-Tools` (pgAdmin), `-SkipMigrations`, `-Down` (stop, keep -data), `-Destroy` (stop and wipe the volumes). +When running the apps from the IDE, `Musify.Api` reads `Authentication:*` +from user-secrets and `web-player` reads its own `web-player/.env` — neither +looks at `deploy/.env`. Copy the three values `zitadel-init` wrote +(`AUTH_CLIENT_ID`, `WEB_CLIENT_ID`, `WEB_CLIENT_SECRET`) into +`dotnet user-secrets set Authentication:ClientId --project +backend/Hosts/Musify.Api` and into `web-player/.env` once; they stay valid +until you wipe the Zitadel volume. + +Add `--profile tools` to also start pgAdmin. ### Ports @@ -49,20 +73,12 @@ data), `-Destroy` (stop and wipe the volumes). | RabbitMQ | `5672` / `15672` | MassTransit / management UI | | SeaweedFS | `8333` / `8888` / `9333` | S3 / filer (→ gateway) / master | | Jaeger | `16686` / `4317` / `4318` | traces UI / OTLP gRPC / OTLP HTTP | -| pgAdmin (`-Tools`) | `5050` | Postgres UI | -| API, gateway, web player (`-Apps`) | `5111` / `8081` / `3000` | | +| pgAdmin (`--profile tools`) | `5050` | Postgres UI | +| API, gateway, web player (`--profile apps`) | `5111` / `8081` / `3000` | | -### Running the apps from the IDE - -```powershell -dotnet run --project backend/Hosts/Musify.Api # :5111 -dotnet run --project backend/Hosts/Musify.StreamingGateway # :8081 -dotnet run --project backend/Hosts/Musify.Worker # background -npm --prefix web-player run dev # :5173 -``` - -Host requirements: .NET 10 SDK, Node 22, ffmpeg and yt-dlp on PATH (the Worker -transcodes and downloads audio), Docker Desktop. +Host requirements: .NET 10 SDK, Node 22, Docker Desktop. (ffmpeg and yt-dlp +are only needed on the host if you run the Worker outside Docker — inside +Docker its image already has them.) ### Why `host.docker.internal` @@ -71,36 +87,51 @@ resolves to the same address from the browser, from apps running on the host and from inside the containers. That keeps the OIDC token issuer and the presigned S3 URLs valid whichever way the apps are running. Zitadel routes by the `Host` header, so the domain has to be identical everywhere; changing it -requires a re-init (`./up.ps1 -Destroy; ./up.ps1`). +requires wiping the Postgres/Zitadel volumes and starting over (`docker +compose -f compose.yml -f compose.dev.yml down -v`, then `up -d` again). + +### Resetting + +```sh +docker compose -f deploy/compose.yml -f deploy/compose.dev.yml down # stop, keep data +docker compose -f deploy/compose.yml -f deploy/compose.dev.yml down -v # stop, wipe volumes +``` --- ## Production -One command on the server, from the repository root: +On a server with a domain, from the repository root: ```sh -./deploy/up.sh +cp deploy/.env.prod.example deploy/.env.prod ``` -It refuses to start until the two things it cannot generate are in place: +Fill it in: -1. **`deploy/.env.prod`** — copied from `.env.prod.example` on the first run. - Replace every `example.com` with your domain and every `CHANGE_ME` with a - random value (`openssl rand -hex 24`; the Zitadel masterkey needs exactly 32 - characters). `chmod 600` it. -2. **`deploy/nginx/certs/origin.pem` and `origin.key`** — the origin +1. Replace every `example.com` with your domain and every `CHANGE_ME` with a + random value (`openssl rand -hex 24`; the Zitadel masterkey needs exactly + 32 characters). `chmod 600 deploy/.env.prod`. +2. Drop **`deploy/nginx/certs/origin.pem` and `origin.key`** — the origin certificate nginx serves. The stack assumes Cloudflare in *Full (strict)* mode in front of it, so a Cloudflare Origin Certificate is enough; any certificate valid for the five hostnames works. +3. Point `example.com`, `www`, `api`, `auth`, `stream` and `s3` at the server, + proxied/TLS-terminated the same way. -DNS: point `example.com`, `www`, `api`, `auth`, `stream` and `s3` at the server. +Then, same two commands as development, with the prod env file and overlay: -Then `up.sh` generates the stream keys, starts the infrastructure and the edge, -provisions Zitadel (it needs `curl` and `jq` on the server), applies the -migrations and finally builds and starts the applications. It is idempotent — -run it again to deploy a new version, or `./deploy/up.sh --no-build` to restart -without rebuilding. +```sh +docker compose --env-file deploy/.env.prod -f deploy/compose.yml -f deploy/compose.prod.yml up -d +docker compose --env-file deploy/.env.prod -f deploy/compose.yml -f deploy/compose.prod.yml --profile apps up -d --build +``` + +The first command starts the infrastructure, the nginx edge and the bootstrap +jobs; the second builds and starts the applications, already wired to the +OIDC client ids the first one produced. Both are idempotent — run them again +to deploy a new version (`docker compose ... --profile apps up -d --build` +alone is enough for an update once the stack already exists; it reuses the +infra containers and only rebuilds and restarts the apps). Only nginx publishes ports (80/443); everything else is reachable only on the internal compose network. @@ -118,30 +149,23 @@ empty if you do not have one. ``` deploy/ -├─ compose.yml # all services, environment-independent +├─ compose.yml # all services, incl. keys-init / migrate / zitadel-init ├─ compose.dev.yml # dev overlay: host ports, pgAdmin ├─ compose.prod.yml # prod overlay: nginx edge, secrets ├─ .env.example / .env.prod.example -├─ up.ps1 # dev orchestrator -├─ up.sh # prod orchestrator ├─ keys/ # RS256 stream-ticket key pair (generated) ├─ secrets/ # mounted into the Worker (prod, optional) ├─ nginx/ │ ├─ templates/default.conf.template # vhosts, rendered with the domain by envsubst │ ├─ snippets/proxy.conf # proxy headers shared by every vhost │ └─ certs/ # origin.pem + origin.key (not versioned) -├─ scripts/ -│ ├─ common.ps1 / lib.sh # shared helpers -│ ├─ stream-keys.ps1 / .sh # RS256 key pair generation -│ └─ migrate.ps1 / .sh # EF Core migrations └─ zitadel/ - ├─ provision.ps1 / .sh # project + OIDC apps, wires the client ids - └─ .output/ # PAT written by Zitadel on init (not versioned) + └─ .output/ # PAT written by Zitadel on init (not versioned) ``` The application images are built from `backend/Musify.*.Dockerfile` (build -context `backend/`, so restore can copy the central props and every csproj) and -`web-player/Dockerfile`. +context `backend/`, so restore can copy the central props and every csproj) +and `web-player/Dockerfile`. ## Notes @@ -151,8 +175,12 @@ context `backend/`, so restore can copy the central props and every csproj) and - **Zitadel** runs `start-from-init`, which is idempotent: the same command works on an empty volume and on an existing one. - **`latest` Zitadel images** require the standalone Login UI v2 container, - which this stack does not run. The provisioning scripts turn that requirement - off so the built-in `/ui/login` is used. + which this stack does not run. `zitadel-init` turns that requirement off so + the built-in `/ui/login` is used. - **`AppSettings*.json` are PascalCase** in `Musify.Api` and `Musify.Worker`; on case-sensitive Linux the host looks for `appsettings*.json`, so their Dockerfiles create lowercase copies at publish time. +- **Re-running `zitadel-init` or `migrate`** is safe: the former reuses the + existing project/apps by name (regenerating the web app's secret only if it + is missing from the env file), the latter is a normal `dotnet ef database + update`, a no-op once the schema is current. diff --git a/deploy/compose.dev.yml b/deploy/compose.dev.yml index 20f281d..519c313 100644 --- a/deploy/compose.dev.yml +++ b/deploy/compose.dev.yml @@ -40,6 +40,11 @@ services: - "4317:4317" # OTLP gRPC - "4318:4318" # OTLP HTTP + # zitadel-init (compose.yml) edits whichever env file is mounted here. + zitadel-init: + volumes: + - ./.env:/workspace/.env + # Optional Postgres UI: docker compose ... --profile tools up -d pgadmin: profiles: [tools] diff --git a/deploy/compose.prod.yml b/deploy/compose.prod.yml index c9ab174..b00ac3f 100644 --- a/deploy/compose.prod.yml +++ b/deploy/compose.prod.yml @@ -4,11 +4,13 @@ name: musify-prod # Nothing but nginx publishes a port; every other service is reachable only on # the internal `musify-prod_default` network. # -# docker compose --env-file deploy/.env.prod \ -# -f deploy/compose.yml -f deploy/compose.prod.yml --profile apps up -d --build +# docker compose --env-file deploy/.env.prod -f deploy/compose.yml -f deploy/compose.prod.yml up -d +# docker compose --env-file deploy/.env.prod -f deploy/compose.yml -f deploy/compose.prod.yml --profile apps up -d --build # -# Use ./up.sh, which also generates the stream keys, provisions Zitadel and -# applies the database migrations. +# The first command starts the infrastructure, the nginx edge and the +# one-shot bootstrap jobs (stream keys, migrations, Zitadel project/apps); the +# second builds and starts the applications, picking up the OIDC client ids +# the first command just wrote into deploy/.env.prod. See deploy/README.md. services: worker: @@ -16,14 +18,23 @@ services: volumes: - ./secrets:/secrets:ro + # zitadel-init (compose.yml) edits whichever env file is mounted here. + zitadel-init: + volumes: + - ./.env.prod:/workspace/.env + + # Every vhost proxies through `set $upstream ...; proxy_pass $upstream;` + # with the embedded Docker DNS resolver (see nginx/templates), so nginx + # does not need api/gateway/web-player to exist at its own startup — only + # at request time. That is what lets it start in the FIRST command + # (profile-gated services aren't part of that invocation yet) and still be + # reachable at the auth domain for zitadel-init to provision through. nginx: image: nginx:1.27-alpine container_name: musify-nginx depends_on: - - api - - gateway - - zitadel - - web-player + zitadel: + condition: service_started environment: # The entrypoint renders /etc/nginx/templates/*.template with envsubst. # The filter keeps it from eating nginx's own $host / $remote_addr. diff --git a/deploy/compose.share.yml b/deploy/compose.share.yml deleted file mode 100644 index 04e5f0b..0000000 --- a/deploy/compose.share.yml +++ /dev/null @@ -1,15 +0,0 @@ -name: musify-share - -# Temporary overlay for exposing the stack outside the LAN, on top of -# compose.dev.yml. Only difference from a normal dev run: Zitadel writes its -# init PAT to a separate host directory, so it never overwrites -# zitadel/.output (used by the musify-dev project's own Zitadel instance). -# -# docker compose --env-file deploy/.env.share \ -# -f deploy/compose.yml -f deploy/compose.dev.yml -f deploy/compose.share.yml \ -# --profile apps up -d --build api gateway web-player - -services: - zitadel: - volumes: - - ./zitadel/.output-share:/output diff --git a/deploy/compose.yml b/deploy/compose.yml index aede0ba..b79806b 100644 --- a/deploy/compose.yml +++ b/deploy/compose.yml @@ -8,9 +8,12 @@ # prod docker compose --env-file deploy/.env.prod -f deploy/compose.yml -f deploy/compose.prod.yml up -d # # The four application services sit behind the `apps` profile, so a plain `up` -# starts only the infrastructure (the usual dev setup: apps from the IDE). -# Add `--profile apps` to run them in Docker. Use ./up.ps1 (dev) or ./up.sh -# (prod) instead of calling compose by hand. +# starts the infrastructure plus the one-shot bootstrap jobs (stream-ticket +# keys, database migrations, Zitadel project/apps) and stops there — the usual +# dev setup, with the apps running from the IDE. Add `--profile apps up -d +# --build` as a SECOND command to also build and run them in Docker: by then +# zitadel-init has already written the OIDC client ids into the env file, so +# the apps start with real values. See deploy/README.md. x-app-depends: &app-depends postgres: @@ -168,6 +171,196 @@ services: aws --endpoint-url http://seaweedfs:8333 s3 ls restart: "no" + # One-shot: generates the RS256 key pair used to sign (API) and verify + # (Gateway) stream tickets, if it is not already in ./keys. No host tools + # required — openssl runs inside the container. + keys-init: + image: alpine:3.20 + container_name: musify-keys-init + entrypoint: ["/bin/sh", "-c"] + command: + - | + set -e + if [ -f /keys/stream_private.pem ] && [ -f /keys/stream_public.pem ]; then + echo "stream-ticket keys already present" + else + apk add --no-cache openssl >/dev/null + openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:2048 -out /keys/stream_private.pem + openssl rsa -in /keys/stream_private.pem -pubout -out /keys/stream_public.pem + chmod 644 /keys/stream_public.pem + chmod 600 /keys/stream_private.pem + echo "generated stream-ticket keys" + fi + volumes: + - ./keys:/keys + restart: "no" + + # One-shot: applies the EF Core migrations, then exits. See + # backend/Musify.Migrator.Dockerfile. + migrate: + build: + context: ../backend + dockerfile: Musify.Migrator.Dockerfile + container_name: musify-migrate + depends_on: + postgres: + condition: service_healthy + environment: + Database__ConnectionString: "Host=postgres;Port=5432;Database=${POSTGRES_DB};Username=${POSTGRES_USER};Password=${POSTGRES_PASSWORD}" + restart: "no" + + # One-shot: creates the Musify project and its two OIDC apps in Zitadel (or + # reuses them if they already exist), then writes AUTH_CLIENT_ID, + # WEB_CLIENT_ID and WEB_CLIENT_SECRET back into the env file mounted at + # /workspace/.env (see compose.dev.yml / compose.prod.yml) — so a later + # `--profile apps up` picks up real values instead of empty ones. + zitadel-init: + image: alpine:3.20 + container_name: musify-zitadel-init + depends_on: + zitadel: + condition: service_started + environment: + PUBLIC_AUTH_URL: ${PUBLIC_AUTH_URL} + PUBLIC_API_URL: ${PUBLIC_API_URL} + PUBLIC_WEB_URL: ${PUBLIC_WEB_URL} + ZITADEL_DEV_MODE: ${ZITADEL_DEV_MODE:-false} + volumes: + - ./zitadel/.output:/output:ro + entrypoint: ["/bin/sh", "-c"] + command: + - | + set -e + apk add --no-cache curl jq >/dev/null + + ENV_FILE=/workspace/.env + if [ ! -f "$$ENV_FILE" ]; then + echo "no env file mounted at $$ENV_FILE, skipping" + exit 0 + fi + + # The env file may have CRLF line endings (edited on Windows), so + # strip \r on read and drop it on write — otherwise a "blank" value + # is actually a lone \r, and `[ -z ... ]` lies. + env_get() { grep -E "^$$2=" "$$1" 2>/dev/null | tail -n1 | cut -d= -f2- | tr -d '\r'; } + env_set() { + # $$1 is a bind-mounted file: rewrite its content in place rather + # than renaming a temp file over it (rename onto a bind-mounted + # inode fails with "Resource busy"). + if grep -qE "^$$2=" "$$1"; then + awk -v k="$$2" -v v="$$3" '{ sub(/\r$$/, "") } index($$0,k "=")==1{print k "=" v; next} {print}' "$$1" >"$$1.tmp" && cat "$$1.tmp" >"$$1" && rm -f "$$1.tmp" + else + printf '%s=%s\n' "$$2" "$$3" >>"$$1" + fi + } + + echo "==> waiting for the Zitadel service-account PAT" + deadline=$$(( $$(date +%s) + 180 )) + while [ ! -f /output/admin-sa.pat ]; do + [ "$$(date +%s)" -lt "$$deadline" ] || { echo "PAT not found, check 'docker logs musify-zitadel'" >&2; exit 1; } + sleep 3 + done + PAT=$$(tr -d '\r\n' waiting for Zitadel to answer at $$PUBLIC_AUTH_URL" + deadline=$$(( $$(date +%s) + 120 )) + until curl -fsS "$$PUBLIC_AUTH_URL/.well-known/openid-configuration" >/dev/null 2>&1; do + [ "$$(date +%s)" -lt "$$deadline" ] || { echo "Zitadel did not respond at $$PUBLIC_AUTH_URL" >&2; exit 1; } + sleep 3 + done + + api_call() { + if [ $$# -ge 3 ]; then + curl -fsS -X "$$1" "$$MGMT$$2" -H "Authorization: Bearer $$PAT" -H "Content-Type: application/json" -d "$$3" + else + curl -fsS -X "$$1" "$$MGMT$$2" -H "Authorization: Bearer $$PAT" + fi + } + name_query() { printf '{"queries":[{"nameQuery":{"name":"%s","method":"TEXT_QUERY_METHOD_EQUALS"}}]}' "$$1"; } + + # `latest` images require the standalone Login UI v2 container, which + # this stack does not run; opting out keeps the built-in /ui/login. + curl -fsS -X PUT "$$PUBLIC_AUTH_URL/v2/features/instance" \ + -H "Authorization: Bearer $$PAT" -H "Content-Type: application/json" \ + -d '{"loginV2":{"required":false}}' >/dev/null 2>&1 || echo "warning: could not disable Login UI v2" + + echo "==> project" + PROJECT_NAME=Musify + PROJECT_ID=$$(api_call POST /projects/_search "$$(name_query "$$PROJECT_NAME")" | jq -r --arg n "$$PROJECT_NAME" '.result // [] | map(select(.name==$$n)) | first | .id // empty') + if [ -z "$$PROJECT_ID" ]; then + PROJECT_ID=$$(api_call POST /projects "$$(printf '{"name":"%s"}' "$$PROJECT_NAME")" | jq -r '.id') + echo " created (id $$PROJECT_ID)" + else + echo " reused (id $$PROJECT_ID)" + fi + + find_app() { api_call POST "/projects/$$PROJECT_ID/apps/_search" "$$(name_query "$$1")" | jq -r --arg n "$$1" '.result // [] | map(select(.name==$$n)) | first | .id // empty'; } + + echo "==> API app" + API_APP_NAME="Musify API (Scalar)" + API_APP_ID=$$(find_app "$$API_APP_NAME") + if [ -n "$$API_APP_ID" ]; then + API_CLIENT_ID=$$(api_call GET "/projects/$$PROJECT_ID/apps/$$API_APP_ID" | jq -r '.app.oidcConfig.clientId') + echo " reused (clientId $$API_CLIENT_ID)" + else + API_CLIENT_ID=$$(api_call POST "/projects/$$PROJECT_ID/apps/oidc" "$$(jq -n --arg name "$$API_APP_NAME" --arg api "$$PUBLIC_API_URL" '{ + name: $$name, + redirectUris: [($$api + "/scalar/"), ($$api + "/scalar/oauth2-redirect.html")], + postLogoutRedirectUris: [($$api + "/scalar/")], + responseTypes: ["OIDC_RESPONSE_TYPE_CODE"], + grantTypes: ["OIDC_GRANT_TYPE_AUTHORIZATION_CODE"], + appType: "OIDC_APP_TYPE_USER_AGENT", + authMethodType: "OIDC_AUTH_METHOD_TYPE_NONE", + version: "OIDC_VERSION_1_0", + devMode: ($$ENV.ZITADEL_DEV_MODE == "true"), + accessTokenType: "OIDC_TOKEN_TYPE_JWT", + accessTokenRoleAssertion: true, + idTokenRoleAssertion: true, + idTokenUserinfoAssertion: true + }')" | jq -r '.clientId') + echo " created (clientId $$API_CLIENT_ID)" + fi + + echo "==> web app" + WEB_APP_NAME="Musify Web" + WEB_APP_ID=$$(find_app "$$WEB_APP_NAME") + WEB_CLIENT_SECRET=$$(env_get "$$ENV_FILE" WEB_CLIENT_SECRET) + if [ -n "$$WEB_APP_ID" ]; then + WEB_CLIENT_ID=$$(api_call GET "/projects/$$PROJECT_ID/apps/$$WEB_APP_ID" | jq -r '.app.oidcConfig.clientId') + if [ -z "$$WEB_CLIENT_SECRET" ]; then + WEB_CLIENT_SECRET=$$(api_call POST "/projects/$$PROJECT_ID/apps/$$WEB_APP_ID/oidc_config/_generate_client_secret" '{}' | jq -r '.clientSecret') + echo " reused (clientId $$WEB_CLIENT_ID, secret regenerated)" + else + echo " reused (clientId $$WEB_CLIENT_ID)" + fi + else + CREATED=$$(api_call POST "/projects/$$PROJECT_ID/apps/oidc" "$$(jq -n --arg name "$$WEB_APP_NAME" --arg web "$$PUBLIC_WEB_URL" '{ + name: $$name, + redirectUris: [($$web + "/auth/callback")], + postLogoutRedirectUris: [($$web + "/")], + responseTypes: ["OIDC_RESPONSE_TYPE_CODE"], + grantTypes: ["OIDC_GRANT_TYPE_AUTHORIZATION_CODE", "OIDC_GRANT_TYPE_REFRESH_TOKEN"], + appType: "OIDC_APP_TYPE_WEB", + authMethodType: "OIDC_AUTH_METHOD_TYPE_BASIC", + version: "OIDC_VERSION_1_0", + devMode: ($$ENV.ZITADEL_DEV_MODE == "true"), + accessTokenType: "OIDC_TOKEN_TYPE_JWT", + accessTokenRoleAssertion: true, + idTokenRoleAssertion: true, + idTokenUserinfoAssertion: true + }')") + WEB_CLIENT_ID=$$(echo "$$CREATED" | jq -r '.clientId') + WEB_CLIENT_SECRET=$$(echo "$$CREATED" | jq -r '.clientSecret') + echo " created (clientId $$WEB_CLIENT_ID)" + fi + + env_set "$$ENV_FILE" AUTH_CLIENT_ID "$$API_CLIENT_ID" + env_set "$$ENV_FILE" WEB_CLIENT_ID "$$WEB_CLIENT_ID" + env_set "$$ENV_FILE" WEB_CLIENT_SECRET "$$WEB_CLIENT_SECRET" + echo "==> wired into $$ENV_FILE" + restart: "no" + # OTLP collector + trace UI (MassTransit / EF Core / HTTP spans). jaeger: image: jaegertracing/all-in-one:latest @@ -184,7 +377,12 @@ services: context: ../backend dockerfile: Musify.Api.Dockerfile container_name: musify-api - depends_on: *app-depends + depends_on: + <<: *app-depends + migrate: + condition: service_completed_successfully + keys-init: + condition: service_completed_successfully environment: <<: *backend-env ASPNETCORE_ENVIRONMENT: ${DOTNET_ENVIRONMENT} @@ -192,8 +390,7 @@ services: StreamGateway__PublicBaseUrl: ${PUBLIC_STREAM_URL} StreamTicket__PrivateKeyPath: /keys/stream_private.pem Workers__Routes__TemporaryFilesDirectory: /tmp/musify - # Client id and secret are written to the .env file by the Zitadel - # provisioning script (deploy/zitadel/provision.*). + # AUTH_CLIENT_ID is written to the env file by the zitadel-init service. Authentication__MetadataAddress: "${PUBLIC_AUTH_URL}/.well-known/openid-configuration" Authentication__IssuerAddress: ${PUBLIC_AUTH_URL} Authentication__ClientId: ${AUTH_CLIENT_ID} @@ -215,7 +412,10 @@ services: context: ../backend dockerfile: Musify.Worker.Dockerfile container_name: musify-worker - depends_on: *app-depends + depends_on: + <<: *app-depends + migrate: + condition: service_completed_successfully environment: <<: *backend-env DOTNET_ENVIRONMENT: ${DOTNET_ENVIRONMENT} @@ -235,6 +435,8 @@ services: depends_on: seaweedfs: condition: service_healthy + keys-init: + condition: service_completed_successfully environment: ASPNETCORE_ENVIRONMENT: ${DOTNET_ENVIRONMENT} ASPNETCORE_URLS: http://+:8081 diff --git a/deploy/scripts/common.ps1 b/deploy/scripts/common.ps1 deleted file mode 100644 index 8d2e659..0000000 --- a/deploy/scripts/common.ps1 +++ /dev/null @@ -1,26 +0,0 @@ -#requires -Version 5.1 -# Shared helpers for the development deploy scripts. - -function Write-Step([string]$Message) { Write-Host "`n==> $Message" -ForegroundColor Cyan } - -# Parses a .env file into an ordered hashtable (ignores comments and blank lines). -function Read-DotEnv([string]$Path) { - $values = [ordered]@{} - if (Test-Path $Path) { - Get-Content $Path | ForEach-Object { - if ($_ -match '^\s*([^#=\s][^=]*)=(.*)$') { $values[$matches[1].Trim()] = $matches[2].Trim() } - } - } - return $values -} - -# Sets (or appends) a key in a .env file, preserving the rest of the file. -function Set-DotEnvValue([string]$Path, [string]$Key, [string]$Value) { - $lines = @(Get-Content $Path) - if ($lines -match "^\s*$Key=") { - $lines = $lines -replace "^\s*$Key=.*", "$Key=$Value" - } else { - $lines += "$Key=$Value" - } - Set-Content -Path $Path -Value $lines -Encoding utf8 -} diff --git a/deploy/scripts/lib.sh b/deploy/scripts/lib.sh deleted file mode 100644 index a54d7e7..0000000 --- a/deploy/scripts/lib.sh +++ /dev/null @@ -1,31 +0,0 @@ -#!/bin/sh -# Shared helpers for the production deploy scripts. - -step() { printf '\n==> %s\n' "$1"; } -fail() { printf 'error: %s\n' "$1" >&2; exit 1; } - -require_command() { - command -v "$1" >/dev/null 2>&1 || fail "$1 is required but not installed." -} - -# env_get — prints the value, empty if the key is absent. -env_get() { - grep -E "^$2=" "$1" 2>/dev/null | tail -n 1 | cut -d= -f2- -} - -# env_set — replaces the key in place, appends if missing. -env_set() { - _file=$1; _key=$2; _value=$3 - _tmp=$(mktemp) - if grep -qE "^$_key=" "$_file"; then - awk -v k="$_key" -v v="$_value" ' - index($0, k "=") == 1 { print k "=" v; next } - { print } - ' "$_file" >"$_tmp" - else - cp "$_file" "$_tmp" - printf '%s=%s\n' "$_key" "$_value" >>"$_tmp" - fi - cat "$_tmp" >"$_file" - rm -f "$_tmp" -} diff --git a/deploy/scripts/migrate.ps1 b/deploy/scripts/migrate.ps1 deleted file mode 100644 index e4cef16..0000000 --- a/deploy/scripts/migrate.ps1 +++ /dev/null @@ -1,15 +0,0 @@ -#requires -Version 5.1 -# Applies EF Core migrations to the Musify database. -[CmdletBinding()] -param([Parameter(Mandatory)][string]$ConnectionString) - -$ErrorActionPreference = "Stop" -$infraProject = Resolve-Path (Join-Path $PSScriptRoot "..\..\backend\Core\Musify.Infrastructure") - -# The design-time factory reads the connection string from user-secrets. -dotnet user-secrets set "Database:ConnectionString" $ConnectionString --project $infraProject | Out-Null - -# Infrastructure is also the startup project: it owns the IDesignTimeDbContextFactory -# and the EFCore.Design package (Musify.Api does not reference Design). -dotnet ef database update --project $infraProject --startup-project $infraProject --context Database -if ($LASTEXITCODE -ne 0) { throw "EF Core migration failed" } diff --git a/deploy/scripts/migrate.sh b/deploy/scripts/migrate.sh deleted file mode 100644 index b0378b7..0000000 --- a/deploy/scripts/migrate.sh +++ /dev/null @@ -1,28 +0,0 @@ -#!/bin/sh -# Applies the EF Core migrations to the production database from a throwaway SDK -# container attached to the compose network (Postgres is not published to the host). -set -eu - -SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd) -DEPLOY_DIR=$(CDPATH= cd -- "$SCRIPT_DIR/.." && pwd) -BACKEND_DIR=$(CDPATH= cd -- "$DEPLOY_DIR/../backend" && pwd) -ENV_FILE="$DEPLOY_DIR/.env.prod" -NETWORK=${MUSIFY_NETWORK:-musify-prod_default} - -. "$SCRIPT_DIR/lib.sh" - -CONN="Host=postgres;Port=5432;Database=$(env_get "$ENV_FILE" POSTGRES_DB);Username=$(env_get "$ENV_FILE" POSTGRES_USER);Password=$(env_get "$ENV_FILE" POSTGRES_PASSWORD)" - -# Musify.Infrastructure is both the migrations and the startup project: it owns -# the IDesignTimeDbContextFactory and the EFCore.Design package. -docker run --rm \ - --network "$NETWORK" \ - -v "$BACKEND_DIR":/src \ - -w /src/Core/Musify.Infrastructure \ - -e CONN="$CONN" \ - mcr.microsoft.com/dotnet/sdk:10.0 \ - sh -c 'dotnet tool install --global dotnet-ef >/dev/null 2>&1 || true - export PATH=$PATH:/root/.dotnet/tools - dotnet restore Musify.Infrastructure.csproj - dotnet user-secrets set "Database:ConnectionString" "$CONN" >/dev/null - dotnet ef database update --context Database' diff --git a/deploy/scripts/stream-keys.ps1 b/deploy/scripts/stream-keys.ps1 deleted file mode 100644 index 50d260a..0000000 --- a/deploy/scripts/stream-keys.ps1 +++ /dev/null @@ -1,28 +0,0 @@ -#requires -Version 5.1 -# Generates the RS256 key pair used to sign (API) and verify (Gateway) stream tickets. -# No-op if both keys already exist. -[CmdletBinding()] -param([Parameter(Mandatory)][string]$KeysDir) - -$ErrorActionPreference = "Stop" -$privateKey = Join-Path $KeysDir "stream_private.pem" -$publicKey = Join-Path $KeysDir "stream_public.pem" - -if ((Test-Path $privateKey) -and (Test-Path $publicKey)) { - Write-Host "Stream-ticket keys already present in $KeysDir" - return -} - -if (-not (Test-Path $KeysDir)) { New-Item -ItemType Directory -Path $KeysDir | Out-Null } - -if (Get-Command openssl -ErrorAction SilentlyContinue) { - & openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:2048 -out $privateKey - & openssl rsa -in $privateKey -pubout -out $publicKey -} else { - $rsa = [System.Security.Cryptography.RSA]::Create(2048) - [System.IO.File]::WriteAllText($privateKey, $rsa.ExportPkcs8PrivateKeyPem()) - [System.IO.File]::WriteAllText($publicKey, $rsa.ExportSubjectPublicKeyInfoPem()) - $rsa.Dispose() -} - -Write-Host "Generated stream-ticket keys in $KeysDir" diff --git a/deploy/scripts/stream-keys.sh b/deploy/scripts/stream-keys.sh deleted file mode 100644 index 34f7ffa..0000000 --- a/deploy/scripts/stream-keys.sh +++ /dev/null @@ -1,21 +0,0 @@ -#!/bin/sh -# Generates the RS256 key pair used to sign (API) and verify (Gateway) stream -# tickets. No-op if both keys already exist. -set -eu - -KEYS_DIR=${1:?usage: stream-keys.sh } -PRIVATE_KEY="$KEYS_DIR/stream_private.pem" -PUBLIC_KEY="$KEYS_DIR/stream_public.pem" - -if [ -f "$PRIVATE_KEY" ] && [ -f "$PUBLIC_KEY" ]; then - echo "Stream-ticket keys already present in $KEYS_DIR" - exit 0 -fi - -mkdir -p "$KEYS_DIR" -openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:2048 -out "$PRIVATE_KEY" 2>/dev/null -openssl rsa -in "$PRIVATE_KEY" -pubout -out "$PUBLIC_KEY" 2>/dev/null -chmod 644 "$PUBLIC_KEY" -chmod 600 "$PRIVATE_KEY" - -echo "Generated stream-ticket keys in $KEYS_DIR" diff --git a/deploy/up.ps1 b/deploy/up.ps1 deleted file mode 100644 index 90c85b4..0000000 --- a/deploy/up.ps1 +++ /dev/null @@ -1,145 +0,0 @@ -#requires -Version 5.1 -<# -.SYNOPSIS - Brings up the Musify development environment. - -.DESCRIPTION - Starts the infrastructure stack, generates the stream-ticket keys, applies - the EF Core migrations and provisions Zitadel (project + OIDC apps). The - .NET apps and the web player then run from the IDE; add -Apps to run them - in Docker instead. - -.EXAMPLE - ./up.ps1 # Infrastructure only (apps from the IDE) - ./up.ps1 -Apps # Also build and run api, worker, gateway, web player - ./up.ps1 -Tools # Also start pgAdmin - ./up.ps1 -SkipMigrations # Skip the database migration step - ./up.ps1 -Down # Stop and remove containers (keep volumes) - ./up.ps1 -Destroy # Stop and remove containers + volumes (wipe data) -#> -[CmdletBinding()] -param( - [switch]$Apps, - [switch]$Tools, - [switch]$SkipMigrations, - [switch]$Down, - [switch]$Destroy -) - -$ErrorActionPreference = "Stop" -$deployDir = $PSScriptRoot -$compose = @( - "-f", (Join-Path $deployDir "compose.yml"), - "-f", (Join-Path $deployDir "compose.dev.yml") -) -$profiles = @("--profile", "apps", "--profile", "tools") - -. (Join-Path $deployDir "scripts\common.ps1") - -# Teardown ------------------------------------------------------------------- -if ($Down -or $Destroy) { - if ($Destroy) { - Write-Step "Stopping the stack and REMOVING volumes (data is wiped)" - docker compose @compose @profiles down -v --remove-orphans - # Drop the Zitadel init secrets so the next start re-initializes cleanly. - Get-ChildItem (Join-Path $deployDir "zitadel\.output") -File -ErrorAction SilentlyContinue | Remove-Item -Force - } else { - Write-Step "Stopping the stack (volumes are kept)" - docker compose @compose @profiles down --remove-orphans - } - return -} - -# Config --------------------------------------------------------------------- -$envFile = Join-Path $deployDir ".env" -if (-not (Test-Path $envFile)) { - Write-Step "Creating .env from .env.example" - Copy-Item (Join-Path $deployDir ".env.example") $envFile -} -$cfg = Read-DotEnv $envFile - -# Host directories the apps expect: RS256 keys (mounted into the api/gateway -# containers, read directly by the IDE runs) and the Worker scratch space. -& (Join-Path $deployDir "scripts\stream-keys.ps1") -KeysDir (Join-Path $deployDir "keys") -$tempDir = Join-Path $deployDir ".tmp" -if (-not (Test-Path $tempDir)) { New-Item -ItemType Directory -Path $tempDir | Out-Null } - -# Infrastructure ------------------------------------------------------------- -Write-Step "Starting the infrastructure stack" -$upArgs = @("up", "-d") -if ($Tools) { $upArgs = @("--profile", "tools") + $upArgs } -docker compose @compose @upArgs -if ($LASTEXITCODE -ne 0) { throw "docker compose up failed" } - -Write-Step "Waiting for Postgres to become healthy" -$deadline = (Get-Date).AddMinutes(2) -do { - Start-Sleep -Seconds 2 - $status = docker inspect --format '{{.State.Health.Status}}' musify-postgres 2>$null -} until ($status -eq "healthy" -or (Get-Date) -gt $deadline) -if ($status -ne "healthy") { throw "Postgres did not become healthy in time" } - -# Database migrations -------------------------------------------------------- -if ($SkipMigrations) { - Write-Host "`nSkipping migrations (-SkipMigrations)." -} else { - Write-Step "Applying EF Core migrations" - $conn = "Host=localhost;Port=$($cfg['POSTGRES_HOST_PORT']);Database=$($cfg['POSTGRES_DB']);Username=$($cfg['POSTGRES_USER']);Password=$($cfg['POSTGRES_PASSWORD'])" - & (Join-Path $deployDir "scripts\migrate.ps1") -ConnectionString $conn -} - -# Zitadel provisioning ------------------------------------------------------- -# Writes the client ids into .env, Musify.Api user-secrets and web-player/.env, -# so it has to run before the app containers start. -& (Join-Path $deployDir "zitadel\provision.ps1") - -# Applications --------------------------------------------------------------- -if ($Apps) { - Write-Step "Building and starting the applications" - docker compose @compose --profile apps up -d --build - if ($LASTEXITCODE -ne 0) { throw "docker compose up (apps) failed" } -} - -# Summary -------------------------------------------------------------------- -$cfg = Read-DotEnv $envFile -$authUrl = $cfg['PUBLIC_AUTH_URL'] -$appLines = if ($Apps) { -@" - Musify API $($cfg['PUBLIC_API_URL'])/scalar/v1 - Streaming gateway $($cfg['PUBLIC_STREAM_URL'])/health - Web player $($cfg['PUBLIC_WEB_URL']) -"@ -} else { -@" - Run the apps from the IDE (or ./up.ps1 -Apps to run them in Docker): - dotnet run --project backend/Hosts/Musify.Api # :5111 - dotnet run --project backend/Hosts/Musify.StreamingGateway # :8081 - dotnet run --project backend/Hosts/Musify.Worker # background - npm --prefix web-player run dev # :5173 -"@ -} - -Write-Host @" - -============================================================ - Musify development environment READY -============================================================ - - Service URL / port - ------------------ -------------------------------------- - PostgreSQL localhost:$($cfg['POSTGRES_HOST_PORT']) (db: $($cfg['POSTGRES_DB'])) - Zitadel console $authUrl/ui/console - RabbitMQ mgmt http://localhost:15672 ($($cfg['RABBITMQ_DEFAULT_USER'])/$($cfg['RABBITMQ_DEFAULT_PASS'])) - SeaweedFS S3 http://localhost:8333 (bucket: $($cfg['S3_BUCKET'])) - SeaweedFS filer http://localhost:8888 - SeaweedFS master http://localhost:9333 - Jaeger traces http://localhost:16686 (OTLP: localhost:4317) - pgAdmin (-Tools) http://localhost:$($cfg['PGADMIN_HOST_PORT']) ($($cfg['PGADMIN_DEFAULT_EMAIL'])/$($cfg['PGADMIN_DEFAULT_PASSWORD'])) - - Zitadel admin $($cfg['ZITADEL_ADMIN_USERNAME'])@zitadel.$($cfg['ZITADEL_EXTERNAL_DOMAIN']) / $($cfg['ZITADEL_ADMIN_PASSWORD']) - API client id $($cfg['AUTH_CLIENT_ID']) - Web client id $($cfg['WEB_CLIENT_ID']) - -$appLines -============================================================ -"@ -ForegroundColor Green diff --git a/deploy/up.sh b/deploy/up.sh deleted file mode 100644 index c9fbd05..0000000 --- a/deploy/up.sh +++ /dev/null @@ -1,103 +0,0 @@ -#!/bin/sh -# Brings up (or updates) the Musify production stack on the server. -# -# ./up.sh build + start everything, provision Zitadel, migrate -# ./up.sh --no-build same, reusing the images that are already built -# ./up.sh --down stop and remove the containers (volumes are kept) -# -# Everything it needs lives in deploy/.env.prod (see .env.prod.example) and in -# deploy/nginx/certs/ (origin.pem + origin.key). -set -eu - -DEPLOY_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd) -ENV_FILE="$DEPLOY_DIR/.env.prod" -INFRA="postgres zitadel rabbitmq seaweedfs seaweedfs-init jaeger" - -. "$DEPLOY_DIR/scripts/lib.sh" - -require_command docker -require_command openssl - -compose() { - docker compose --env-file "$ENV_FILE" \ - -f "$DEPLOY_DIR/compose.yml" -f "$DEPLOY_DIR/compose.prod.yml" \ - --profile apps "$@" -} - -BUILD=--build -for arg in "$@"; do - case "$arg" in - --no-build) BUILD= ;; - --down) step "Stopping the stack (volumes are kept)"; compose down --remove-orphans; exit 0 ;; - *) fail "unknown option: $arg" ;; - esac -done - -# Preflight ------------------------------------------------------------------ -if [ ! -f "$ENV_FILE" ]; then - cp "$DEPLOY_DIR/.env.prod.example" "$ENV_FILE" - chmod 600 "$ENV_FILE" - fail "created $ENV_FILE from the template — fill it in (domain + secrets) and run again." -fi - -if grep -q 'CHANGE_ME\|example\.com' "$ENV_FILE"; then - fail "$ENV_FILE still contains CHANGE_ME or example.com placeholders." -fi - -for cert in origin.pem origin.key; do - [ -f "$DEPLOY_DIR/nginx/certs/$cert" ] \ - || fail "missing nginx/certs/$cert (Cloudflare Origin Certificate)." -done - -mkdir -p "$DEPLOY_DIR/secrets" "$DEPLOY_DIR/keys" "$DEPLOY_DIR/zitadel/.output" -case "$(env_get "$ENV_FILE" YTDLP_ADDITIONAL_ARGUMENTS)" in - *--cookies*) - [ -f "$DEPLOY_DIR/secrets/youtube_cookies.txt" ] \ - || echo "warning: YTDLP_ADDITIONAL_ARGUMENTS expects secrets/youtube_cookies.txt, which is missing." ;; -esac - -step "Ensuring stream-ticket keys" -sh "$DEPLOY_DIR/scripts/stream-keys.sh" "$DEPLOY_DIR/keys" - -# Infrastructure ------------------------------------------------------------- -step "Starting the infrastructure" -# shellcheck disable=SC2086 -compose up -d $INFRA - -step "Waiting for Postgres" -deadline=$(( $(date +%s) + 120 )) -until [ "$(docker inspect --format '{{.State.Health.Status}}' musify-postgres 2>/dev/null)" = healthy ]; do - [ "$(date +%s)" -lt "$deadline" ] || fail "Postgres did not become healthy in time." - sleep 2 -done - -# nginx has to be up before provisioning: it terminates TLS for the Zitadel -# domain. --no-deps keeps the apps from starting before they have a client id. -step "Starting the edge" -compose up -d --no-deps nginx - -# Zitadel + database --------------------------------------------------------- -sh "$DEPLOY_DIR/zitadel/provision.sh" - -step "Applying EF Core migrations" -sh "$DEPLOY_DIR/scripts/migrate.sh" - -# Applications --------------------------------------------------------------- -step "Starting the applications" -# shellcheck disable=SC2086 -compose up -d $BUILD - -step "Done" -compose ps -cat < -EOF diff --git a/deploy/zitadel/.output-share/admin-sa.pat b/deploy/zitadel/.output-share/admin-sa.pat deleted file mode 100644 index bfdaf76..0000000 --- a/deploy/zitadel/.output-share/admin-sa.pat +++ /dev/null @@ -1 +0,0 @@ -eyJhbGciOiJBMjU2R0NNS1ciLCJlbmMiOiJBMjU2R0NNIiwiaXYiOiJ5Yk1hUFdVUmNqcU44WDNGIiwia2lkIjoib2lkY0tleSIsInRhZyI6Ikw2NjYwN3FRVzZNYVJLaVl3Q0w1N3cifQ.BE_7qqA1Auer5yEoGnyN0v35JlZlK3XgCo0s_CQtG7s.cOf9oudMHRnVTOGj.0DRh9kj7oahl4imvMjxqRLUOZo6NhYlQyslf9_x4XF_TKi1dnQ.Uu36qEL5J1VKSIYcvWlbAg diff --git a/deploy/zitadel/provision.ps1 b/deploy/zitadel/provision.ps1 deleted file mode 100644 index 42dc38a..0000000 --- a/deploy/zitadel/provision.ps1 +++ /dev/null @@ -1,193 +0,0 @@ -#requires -Version 5.1 -<# -.SYNOPSIS - Provisions Zitadel for Musify in development (idempotent). - -.DESCRIPTION - Creates the Musify project, the API OIDC app (public, PKCE, JWT access - token) and the web player OIDC app (confidential), then wires the resulting - client ids into deploy/.env, Musify.Api user-secrets and web-player/.env. - Authenticates with the service-account PAT that Zitadel writes on init - (zitadel/.output/admin-sa.pat). Re-running reuses the existing project and - apps and just refreshes the wired values. -#> -[CmdletBinding()] -param( - [string]$ProjectName = "Musify", - [string]$ApiAppName = "Musify API (Scalar)", - [string]$WebAppName = "Musify Web", - # Alternate .env to read/write (e.g. deploy/.env.share). Defaults to deploy/.env. - [string]$EnvFile, - # Skip writing Musify.Api user-secrets and web-player/.env: use when the apps - # only ever run in Docker for this environment and get their config from compose. - [switch]$SkipHostWiring, - # Matches the zitadel volume mount for this environment (see compose.share.yml). - [string]$OutputDir = ".output" -) - -$ErrorActionPreference = "Stop" -$deployDir = Split-Path $PSScriptRoot -Parent -$repoDir = Split-Path $deployDir -Parent -$apiProject = Join-Path $repoDir "backend\Hosts\Musify.Api" -$webEnvFile = Join-Path $repoDir "web-player\.env" -$envFile = if ($EnvFile) { $EnvFile } else { Join-Path $deployDir ".env" } -$patFile = Join-Path $PSScriptRoot "$OutputDir\admin-sa.pat" - -. (Join-Path $deployDir "scripts\common.ps1") - -$cfg = Read-DotEnv $envFile -$issuer = $cfg['PUBLIC_AUTH_URL'] -$apiUrl = $cfg['PUBLIC_API_URL'] -$webUrl = $cfg['PUBLIC_WEB_URL'] -$viteUrl = "http://localhost:5173" # `npm run dev` in web-player/ - -Write-Step "Provisioning Zitadel at $issuer" - -# Zitadel writes the PAT during init; on a cold start it may take a moment. -$deadline = (Get-Date).AddMinutes(3) -while (-not (Test-Path $patFile) -and (Get-Date) -lt $deadline) { Start-Sleep -Seconds 3 } -if (-not (Test-Path $patFile)) { throw "PAT not found ($patFile). Check 'docker logs musify-zitadel'." } - -$headers = @{ Authorization = "Bearer $((Get-Content $patFile -Raw).Trim())" } -$mgmt = "$issuer/management/v1" - -function Invoke-Zitadel($method, $path, $body) { - $params = @{ Method = $method; Uri = "$mgmt$path"; Headers = $headers; ContentType = "application/json"; TimeoutSec = 20 } - if ($body) { $params.Body = ($body | ConvertTo-Json -Depth 10) } - Invoke-RestMethod @params -} - -$deadline = (Get-Date).AddMinutes(2) -do { - try { Invoke-RestMethod "$issuer/.well-known/openid-configuration" -TimeoutSec 5 | Out-Null; $ready = $true } - catch { $ready = $false; Start-Sleep 3 } -} until ($ready -or (Get-Date) -gt $deadline) -if (-not $ready) { throw "Zitadel did not respond in time." } - -# The images tagged `latest` require the standalone Login UI v2 container, which -# this stack does not run, so OIDC would redirect to /ui/v2/login and 404. -# Opting out keeps the built-in v1 login. -try { - Invoke-RestMethod -Method PUT -Uri "$issuer/v2/features/instance" -Headers $headers ` - -ContentType "application/json" -Body (@{ loginV2 = @{ required = $false } } | ConvertTo-Json) -TimeoutSec 20 | Out-Null -} catch { - Write-Host " warning: could not disable the required Login UI v2: $($_.Exception.Message)" -} - -# Project -------------------------------------------------------------------- -$project = (Invoke-Zitadel POST "/projects/_search" @{ - queries = @(@{ nameQuery = @{ name = $ProjectName; method = "TEXT_QUERY_METHOD_EQUALS" } }) -}).result | Where-Object { $_.name -eq $ProjectName } | Select-Object -First 1 - -if ($project) { - $projectId = $project.id - Write-Host " project '$ProjectName' reused (id $projectId)" -} else { - $projectId = (Invoke-Zitadel POST "/projects" @{ name = $ProjectName }).id - Write-Host " project '$ProjectName' created (id $projectId)" -} - -function Get-ZitadelApp($name) { - (Invoke-Zitadel POST "/projects/$projectId/apps/_search" @{ - queries = @(@{ nameQuery = @{ name = $name; method = "TEXT_QUERY_METHOD_EQUALS" } }) - }).result | Where-Object { $_.name -eq $name } | Select-Object -First 1 -} - -# API app: public client used by the Scalar UI to obtain a token ------------- -$apiApp = Get-ZitadelApp $ApiAppName -if ($apiApp) { - $apiClientId = (Invoke-Zitadel GET "/projects/$projectId/apps/$($apiApp.id)" $null).app.oidcConfig.clientId - Write-Host " app '$ApiAppName' reused (clientId $apiClientId)" -} else { - $apiClientId = (Invoke-Zitadel POST "/projects/$projectId/apps/oidc" @{ - name = $ApiAppName - redirectUris = @("$apiUrl/scalar/", "$apiUrl/scalar/oauth2-redirect.html") - postLogoutRedirectUris = @("$apiUrl/scalar/") - responseTypes = @("OIDC_RESPONSE_TYPE_CODE") - grantTypes = @("OIDC_GRANT_TYPE_AUTHORIZATION_CODE") - appType = "OIDC_APP_TYPE_USER_AGENT" - authMethodType = "OIDC_AUTH_METHOD_TYPE_NONE" - version = "OIDC_VERSION_1_0" - devMode = $true # allow http redirect URIs - accessTokenType = "OIDC_TOKEN_TYPE_JWT" # required so JwtBearer can validate it - accessTokenRoleAssertion = $true - idTokenRoleAssertion = $true - idTokenUserinfoAssertion = $true - }).clientId - Write-Host " app '$ApiAppName' created (clientId $apiClientId)" -} - -# Web app: confidential client used by the SvelteKit server ------------------ -$webApp = Get-ZitadelApp $WebAppName -$webSecret = $cfg['WEB_CLIENT_SECRET'] -if ($webApp) { - $webClientId = (Invoke-Zitadel GET "/projects/$projectId/apps/$($webApp.id)" $null).app.oidcConfig.clientId - # The secret is only returned once, so regenerate it if we no longer have it. - if (-not $webSecret) { - $webSecret = (Invoke-Zitadel POST "/projects/$projectId/apps/$($webApp.id)/oidc_config/_generate_client_secret" @{}).clientSecret - Write-Host " app '$WebAppName' reused (clientId $webClientId, secret regenerated)" - } else { - Write-Host " app '$WebAppName' reused (clientId $webClientId)" - } -} else { - $created = Invoke-Zitadel POST "/projects/$projectId/apps/oidc" @{ - name = $WebAppName - redirectUris = @("$webUrl/auth/callback", "$viteUrl/auth/callback") - postLogoutRedirectUris = @("$webUrl/", "$viteUrl/") - responseTypes = @("OIDC_RESPONSE_TYPE_CODE") - grantTypes = @("OIDC_GRANT_TYPE_AUTHORIZATION_CODE", "OIDC_GRANT_TYPE_REFRESH_TOKEN") - appType = "OIDC_APP_TYPE_WEB" - authMethodType = "OIDC_AUTH_METHOD_TYPE_BASIC" - version = "OIDC_VERSION_1_0" - devMode = $true - accessTokenType = "OIDC_TOKEN_TYPE_JWT" - accessTokenRoleAssertion = $true - idTokenRoleAssertion = $true - idTokenUserinfoAssertion = $true - } - $webClientId = $created.clientId - $webSecret = $created.clientSecret - Write-Host " app '$WebAppName' created (clientId $webClientId)" -} - -# Wire the values ------------------------------------------------------------ -Set-DotEnvValue $envFile "AUTH_CLIENT_ID" $apiClientId -Set-DotEnvValue $envFile "WEB_CLIENT_ID" $webClientId -Set-DotEnvValue $envFile "WEB_CLIENT_SECRET" $webSecret - -if ($SkipHostWiring) { - Write-Host " wired into $envFile only (-SkipHostWiring: apps run in Docker)" -} else { - # Musify.Api reads these from user-secrets when it runs outside Docker. - @{ - "Authentication:ClientId" = $apiClientId - "Authentication:AudienceAddress" = $apiClientId - "Authentication:IssuerAddress" = $issuer - "Authentication:MetadataAddress" = "$issuer/.well-known/openid-configuration" - "Authentication:AuthorizationEndpoint" = "$issuer/oauth/v2/authorize?prompt=login" - "Authentication:TokenEndpoint" = "$issuer/oauth/v2/token" - "Authentication:ScalarRedirectUri" = "$apiUrl/scalar/" - "Authentication:RequireHttpsMetadata" = "false" - }.GetEnumerator() | ForEach-Object { - dotnet user-secrets set $_.Key $_.Value --project $apiProject | Out-Null - } - - # web-player/.env is what `npm run dev` reads (the container gets its config - # from compose instead). - if (-not (Test-Path $webEnvFile)) { - Copy-Item (Join-Path $repoDir "web-player\.env.example") $webEnvFile - } - @{ - ZITADEL_ISSUER = $issuer - ZITADEL_CLIENT_ID = $webClientId - ZITADEL_CLIENT_SECRET = $webSecret - AUTH_REDIRECT_URI = "$viteUrl/auth/callback" - AUTH_POST_LOGOUT_URI = "$viteUrl/" - SESSION_SECRET = $cfg['SESSION_SECRET'] - API_BASE_URL = $apiUrl - }.GetEnumerator() | ForEach-Object { - Set-DotEnvValue $webEnvFile $_.Key $_.Value - } - - Write-Host " wired into deploy/.env, Musify.Api user-secrets and web-player/.env" -} diff --git a/deploy/zitadel/provision.sh b/deploy/zitadel/provision.sh deleted file mode 100644 index 1ab2890..0000000 --- a/deploy/zitadel/provision.sh +++ /dev/null @@ -1,145 +0,0 @@ -#!/bin/sh -# Provisions Zitadel for Musify in production (idempotent). -# -# Creates the Musify project, the API OIDC app (public, PKCE, JWT access token) -# and the web player OIDC app (confidential), then writes AUTH_CLIENT_ID, -# WEB_CLIENT_ID and WEB_CLIENT_SECRET into deploy/.env.prod. Authenticates with -# the service-account PAT that Zitadel writes on init. Re-running reuses the -# existing project and apps. -set -eu - -SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd) -DEPLOY_DIR=$(CDPATH= cd -- "$SCRIPT_DIR/.." && pwd) -ENV_FILE="$DEPLOY_DIR/.env.prod" -PAT_FILE="$SCRIPT_DIR/.output/admin-sa.pat" - -. "$DEPLOY_DIR/scripts/lib.sh" - -require_command curl -require_command jq - -PROJECT_NAME=${PROJECT_NAME:-Musify} -API_APP_NAME=${API_APP_NAME:-Musify API} -WEB_APP_NAME=${WEB_APP_NAME:-Musify Web} - -ISSUER=$(env_get "$ENV_FILE" PUBLIC_AUTH_URL) -API_URL=$(env_get "$ENV_FILE" PUBLIC_API_URL) -WEB_URL=$(env_get "$ENV_FILE" PUBLIC_WEB_URL) -MGMT="$ISSUER/management/v1" - -step "Provisioning Zitadel at $ISSUER" - -# Zitadel writes the PAT during init; on a cold start it may take a moment. -deadline=$(( $(date +%s) + 180 )) -while [ ! -f "$PAT_FILE" ] && [ "$(date +%s)" -lt "$deadline" ]; do sleep 3; done -[ -f "$PAT_FILE" ] || fail "PAT not found ($PAT_FILE). Check 'docker logs musify-zitadel'." -PAT=$(tr -d '\r\n' <"$PAT_FILE") - -api_call() { - # api_call [json body] - if [ $# -ge 3 ]; then - curl -fsS -X "$1" "$MGMT$2" \ - -H "Authorization: Bearer $PAT" -H "Content-Type: application/json" -d "$3" - else - curl -fsS -X "$1" "$MGMT$2" -H "Authorization: Bearer $PAT" - fi -} - -deadline=$(( $(date +%s) + 120 )) -until curl -fsS "$ISSUER/.well-known/openid-configuration" >/dev/null 2>&1; do - [ "$(date +%s)" -lt "$deadline" ] || fail "Zitadel did not respond at $ISSUER." - sleep 3 -done - -# The images tagged `latest` require the standalone Login UI v2 container, which -# this stack does not run, so OIDC would redirect to /ui/v2/login and 404. -curl -fsS -X PUT "$ISSUER/v2/features/instance" \ - -H "Authorization: Bearer $PAT" -H "Content-Type: application/json" \ - -d '{"loginV2":{"required":false}}' >/dev/null 2>&1 \ - || echo " warning: could not disable the required Login UI v2" - -name_query() { - printf '{"queries":[{"nameQuery":{"name":"%s","method":"TEXT_QUERY_METHOD_EQUALS"}}]}' "$1" -} - -# Project -------------------------------------------------------------------- -PROJECT_ID=$(api_call POST "/projects/_search" "$(name_query "$PROJECT_NAME")" \ - | jq -r --arg n "$PROJECT_NAME" '.result // [] | map(select(.name == $n)) | first | .id // empty') - -if [ -n "$PROJECT_ID" ]; then - echo " project '$PROJECT_NAME' reused (id $PROJECT_ID)" -else - PROJECT_ID=$(api_call POST "/projects" "$(printf '{"name":"%s"}' "$PROJECT_NAME")" | jq -r '.id') - echo " project '$PROJECT_NAME' created (id $PROJECT_ID)" -fi - -find_app() { - api_call POST "/projects/$PROJECT_ID/apps/_search" "$(name_query "$1")" \ - | jq -r --arg n "$1" '.result // [] | map(select(.name == $n)) | first | .id // empty' -} - -# API app: public client used by the Scalar UI to obtain a token ------------- -API_APP_ID=$(find_app "$API_APP_NAME") -if [ -n "$API_APP_ID" ]; then - API_CLIENT_ID=$(api_call GET "/projects/$PROJECT_ID/apps/$API_APP_ID" | jq -r '.app.oidcConfig.clientId') - echo " app '$API_APP_NAME' reused (clientId $API_CLIENT_ID)" -else - API_CLIENT_ID=$(api_call POST "/projects/$PROJECT_ID/apps/oidc" "$(jq -n \ - --arg name "$API_APP_NAME" --arg api "$API_URL" '{ - name: $name, - redirectUris: [($api + "/scalar/"), ($api + "/scalar/oauth2-redirect.html")], - postLogoutRedirectUris: [($api + "/scalar/")], - responseTypes: ["OIDC_RESPONSE_TYPE_CODE"], - grantTypes: ["OIDC_GRANT_TYPE_AUTHORIZATION_CODE"], - appType: "OIDC_APP_TYPE_USER_AGENT", - authMethodType: "OIDC_AUTH_METHOD_TYPE_NONE", - version: "OIDC_VERSION_1_0", - devMode: false, - accessTokenType: "OIDC_TOKEN_TYPE_JWT", - accessTokenRoleAssertion: true, - idTokenRoleAssertion: true, - idTokenUserinfoAssertion: true - }')" | jq -r '.clientId') - echo " app '$API_APP_NAME' created (clientId $API_CLIENT_ID)" -fi - -# Web app: confidential client used by the SvelteKit server ------------------ -WEB_APP_ID=$(find_app "$WEB_APP_NAME") -WEB_CLIENT_SECRET=$(env_get "$ENV_FILE" WEB_CLIENT_SECRET) -if [ -n "$WEB_APP_ID" ]; then - WEB_CLIENT_ID=$(api_call GET "/projects/$PROJECT_ID/apps/$WEB_APP_ID" | jq -r '.app.oidcConfig.clientId') - if [ -z "$WEB_CLIENT_SECRET" ]; then - # The secret is only returned once, so regenerate it if we no longer have it. - WEB_CLIENT_SECRET=$(api_call POST "/projects/$PROJECT_ID/apps/$WEB_APP_ID/oidc_config/_generate_client_secret" '{}' \ - | jq -r '.clientSecret') - echo " app '$WEB_APP_NAME' reused (clientId $WEB_CLIENT_ID, secret regenerated)" - else - echo " app '$WEB_APP_NAME' reused (clientId $WEB_CLIENT_ID)" - fi -else - CREATED=$(api_call POST "/projects/$PROJECT_ID/apps/oidc" "$(jq -n \ - --arg name "$WEB_APP_NAME" --arg web "$WEB_URL" '{ - name: $name, - redirectUris: [($web + "/auth/callback")], - postLogoutRedirectUris: [($web + "/")], - responseTypes: ["OIDC_RESPONSE_TYPE_CODE"], - grantTypes: ["OIDC_GRANT_TYPE_AUTHORIZATION_CODE", "OIDC_GRANT_TYPE_REFRESH_TOKEN"], - appType: "OIDC_APP_TYPE_WEB", - authMethodType: "OIDC_AUTH_METHOD_TYPE_BASIC", - version: "OIDC_VERSION_1_0", - devMode: false, - accessTokenType: "OIDC_TOKEN_TYPE_JWT", - accessTokenRoleAssertion: true, - idTokenRoleAssertion: true, - idTokenUserinfoAssertion: true - }')") - WEB_CLIENT_ID=$(echo "$CREATED" | jq -r '.clientId') - WEB_CLIENT_SECRET=$(echo "$CREATED" | jq -r '.clientSecret') - echo " app '$WEB_APP_NAME' created (clientId $WEB_CLIENT_ID)" -fi - -env_set "$ENV_FILE" AUTH_CLIENT_ID "$API_CLIENT_ID" -env_set "$ENV_FILE" WEB_CLIENT_ID "$WEB_CLIENT_ID" -env_set "$ENV_FILE" WEB_CLIENT_SECRET "$WEB_CLIENT_SECRET" - -echo " wired into $(basename "$ENV_FILE")" diff --git a/docs/development.md b/docs/development.md index 2aae9a2..6986d01 100644 --- a/docs/development.md +++ b/docs/development.md @@ -5,17 +5,15 @@ producción) está en [deploy/README.md](../deploy/README.md). ## Arranque -```powershell -./deploy/up.ps1 # infraestructura; las apps se ejecutan desde el IDE -./deploy/up.ps1 -Apps # todo en Docker (api, worker, gateway, web player) +```sh +docker compose -f deploy/compose.yml -f deploy/compose.dev.yml up -d ``` -`up.ps1` hace el bootstrap completo: crea `deploy/.env`, genera las claves RS256 -de stream-ticket en `deploy/keys/`, levanta el stack, aplica las migraciones y -provisiona Zitadel (proyecto + apps OIDC), dejando los client ids en -`deploy/.env`, en los user-secrets de `Musify.Api` y en `web-player/.env`. - -Otros modificadores: `-Tools` (pgAdmin), `-SkipMigrations`, `-Down`, `-Destroy`. +Ese único comando levanta la infraestructura y corre los jobs *one-shot* de +bootstrap: genera las claves RS256 de stream-ticket en `deploy/keys/`, aplica +las migraciones EF Core y provisiona Zitadel (proyecto + apps OIDC), dejando +los client ids en `deploy/.env`. Detalle completo, incluido cómo arrancar las +apps en Docker con `--profile apps`, en [deploy/README.md](../deploy/README.md). ## Aplicaciones @@ -46,21 +44,25 @@ DB: `musify_db`, usuario `postgres`/`postgres`. Bucket S3: `webapi-storage` ## Migraciones -No hay migración automática al arrancar; `up.ps1` las aplica en el bootstrap. -A mano, tras cambios de esquema: +El servicio `migrate` de `deploy/compose.yml` las aplica en cada `docker +compose up -d` (no-op si el esquema ya está al día). A mano, tras cambios de +esquema, contra la base local: ```powershell dotnet ef database update --project backend/Core/Musify.Infrastructure --startup-project backend/Core/Musify.Infrastructure --context Database ``` `Musify.Infrastructure` es a la vez el proyecto de migraciones y el de arranque: -tiene el `IDesignTimeDbContextFactory` y el paquete `EFCore.Design`. La cadena de -conexión se lee de sus user-secrets (`up.ps1` la deja puesta). +tiene el `IDesignTimeDbContextFactory` y el paquete `EFCore.Design`. La cadena +de conexión se lee de sus user-secrets, o de la variable de entorno +`Database__ConnectionString` si no hay user-secrets (así es como la usa el +contenedor `migrate`, ver `backend/Musify.Migrator.Dockerfile`). ## Stream-tickets (claves RS256) La API firma los stream-tickets con la clave privada y el gateway los valida con -la pública. `up.ps1` las genera en `deploy/keys/`; a mano: +la pública. El servicio `keys-init` de `deploy/compose.yml` las genera en +`deploy/keys/` si no existen; a mano: ```powershell openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:2048 -out deploy/keys/stream_private.pem @@ -75,8 +77,10 @@ openssl rsa -in deploy/keys/stream_private.pem -pubout -out deploy/keys/stream_p Convención: `AppSettings.json` es la plantilla con valores vacíos o neutros; `AppSettings.Development.json` tiene los valores reales de dev; en Docker todo se sobreescribe con variables de entorno (`Seccion__Clave`) desde -`deploy/compose.yml`. Los valores de `Authentication` los deja el provisioning -en user-secrets, no en el repo. +`deploy/compose.yml`. Para correr `Musify.Api` desde el IDE, los valores de +`Authentication` van en user-secrets (no en el repo) — cópialos de +`deploy/.env` una vez que `zitadel-init` los haya generado (ver +[deploy/README.md](../deploy/README.md)). Secciones clave: `Authentication` (Zitadel), `Database`, `MassTransit`, `InfrastructureStorage` (S3), `ApplicationStorage` (bucket), `Track`, `PlayList`, diff --git a/web-player/.env.example b/web-player/.env.example index c538582..8efa0a2 100644 --- a/web-player/.env.example +++ b/web-player/.env.example @@ -1,5 +1,6 @@ -# Local configuration for `npm run dev`. deploy/zitadel/provision.ps1 fills in -# the Zitadel values; in Docker the same variables come from deploy/compose.yml. +# Local configuration for `npm run dev`. Copy the WEB_CLIENT_ID / +# WEB_CLIENT_SECRET that deploy/compose.yml's zitadel-init service writes to +# deploy/.env; in Docker the same variables come from deploy/compose.yml. # Zitadel OIDC (server-side auth) ZITADEL_ISSUER=http://host.docker.internal:8080 diff --git a/web-player/README.md b/web-player/README.md index 2ddec23..842f370 100644 --- a/web-player/README.md +++ b/web-player/README.md @@ -12,10 +12,10 @@ npm install npm run dev # :5173 ``` -Configuration lives in `.env` (template in `.env.example`). -`deploy/zitadel/provision.ps1` fills in the Zitadel values automatically, so the -usual flow is to bring the stack up first (`./deploy/up.ps1`) and then run the -dev server. +Configuration lives in `.env` (template in `.env.example`). Bring the +infrastructure stack up first (see [../deploy/README.md](../deploy/README.md)) +and copy the `WEB_CLIENT_ID` / `WEB_CLIENT_SECRET` it writes to `deploy/.env` +into this `.env` once, then run the dev server. ```sh npm run check # svelte-check — must be clean before closing a change