feat(deploy): consolidated dev stack with auto-provisioned Zitadel
Single docker-compose stack for local development (Postgres, Zitadel, RabbitMQ, SeaweedFS + bucket init, Jaeger) plus an optional overlay to run the .NET apps in Docker. up.ps1 orchestrates the full bootstrap: stack up, stream-ticket keys, EF Core migrations and Zitadel provisioning. - Zitadel is provisioned automatically (service-account PAT -> project + OIDC app with JWT access token); the generated ClientId is wired into Musify.Api user-secrets and .env. No manual console steps. - ZITADEL_EXTERNAL_DOMAIN=host.docker.internal so token validation works both with local apps and full-docker. - App Dockerfiles copy AppSettings*.json to lowercase so config loads on case-sensitive Linux; Worker image ships ffmpeg. - Scripts split by responsibility (common/stream-keys/migrate/provision); apps overlay de-duped via YAML anchors. - Remove the superseded standalone seaweedfs compose; point docs/development at the unified stack.
This commit is contained in:
@@ -0,0 +1,192 @@
|
||||
# Development deploy
|
||||
|
||||
Everything the backend needs for development, in one place. A single
|
||||
`docker-compose.yml` brings up **PostgreSQL, Zitadel, RabbitMQ, SeaweedFS and
|
||||
Jaeger**; the .NET apps run from the IDE (or, optionally, in Docker too).
|
||||
|
||||
## Quick start
|
||||
|
||||
```powershell
|
||||
# From the repo root (Musify/)
|
||||
./deploy/up.ps1
|
||||
```
|
||||
|
||||
`up.ps1` does the full bootstrap:
|
||||
|
||||
1. Creates `deploy/.env` from `.env.example` if missing.
|
||||
2. Starts the stack (`docker compose up -d`) and waits for Postgres to be healthy.
|
||||
3. Creates the `webapi-storage` bucket in SeaweedFS (`seaweedfs-init` service).
|
||||
4. Generates the RS256 stream-ticket keys in `D:\weed` if missing.
|
||||
5. Creates the Worker temp directory (`D:\tempsFilesDev`).
|
||||
6. Applies the EF Core migrations to `musify_db`.
|
||||
7. **Provisions Zitadel automatically** (project + OIDC app with JWT token) and
|
||||
wires the ClientId into `Musify.Api` user-secrets and `.env`.
|
||||
|
||||
When it finishes it prints a summary with URLs and credentials. **No manual steps.**
|
||||
|
||||
### Without the script (infra only)
|
||||
|
||||
```powershell
|
||||
docker compose -f deploy/docker-compose.yml up -d
|
||||
```
|
||||
|
||||
The compose file still creates the bucket on its own; run migrations with
|
||||
`dotnet ef database update` (see [../docs/development.md](../docs/development.md)).
|
||||
|
||||
## Services and ports
|
||||
|
||||
| Service | Port(s) | Used for |
|
||||
|---|---|---|
|
||||
| PostgreSQL | `59000` → 5432 | API + Worker (EF Core) and the Zitadel database |
|
||||
| Zitadel | `8080` | OIDC/OAuth2 (console at `/ui/console`) |
|
||||
| RabbitMQ | `5672` / `15672` | MassTransit / management UI |
|
||||
| SeaweedFS S3 | `8333` | S3 (presigned PUT/GET) |
|
||||
| SeaweedFS filer | `8888` | serves files → StreamingGateway |
|
||||
| SeaweedFS master | `9333` | admin/UI |
|
||||
| Jaeger | `16686` / `4317` / `4318` | OTLP traces (UI / gRPC / HTTP) |
|
||||
|
||||
Ports are published to the host, so apps run locally connect without changing
|
||||
`AppSettings.Development.json` (Postgres, RabbitMQ, S3, OTLP).
|
||||
|
||||
## Authentication (Zitadel) — automatic
|
||||
|
||||
There are **no manual steps**. `up.ps1` runs `zitadel/provision.ps1`, which
|
||||
idempotently:
|
||||
|
||||
1. Uses a **service account** with a **PAT** that Zitadel writes on init
|
||||
(`zitadel/.output/admin-sa.pat`).
|
||||
2. Creates (or reuses) the **project** `Musify` and an **OIDC app** (PKCE,
|
||||
`devMode`, **access token = JWT** — required for `JwtBearer` to validate).
|
||||
3. Captures the generated **ClientId** and writes it to:
|
||||
- **`Musify.Api` user-secrets** (for running the apps locally), and
|
||||
- `deploy/.env` as `AUTH_CLIENT_ID` (for the full-docker path).
|
||||
|
||||
The values stay stable while the Postgres volume persists. After `up.ps1 -Destroy`
|
||||
the next `up.ps1` re-provisions (new ClientId, re-wired).
|
||||
|
||||
- Console: `http://host.docker.internal:8080/ui/console`
|
||||
- Admin login: `admin@zitadel.host.docker.internal` / `Password1!` (see `.env`).
|
||||
- Test login: open Scalar (`http://localhost:5111/scalar/v1`) and authorize.
|
||||
|
||||
### Why `host.docker.internal`
|
||||
|
||||
`ZITADEL_EXTERNAL_DOMAIN=host.docker.internal` forms the token *issuer* and
|
||||
resolves the **same** from the browser, local apps and containers, so token
|
||||
validation works both ways (local and full-docker). Zitadel routes by the `Host`
|
||||
header, so the domain must be identical everywhere. If you only run the apps
|
||||
locally you can use `localhost`. Changing the domain requires a re-init:
|
||||
`up.ps1 -Destroy; up.ps1`.
|
||||
|
||||
## Running the apps
|
||||
|
||||
### Recommended: from the IDE (best for debugging)
|
||||
|
||||
```powershell
|
||||
dotnet run --project backend/Musify.Api # :5111
|
||||
dotnet run --project backend/Musify.StreamingGateway # :8081
|
||||
dotnet run --project backend/Musify.Worker # background
|
||||
```
|
||||
|
||||
Host requirements: **.NET 10 SDK**, **ffmpeg** on PATH (Worker audio transcode).
|
||||
|
||||
### Optional: everything in Docker
|
||||
|
||||
```powershell
|
||||
docker compose -f deploy/docker-compose.yml -f deploy/docker-compose.apps.yml up -d --build
|
||||
```
|
||||
|
||||
Builds and runs `api`, `worker` and `gateway` as containers (the Worker already
|
||||
ships ffmpeg). The stream-ticket keys are mounted from `D:\weed` (configurable via
|
||||
`STREAM_KEYS_DIR` in `.env`). Auth works in this mode too (the API downloads the
|
||||
Zitadel JWKS via `host.docker.internal`).
|
||||
|
||||
> Note: on Linux the apps look for `appsettings*.json` in lowercase. The `Api`/
|
||||
> `Worker` projects use `AppSettings*.json` (PascalCase), so their Dockerfiles
|
||||
> create lowercase copies at publish time. On Windows it did not matter
|
||||
> (case-insensitive).
|
||||
|
||||
## Useful commands
|
||||
|
||||
```powershell
|
||||
./deploy/up.ps1 -SkipMigrations # bring up without touching the DB
|
||||
./deploy/up.ps1 -Down # stop and remove containers (keep data)
|
||||
./deploy/up.ps1 -Destroy # stop and remove containers + volumes (wipe data)
|
||||
|
||||
docker compose -f deploy/docker-compose.yml logs -f zitadel
|
||||
docker compose -f deploy/docker-compose.yml ps
|
||||
```
|
||||
|
||||
## Project layout — what each file does
|
||||
|
||||
```
|
||||
deploy/
|
||||
├─ docker-compose.yml # infrastructure stack (single source of truth)
|
||||
├─ docker-compose.apps.yml # optional overlay: the 3 .NET apps in Docker
|
||||
├─ .env # local config (secrets/ports); not versioned
|
||||
├─ .env.example # config template (versioned)
|
||||
├─ up.ps1 # orchestrator: up / down / destroy
|
||||
├─ README.md # this file
|
||||
├─ scripts/
|
||||
│ ├─ common.ps1 # shared helpers (Read-DotEnv, Write-Step)
|
||||
│ ├─ stream-keys.ps1 # generates the RS256 stream-ticket key pair
|
||||
│ └─ migrate.ps1 # applies EF Core migrations
|
||||
├─ seaweedfs/
|
||||
│ └─ s3.conf # SeaweedFS S3 identities/credentials
|
||||
└─ zitadel/
|
||||
├─ provision.ps1 # creates project + OIDC app, wires the ClientId
|
||||
└─ .output/ # PAT + init secrets written by Zitadel (not versioned)
|
||||
|
||||
backend/
|
||||
├─ Musify.Api.Dockerfile # API image
|
||||
├─ Musify.Worker.Dockerfile # Worker image (includes ffmpeg)
|
||||
├─ Musify.StreamingGateway.Dockerfile # Gateway image
|
||||
└─ .dockerignore
|
||||
```
|
||||
|
||||
### Compose
|
||||
|
||||
- **`docker-compose.yml`** — the infra stack (`postgres`, `zitadel`, `rabbitmq`,
|
||||
`seaweedfs`, the one-shot `seaweedfs-init` bucket creator, and `jaeger`). This is
|
||||
the single source of truth for services, ports and volumes. Values come from `.env`.
|
||||
- **`docker-compose.apps.yml`** — an optional overlay that adds the `api`, `worker`
|
||||
and `gateway` services. It merges with the infra file (same `musify-dev` project
|
||||
and network) and shares the backend env via YAML anchors. Used to run the whole
|
||||
stack in Docker; for active development prefer running the apps from the IDE.
|
||||
|
||||
### Config
|
||||
|
||||
- **`.env`** — the actual values used by compose (Postgres/RabbitMQ/Zitadel/S3
|
||||
credentials, ports, host paths, `AUTH_CLIENT_ID`). Ignored by git (`*.env`).
|
||||
- **`.env.example`** — the versioned template. Copy it to `.env` (or let `up.ps1`
|
||||
do it) and adjust.
|
||||
|
||||
### Scripts (PowerShell)
|
||||
|
||||
- **`up.ps1`** — the orchestrator. Parses `.env`, starts the stack, waits for
|
||||
Postgres, then calls the sub-scripts in order (keys → migrations → provisioning)
|
||||
and prints the summary. Also handles `-Down` / `-Destroy` / `-SkipMigrations`.
|
||||
- **`scripts/common.ps1`** — shared helpers dot-sourced by the others: `Read-DotEnv`
|
||||
(parse `.env` into a hashtable) and `Write-Step` (step logging).
|
||||
- **`scripts/stream-keys.ps1`** — generates the RS256 key pair that the API uses to
|
||||
sign stream tickets and the Gateway uses to verify them. No-op if the keys exist.
|
||||
- **`scripts/migrate.ps1`** — sets the connection string in user-secrets and runs
|
||||
`dotnet ef database update` (startup project = `Musify.Infrastructure`).
|
||||
- **`zitadel/provision.ps1`** — reads the service-account PAT, creates/reuses the
|
||||
Zitadel project and OIDC app (JWT token), and wires the ClientId into
|
||||
`Musify.Api` user-secrets and `.env`. Idempotent; can be run standalone.
|
||||
|
||||
### Service configs
|
||||
|
||||
- **`seaweedfs/s3.conf`** — the S3 identities (access/secret keys) SeaweedFS loads.
|
||||
Must match `InfrastructureStorage:*` in the app settings and `S3_*` in `.env`.
|
||||
- **`zitadel/.output/`** — created at runtime; holds the PAT Zitadel writes on init
|
||||
(`admin-sa.pat`). Not versioned; wiped by `up.ps1 -Destroy`.
|
||||
|
||||
### Dockerfiles (in `backend/`)
|
||||
|
||||
- **`Musify.Api.Dockerfile`** / **`Musify.Worker.Dockerfile`** /
|
||||
**`Musify.StreamingGateway.Dockerfile`** — multi-stage builds for each host. The
|
||||
Worker image also installs ffmpeg. Built by the apps overlay; build context is
|
||||
`backend/` so restore can copy the central props and all csproj files.
|
||||
- **`.dockerignore`** — keeps `bin/`, `obj/`, IDE folders and `DesignSettings.json`
|
||||
out of the build context.
|
||||
Reference in New Issue
Block a user