SharedServices/README.md

# Infrastructure

The technology services shared across projects (currently Musify, Ping and
docker-manager, each its own git repository): PostgreSQL, Zitadel (OIDC),
SeaweedFS (S3-compatible storage), RabbitMQ and Jaeger. None of these have
any business logic of their own, so instead of each project running (and
provisioning, and keeping in sync) its own copy, they run once, here.

This repo doesn't assume anything about where the other projects are
checked out. Each one just needs a local path to this repo's checkout (see
"The Zitadel PAT" below), which you set once in that project's own `.env`.

Each project keeps its own independent, fast `docker compose up` for the
things that actually are its own: API/worker/gateway containers, database
migrations, its own OIDC apps and storage bucket. Those get created against
the shared Zitadel/SeaweedFS below by the project's own `zitadel-init` and
`seaweedfs-init` one-shot jobs (see e.g. `Musify/MusifyM/deploy/compose.yml`).
A project's dev stack only needs this one already running; nothing here
needs to know a consuming project exists.

## What's here vs. what stays in each project

| Here (shared) | Stays in each project |
|---|---|
| PostgreSQL (one instance, one database and role per project) | The project's own EF Core migrations, connection pooling config |
| Zitadel (OIDC provider) | The project's own OIDC app registration (`zitadel-init`), JWT validation config |
| SeaweedFS (S3 storage) | The project's own bucket (`seaweedfs-init`), key layout |
| RabbitMQ (broker) | The project's own exchanges/queues/consumers |
| Jaeger (OTLP collector + UI) | The project's own instrumentation |
| (not shared) | LiveKit (Ping only, a single-project SFU with its own port range) |

## Running it

```bash
cp .env.example .env
docker compose -f compose.yml -f compose.dev.yml up -d
```

That starts Postgres, Zitadel (and its login UI), SeaweedFS, RabbitMQ and Jaeger, with their
ports published on `127.0.0.1` only (`DEV_BIND_ADDRESS`), so the dev
credentials aren't reachable from the LAN. Add `--profile tools` to also
start pgAdmin.

In production, the base `compose.yml` file **is** the production shape
already. There are no published ports, and everything is reachable only on
the internal `infra-net` network, so no overlay is needed:

```bash
cp .env.prod.example .env.prod
# fill in real values, then: chmod 600 .env.prod
docker compose --env-file .env.prod -f compose.yml up -d
```

A consuming project's own production compose joins `infra-net` as an
external network to reach `postgres`, `zitadel`, `seaweedfs`, `rabbitmq` and
`jaeger` by name (see Musify's `deploy/compose.prod.yml`).

Every project on `infra-net` shares one DNS namespace, and Docker answers a
service name with *every* container that has it, round-robin. Two projects
each with a service called `api` would get each other's traffic. So anything
that crosses projects (the edge, or one project calling into this stack)
should use the unique `container_name` (`infra-postgres`, `musify-api`, ...),
never a generic service name.

### RabbitMQ and its hostname

RabbitMQ keeps its data under its node name, `rabbit@<hostname>`, so the
container has a fixed `hostname: rabbitmq`. Without it every recreated
container (an image update, an `.env` change) came up with a new hostname
and an empty broker. A deployment that ran before this was fixed has its
data under the old random name; carry the definitions (users, vhosts,
permissions, queues, exchanges) over when applying the change, ideally with
the queues drained, since queued messages aren't part of the export:

```bash
docker exec infra-rabbitmq rabbitmqctl export_definitions /var/lib/rabbitmq/defs.json
docker compose --env-file .env.prod -f compose.yml up -d rabbitmq
docker exec infra-rabbitmq rabbitmqctl import_definitions /var/lib/rabbitmq/defs.json
```

### Ports (development)

| Service | Port | Notes |
|---|---|---|
| PostgreSQL | `5432` | one instance, one database and role per project |
| Zitadel | `8080` | OIDC/OAuth2, console at `/ui/console` |
| Zitadel login | `3900` | Login V2 UI at `/ui/v2/login` (under `8080`'s hostname in prod) |
| SeaweedFS | `8333` / `8888` / `9333` | S3 API / filer / master |
| RabbitMQ | `5672` / `15672` | AMQP / management UI |
| Jaeger | `16686` / `4317` / `4318` | UI / OTLP gRPC / OTLP HTTP |
| pgAdmin (`--profile tools`) | `5050` | Postgres UI |

A consuming project reaches all of these at `host.docker.internal:<port>`
from inside its own containers. That's the same address the browser and the
host use, so it stays valid whichever way that project's own apps are
running (in Docker or from the IDE). No shared Docker network is needed in
development, which is what keeps every project's own `up` independent and
fast, with this stack as the only prerequisite.

### Per-project credentials

No project gets an admin account. `provision` (a one-shot that runs on
every `up`, see `provision/`) gives each project listed in `compose.yml`'s
`x-projects` its own credentials, from the `<PROJECT>_*` settings in the env
file:

| Service | What the project gets | Setting |
|---|---|---|
| PostgreSQL | role `<project>`, owner of database `<project>_db`; no other role can connect to it, and it can't connect to any other | `<PROJECT>_DB_PASSWORD` |
| RabbitMQ | user `<project>`, only on vhost `<project>` | `<PROJECT>_RABBITMQ_PASSWORD` |
| SeaweedFS | S3 identity limited to the project's bucket (it may create that bucket, nothing else) | `<PROJECT>_S3_ACCESS_KEY` / `_SECRET_KEY` |
| Zitadel | service account `<project>-provisioner`, owner of the project's own Zitadel project only; its PAT is in `zitadel/.output/<project>/zitadel.pat` | (the project name, in `compose.yml`) |

An empty setting means that project doesn't use that service. Each
project's own `.env.example` copies its values from here and says so; this
repo's env file is the source of truth. Passwords are re-applied on every
`up`, so rotating one is: change it here and in the project, `up` both.

The admin accounts (`POSTGRES_*`, `RABBITMQ_DEFAULT_*`, `S3_ACCESS_KEY` /
`S3_SECRET_KEY`, `zitadel/.output/admin-sa.pat`) stay with this stack.

Two limits worth knowing: the Zitadel `PROJECT_OWNER` role can still read
the organization's users (every project's users are in the one
organization); and S3 keys must only use `A-Z a-z 0-9 _ + / = . -` (they go
into SeaweedFS's JSON config as they are).

**Adding a project**: add it to `x-projects`, add its `<PROJECT>_*` lines
to `provision`'s and `seaweedfs`'s environment in `compose.yml` (with its
bucket and Zitadel project name), put its values in `.env.example` /
`.env.prod.example`, and `up`.

**Moving an existing deployment over**: fill in the new `<PROJECT>_*`
values and `up`. `provision` hands over the existing databases (and every
table, sequence, type, view and function in them) to each project's role,
reuses the existing Zitadel projects and apps, and leaves the buckets as
they are. Then switch each project's env file to its own credentials. The
only thing that doesn't carry over is RabbitMQ: a project now uses its own
vhost, so drain its queues on `/` first; its consumers recreate their
exchanges and queues in the new vhost.

### The Zitadel PAT

Each project's `zitadel-init` mounts **its own** folder,
`zitadel/.output/<project>/`, read-only (via that project's
`ZITADEL_PAT_DIR` setting) and uses the PAT in it to create its OIDC apps in
its own Zitadel project. It can't see or change any other project's.

That setting has no universally correct value. It's a path (relative or
absolute) to wherever *you* cloned this repo on your machine, from the
consuming project's `deploy/` folder. Each project's `.env.example` ships
with an example value that assumes these repos happen to sit as sibling
folders. If yours don't, change it. Not versioned; `provision` issues a new
PAT whenever the one there stops working (e.g. after a Zitadel volume reset).

### The Zitadel login (Login V2)

Zitadel v4 serves its sign-in pages from a separate container,
`zitadel-login` (Next.js), under `/ui/v2/login`. Zitadel redirects every
login there, so it has to be running for anyone to sign in. It calls the
Zitadel API with its own service account (`login-client`), whose PAT Zitadel
writes on init to the `zitadel_bootstrap` volume, which only `zitadel-login`
mounts.

- **Dev**: no proxy, so the login has its own port (`ZITADEL_LOGIN_HOST_PORT`,
  `3900`) and `ZITADEL_LOGIN_BASE_URL` points the browser at it.
- **Prod**: the edge routes `/ui/v2/login` on Zitadel's hostname to
  `infra-zitadel-login`, and everything else to Zitadel.
- `zitadel` and `zitadel-login` are pinned to the same version; bump both
  together.

`ZITADEL_LOGIN_BASE_URL` and the `login-client` PAT are only applied when the
instance is created. On an instance created before `zitadel-login` existed:

1. In the console, create a service user `login-client`, give it the
   instance role **IAM Login Client**, and create a PAT for it.
2. Copy it into the volume and restart the login:

   ```bash
   docker cp login-client.pat infra-zitadel:/bootstrap/login-client.pat
   docker restart infra-zitadel-login
   ```

3. If the instance still uses the old login, set Login V2 as required with
   base URI `${ZITADEL_LOGIN_BASE_URL}/` in the instance's feature settings.

## The public edge

`edge/compose.yml` is the only thing that publishes host ports 80/443: one
nginx that terminates TLS for every project's domains (Gitea, Musify, ...)
and proxies to their containers by name, plus a certbot container that
renews the Let's Encrypt certificates (webroot) and lets nginx reload them
every 6 hours. It joins `infra-net` and the `gitea_gitea` network (both
external, created by the stacks that own them).

```bash
docker compose -f edge/compose.yml up -d
```

- **Vhosts** live in `edge/conf.d/` (one file per project; `00-common.conf`
  has the shared resolver). Every upstream is a `set $upstream ...`, resolved
  per request, so a project's stack being down never stops the edge from
  starting. Upstreams use the target's `container_name` (see "Running it"
  above for why). Shared bits are in `edge/snippets/`. Check a change with
  `docker exec edge-nginx nginx -t`, then `docker exec edge-nginx nginx -s reload`.
- **Unknown hostnames** hit `01-default.conf`: port 80 closes the connection
  (ACME challenges still work), port 443 refuses the TLS handshake. Nothing
  reaches a project unless its vhost names the host.
- **TLS** settings (`snippets/tls.conf`) follow Mozilla's intermediate
  profile and send HSTS with a 1-day `max-age`. Raise it once every hostname
  is confirmed to work over https: browsers remember it, so a long value
  can't be taken back quickly.
- **Certificates** are state, not versioned (`edge/certbot/`). To issue one
  for new hostnames (their DNS has to already point at the host):

  ```bash
  docker compose -f edge/compose.yml run --rm --entrypoint certbot certbot \
    certonly --webroot -w /var/www/certbot -d host1.example.com -d host2.example.com \
    --non-interactive --agree-tos -m you@example.com
  ```

  Then add the `server` blocks pointing at
  `/etc/letsencrypt/live/<first -d name>/`.

## Resetting

```bash
docker compose -f compose.yml -f compose.dev.yml down          # stop, keep data
docker compose -f compose.yml -f compose.dev.yml down -v       # stop, wipe volumes
```

Wiping the volumes resets Zitadel (all projects/apps/users) and SeaweedFS
(all buckets). Every consuming project's own `zitadel-init`/`seaweedfs-init`
re-creates its piece on the next `up`, since both are idempotent.

## Layout

```
Infrastructure/
├─ compose.yml            # postgres, zitadel (+ login), seaweedfs, rabbitmq, jaeger: the prod shape
├─ edge/                  # shared public nginx + certbot (prod only): compose.yml, conf.d/, snippets/
├─ compose.dev.yml        # dev overlay: publishes ports, adds pgAdmin
├─ .env.example / .env.prod.example
├─ provision/            # per-project credentials: Postgres, RabbitMQ, Zitadel, S3 identities
└─ zitadel/.output/       # admin PAT + one folder per project with its own PAT (not versioned)
```