Give every project its own credentials on the shared services

Every project used the Postgres superuser, the RabbitMQ administrator, an
admin S3 identity and the Zitadel instance-admin PAT, so any one of them
could read or change the others' data.

A new one-shot, `provision` (runs on every `up`, idempotent), gives each
project in x-projects its own:

- Postgres role, owner of its database. No database is open to PUBLIC
  anymore, so a role can only connect to its own. An existing database
  and everything in it is handed over to the project's role.
- RabbitMQ user, with permissions on its own vhost only.
- Zitadel service account, PROJECT_OWNER of its own project only; its
  PAT is written to zitadel/.output/<project>/ for that project's
  zitadel-init to mount instead of the admin PAT.

SeaweedFS renders one S3 identity per project, limited to its bucket, and
refuses to start with a clear message on a malformed or repeated key.

The admin accounts stay with this stack. postgres-init is gone: provision
creates the databases. *.sh files are checked out with LF so they run in
the Linux containers from a Windows checkout.

.env.example

@@ -8,7 +8,7 @@
 # (then the browser can't reach 127.0.0.1-only ports through that name).
 DEV_BIND_ADDRESS=127.0.0.1
 
-# --- PostgreSQL (shared instance, one database per project) -----------------
+# --- PostgreSQL (shared instance; superuser for this stack only) -------------
 POSTGRES_USER=postgres
 POSTGRES_PASSWORD=postgres
 POSTGRES_HOST_PORT=5432
@@ -32,11 +32,11 @@ ZITADEL_DATABASE_PASSWORD=zitadel
 ZITADEL_ADMIN_USERNAME=admin
 ZITADEL_ADMIN_PASSWORD=Password1!
 
-# --- SeaweedFS (S3) ----------------------------------------------------------
+# --- SeaweedFS (S3; admin identity for this stack only) ---------------------
 S3_ACCESS_KEY=admin_access_key
 S3_SECRET_KEY=admin_secret_key
 
-# --- RabbitMQ -----------------------------------------------------------------
+# --- RabbitMQ (administrator for this stack only) ---------------------------
 RABBITMQ_DEFAULT_USER=guest
 RABBITMQ_DEFAULT_PASS=guest
 
@@ -44,3 +44,23 @@ RABBITMQ_DEFAULT_PASS=guest
 PGADMIN_DEFAULT_EMAIL=admin@infra.dev
 PGADMIN_DEFAULT_PASSWORD=admin
 PGADMIN_HOST_PORT=5050
+
+# --- Per-project credentials (see provision/) ---------------------------------
+# Each project gets its own Postgres role/database, RabbitMQ user/vhost and S3
+# identity limited to its bucket; the admin accounts above stay with this
+# stack. Each project's own .env copies its values from here. An empty value
+# means that project doesn't get that service.
+MUSIFY_DB_PASSWORD=musify
+MUSIFY_RABBITMQ_PASSWORD=musify
+MUSIFY_S3_ACCESS_KEY=musify_access_key
+MUSIFY_S3_SECRET_KEY=musify_secret_key
+
+PING_DB_PASSWORD=ping
+PING_RABBITMQ_PASSWORD=
+PING_S3_ACCESS_KEY=ping_access_key
+PING_S3_SECRET_KEY=ping_secret_key
+
+DOCKERMANAGER_DB_PASSWORD=dockermanager
+DOCKERMANAGER_RABBITMQ_PASSWORD=
+DOCKERMANAGER_S3_ACCESS_KEY=
+DOCKERMANAGER_S3_SECRET_KEY=

.env.prod.example

@@ -10,7 +10,7 @@
 # the Zitadel masterkey needs exactly 32 characters) and auth.example.com
 # with the real hostname the shared edge (edge/) terminates TLS for.
 
-# --- PostgreSQL ---------------------------------------------------------------
+# --- PostgreSQL (superuser for this stack only) -----------------------------
 POSTGRES_USER=postgres
 POSTGRES_PASSWORD=CHANGE_ME
 
@@ -29,12 +29,32 @@ ZITADEL_ADMIN_USERNAME=admin@ikerdemo.cat
 ZITADEL_ADMIN_FIRST_NAME=Iker Demo
 ZITADEL_ADMIN_LAST_NAME=Admin
 ZITADEL_ADMIN_DISPLAY_NAME=Iker Demo Admin
+# Must pass Zitadel's password policy (upper, lower, number, symbol, 8+).
 ZITADEL_ADMIN_PASSWORD=CHANGE_ME
 
-# --- SeaweedFS (S3) ----------------------------------------------------------
+# --- SeaweedFS (S3; admin identity for this stack only) ---------------------
 S3_ACCESS_KEY=CHANGE_ME
 S3_SECRET_KEY=CHANGE_ME
 
-# --- RabbitMQ -----------------------------------------------------------------
+# --- RabbitMQ (administrator for this stack only) ---------------------------
 RABBITMQ_DEFAULT_USER=CHANGE_ME
 RABBITMQ_DEFAULT_PASS=CHANGE_ME
+
+# --- Per-project credentials (see provision/) ---------------------------------
+# Each project's own .env.prod copies its values from here. S3 keys: only
+# A-Z a-z 0-9 _ + / = . - (hex is fine). Empty = that project doesn't get
+# that service.
+MUSIFY_DB_PASSWORD=CHANGE_ME
+MUSIFY_RABBITMQ_PASSWORD=CHANGE_ME
+MUSIFY_S3_ACCESS_KEY=CHANGE_ME
+MUSIFY_S3_SECRET_KEY=CHANGE_ME
+
+PING_DB_PASSWORD=CHANGE_ME
+PING_RABBITMQ_PASSWORD=
+PING_S3_ACCESS_KEY=CHANGE_ME
+PING_S3_SECRET_KEY=CHANGE_ME
+
+DOCKERMANAGER_DB_PASSWORD=CHANGE_ME
+DOCKERMANAGER_RABBITMQ_PASSWORD=
+DOCKERMANAGER_S3_ACCESS_KEY=
+DOCKERMANAGER_S3_SECRET_KEY=

.gitattributes

@@ -0,0 +1,3 @@
+# Shell scripts run inside Linux containers: a CRLF checkout (Windows,
+# core.autocrlf) would break them.
+*.sh text eol=lf

README.md

@@ -8,8 +8,7 @@ provisioning, and keeping in sync) its own copy, they run once, here.
 
 This repo doesn't assume anything about where the other projects are
 checked out. Each one just needs a local path to this repo's checkout (see
-"The Zitadel admin PAT" below), which you set once in that project's own
+"The Zitadel PAT" below), which you set once in that project's own `.env`.
-`.env`.
 
 Each project keeps its own independent, fast `docker compose up` for the
 things that actually are its own: API/worker/gateway containers, database
@@ -23,7 +22,7 @@ needs to know a consuming project exists.
 
 | Here (shared) | Stays in each project |
 |---|---|
-| PostgreSQL (one instance, one database per project) | The project's own EF Core migrations, connection pooling config |
+| PostgreSQL (one instance, one database and role per project) | The project's own EF Core migrations, connection pooling config |
 | Zitadel (OIDC provider) | The project's own OIDC app registration (`zitadel-init`), JWT validation config |
 | SeaweedFS (S3 storage) | The project's own bucket (`seaweedfs-init`), key layout |
 | RabbitMQ (broker) | The project's own exchanges/queues/consumers |
@@ -83,7 +82,7 @@ docker exec infra-rabbitmq rabbitmqctl import_definitions /var/lib/rabbitmq/defs
 
 | Service | Port | Notes |
 |---|---|---|
-| PostgreSQL | `5432` | one instance, one database per project |
+| PostgreSQL | `5432` | one instance, one database and role per project |
 | Zitadel | `8080` | OIDC/OAuth2, console at `/ui/console` |
 | Zitadel login | `3900` | Login V2 UI at `/ui/v2/login` (under `8080`'s hostname in prod) |
 | SeaweedFS | `8333` / `8888` / `9333` | S3 API / filer / master |
@@ -98,27 +97,60 @@ running (in Docker or from the IDE). No shared Docker network is needed in
 development, which is what keeps every project's own `up` independent and
 fast, with this stack as the only prerequisite.
 
-### Shared secrets
+### Per-project credentials
 
-`POSTGRES_USER`/`POSTGRES_PASSWORD`, `S3_ACCESS_KEY`/`S3_SECRET_KEY`,
+No project gets an admin account. `provision` (a one-shot that runs on
-`RABBITMQ_DEFAULT_USER`/`RABBITMQ_DEFAULT_PASS` and `PUBLIC_AUTH_URL` (i.e.
+every `up`, see `provision/`) gives each project listed in `compose.yml`'s
-`http://host.docker.internal:${ZITADEL_EXTERNAL_PORT}` in dev) are the values
+`x-projects` its own credentials, from the `<PROJECT>_*` settings in the env
-every consuming project's own `.env.example` copies and documents as "must
+file:
-match this repo's `.env`". This file is the source of truth for them.
 
-### The Zitadel admin PAT
+| Service | What the project gets | Setting |
+|---|---|---|
+| PostgreSQL | role `<project>`, owner of database `<project>_db`; no other role can connect to it, and it can't connect to any other | `<PROJECT>_DB_PASSWORD` |
+| RabbitMQ | user `<project>`, only on vhost `<project>` | `<PROJECT>_RABBITMQ_PASSWORD` |
+| SeaweedFS | S3 identity limited to the project's bucket (it may create that bucket, nothing else) | `<PROJECT>_S3_ACCESS_KEY` / `_SECRET_KEY` |
+| Zitadel | service account `<project>-provisioner`, owner of the project's own Zitadel project only; its PAT is in `zitadel/.output/<project>/zitadel.pat` | (the project name, in `compose.yml`) |
 
-Zitadel itself writes a service-account personal access token to
+An empty setting means that project doesn't use that service. Each
-`zitadel/.output/admin-sa.pat` when the instance is created. Every consuming project mounts that same folder read-only, via
+project's own `.env.example` copies its values from here and says so; this
-its own `ZITADEL_ADMIN_PAT_DIR` setting, to call the Zitadel management API
+repo's env file is the source of truth. Passwords are re-applied on every
-and create its own project/apps.
+`up`, so rotating one is: change it here and in the project, `up` both.
+
+The admin accounts (`POSTGRES_*`, `RABBITMQ_DEFAULT_*`, `S3_ACCESS_KEY` /
+`S3_SECRET_KEY`, `zitadel/.output/admin-sa.pat`) stay with this stack.
+
+Two limits worth knowing: the Zitadel `PROJECT_OWNER` role can still read
+the organization's users (every project's users are in the one
+organization); and S3 keys must only use `A-Z a-z 0-9 _ + / = . -` (they go
+into SeaweedFS's JSON config as they are).
+
+**Adding a project**: add it to `x-projects`, add its `<PROJECT>_*` lines
+to `provision`'s and `seaweedfs`'s environment in `compose.yml` (with its
+bucket and Zitadel project name), put its values in `.env.example` /
+`.env.prod.example`, and `up`.
+
+**Moving an existing deployment over**: fill in the new `<PROJECT>_*`
+values and `up`. `provision` hands over the existing databases (and every
+table, sequence, type, view and function in them) to each project's role,
+reuses the existing Zitadel projects and apps, and leaves the buckets as
+they are. Then switch each project's env file to its own credentials. The
+only thing that doesn't carry over is RabbitMQ: a project now uses its own
+vhost, so drain its queues on `/` first; its consumers recreate their
+exchanges and queues in the new vhost.
+
+### The Zitadel PAT
+
+Each project's `zitadel-init` mounts **its own** folder,
+`zitadel/.output/<project>/`, read-only (via that project's
+`ZITADEL_PAT_DIR` setting) and uses the PAT in it to create its OIDC apps in
+its own Zitadel project. It can't see or change any other project's.
 
 That setting has no universally correct value. It's a path (relative or
 absolute) to wherever *you* cloned this repo on your machine, from the
 consuming project's `deploy/` folder. Each project's `.env.example` ships
 with an example value that assumes these repos happen to sit as sibling
-folders. If yours don't, change it. Not versioned, and regenerated whenever
+folders. If yours don't, change it. Not versioned; `provision` issues a new
-the Zitadel volume is reset.
+PAT whenever the one there stops working (e.g. after a Zitadel volume reset).
 
 ### The Zitadel login (Login V2)
 
@@ -208,6 +240,6 @@ Infrastructure/
 ├─ edge/                  # shared public nginx + certbot (prod only): compose.yml, conf.d/, snippets/
 ├─ compose.dev.yml        # dev overlay: publishes ports, adds pgAdmin
 ├─ .env.example / .env.prod.example
-├─ postgres-init/         # creates the per-project application databases
+├─ provision/            # per-project credentials: Postgres, RabbitMQ, Zitadel, S3 identities
-└─ zitadel/.output/       # PAT written by Zitadel on init (not versioned)
+└─ zitadel/.output/       # admin PAT + one folder per project with its own PAT (not versioned)
 ```

compose.yml

@@ -22,15 +22,21 @@ x-logging: &default-logging
     max-size: "10m"
     max-file: "3"
 
+# The consuming projects that get their own credentials (see provision/).
+# Adding one: add it here, give it the <PROJECT>_* settings in both
+# `provision` and `seaweedfs` below, and their values in the env files.
+x-projects: &projects musify ping dockermanager
+
 networks:
   infra-net:
     name: infra-net
 
 services:
   # --- Database ---------------------------------------------------------
-  # One instance, one database per consuming project (see postgres-init/).
+  # One instance, one database and one role per consuming project (created
-  # Zitadel provisions its own `zitadel` database itself, from the admin
+  # by `provision` below). Zitadel creates its own `zitadel` database and
-  # credentials below, so it does not need an init script.
+  # user itself, from the admin credentials. The superuser is only for this
+  # stack: no project gets it.
   postgres:
     image: postgres:17-alpine
     container_name: infra-postgres
@@ -39,7 +45,6 @@ services:
       POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
     volumes:
       - postgres_data:/var/lib/postgresql/data
-      - ./postgres-init:/docker-entrypoint-initdb.d:ro
     healthcheck:
       test: ["CMD-SHELL", "pg_isready -U ${POSTGRES_USER}"]
       interval: 10s
@@ -94,8 +99,8 @@ services:
       ZITADEL_FIRSTINSTANCE_ORG_HUMAN_DISPLAYNAME: ${ZITADEL_ADMIN_DISPLAY_NAME:-ZITADEL Admin}
       ZITADEL_FIRSTINSTANCE_ORG_HUMAN_PASSWORD: ${ZITADEL_ADMIN_PASSWORD}
       ZITADEL_FIRSTINSTANCE_ORG_HUMAN_PASSWORDCHANGEREQUIRED: "false"
-      # Service account + PAT written on init; each consuming project's own
+      # Instance-admin service account + PAT written on init. Only `provision`
-      # zitadel-init one-shot uses it to create its own project/apps.
+      # uses it, to give each project its own, project-scoped account.
       ZITADEL_FIRSTINSTANCE_ORG_MACHINE_MACHINE_USERNAME: zitadel-admin-sa
       ZITADEL_FIRSTINSTANCE_ORG_MACHINE_MACHINE_NAME: Admin Service Account
       ZITADEL_FIRSTINSTANCE_ORG_MACHINE_PAT_EXPIRATIONDATE: "2099-01-01T00:00:00Z"
@@ -158,27 +163,31 @@ services:
 
   # --- Storage --------------------------------------------------------------
   # S3-compatible object store shared by every project. All-in-one: master +
-  # volume + filer + S3 gateway. The S3 identity file is rendered from the
+  # volume + filer + S3 gateway. The S3 identities are rendered from the
-  # environment on start so the credentials can never drift from the ones
+  # environment on start (provision/seaweedfs-s3-config.sh): an admin one
-  # handed to consuming projects. No bucket is created here; each
+  # for this stack, and one per project limited to that project's bucket.
-  # consuming project's own seaweedfs-init one-shot creates its own.
+  # No bucket is created here; each consuming project's own seaweedfs-init
+  # one-shot creates its own.
   seaweedfs:
     image: chrislusf/seaweedfs:4.47
     container_name: infra-seaweedfs
+    environment:
+      PROJECTS: *projects
+      S3_ACCESS_KEY: ${S3_ACCESS_KEY}
+      S3_SECRET_KEY: ${S3_SECRET_KEY}
+      MUSIFY_S3_ACCESS_KEY: ${MUSIFY_S3_ACCESS_KEY:-}
+      MUSIFY_S3_SECRET_KEY: ${MUSIFY_S3_SECRET_KEY:-}
+      MUSIFY_S3_BUCKET: webapi-storage
+      PING_S3_ACCESS_KEY: ${PING_S3_ACCESS_KEY:-}
+      PING_S3_SECRET_KEY: ${PING_S3_SECRET_KEY:-}
+      PING_S3_BUCKET: ping-storage
+      DOCKERMANAGER_S3_ACCESS_KEY: ${DOCKERMANAGER_S3_ACCESS_KEY:-}
+      DOCKERMANAGER_S3_SECRET_KEY: ${DOCKERMANAGER_S3_SECRET_KEY:-}
+      DOCKERMANAGER_S3_BUCKET: dockermanager
     entrypoint: ["/bin/sh", "-c"]
     command:
       - |
-        cat > /tmp/s3.conf <<'JSON'
+        sh /provision/seaweedfs-s3-config.sh >/tmp/s3.conf || exit 1
-        {
-          "identities": [
-            {
-              "name": "infra",
-              "credentials": [{ "accessKey": "${S3_ACCESS_KEY}", "secretKey": "${S3_SECRET_KEY}" }],
-              "actions": ["Admin", "Read", "Write"]
-            }
-          ]
-        }
-        JSON
         exec weed server \
           -ip.bind=0.0.0.0 \
           -dir=/data \
@@ -189,6 +198,7 @@ services:
           -s3.config=/tmp/s3.conf
     volumes:
       - seaweedfs_data:/data
+      - ./provision:/provision:ro
     healthcheck:
       test: ["CMD", "wget", "-q", "-O", "-", "http://localhost:9333/cluster/healthz"]
       interval: 10s
@@ -243,6 +253,46 @@ services:
     logging: *default-logging
     restart: unless-stopped
 
+  # --- Per-project credentials ----------------------------------------------
+  # One-shot, runs on every `up` (idempotent): each project's own Postgres
+  # role/database, RabbitMQ user/vhost and Zitadel service account, from the
+  # <PROJECT>_* settings (see provision/provision.sh). An empty setting skips
+  # that service for that project.
+  provision:
+    image: postgres:17-alpine
+    container_name: infra-provision
+    entrypoint: ["/bin/sh", "/provision/provision.sh"]
+    environment:
+      PROJECTS: *projects
+      POSTGRES_USER: ${POSTGRES_USER}
+      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
+      RABBITMQ_DEFAULT_USER: ${RABBITMQ_DEFAULT_USER}
+      RABBITMQ_DEFAULT_PASS: ${RABBITMQ_DEFAULT_PASS}
+      ZITADEL_EXTERNAL_DOMAIN: ${ZITADEL_EXTERNAL_DOMAIN}
+      ZITADEL_PUBLIC_SCHEME: ${ZITADEL_PUBLIC_SCHEME}
+      MUSIFY_DB_PASSWORD: ${MUSIFY_DB_PASSWORD:-}
+      MUSIFY_RABBITMQ_PASSWORD: ${MUSIFY_RABBITMQ_PASSWORD:-}
+      MUSIFY_ZITADEL_PROJECT: Musify
+      PING_DB_PASSWORD: ${PING_DB_PASSWORD:-}
+      PING_RABBITMQ_PASSWORD: ${PING_RABBITMQ_PASSWORD:-}
+      PING_ZITADEL_PROJECT: Ping
+      DOCKERMANAGER_DB_PASSWORD: ${DOCKERMANAGER_DB_PASSWORD:-}
+      DOCKERMANAGER_RABBITMQ_PASSWORD: ${DOCKERMANAGER_RABBITMQ_PASSWORD:-}
+      DOCKERMANAGER_ZITADEL_PROJECT: ""
+    volumes:
+      - ./provision:/provision:ro
+      - ./zitadel/.output:/output
+    depends_on:
+      postgres:
+        condition: service_healthy
+      rabbitmq:
+        condition: service_healthy
+      zitadel:
+        condition: service_healthy
+    networks: [infra-net]
+    logging: *default-logging
+    restart: "no"
+
 volumes:
   postgres_data:
   zitadel_bootstrap:

postgres-init/010-create-app-databases.sql

@@ -1,9 +0,0 @@
--- Runs once, only on first init of the postgres_data volume.
--- Zitadel provisions its own "zitadel" database itself, from the
--- ZITADEL_DATABASE_POSTGRES_ADMIN_* credentials in compose.yml, so it does
--- not need to be created here. This script only creates the plain
--- application databases each project's own connection string points at.
-
-CREATE DATABASE musify_db;
-CREATE DATABASE ping_db;
-CREATE DATABASE dockermanager_db;

provision/provision.sh

@@ -0,0 +1,227 @@
+#!/bin/sh
+# Gives every consuming project its own credentials on the shared services,
+# so no project holds another's data or the admin accounts:
+#
+#   Postgres  role <project>, owner of database <project>_db, which no
+#             other role can connect to
+#   RabbitMQ  user <project> with permissions only on vhost <project>
+#   Zitadel   service account <project>-provisioner, PROJECT_OWNER of the
+#             project's own Zitadel project only; its PAT is written to
+#             /output/<project>/zitadel.pat for that project's zitadel-init
+#
+# (S3 identities are rendered into SeaweedFS's own config when it starts, see
+# seaweedfs-s3-config.sh.)
+#
+# Idempotent: runs on every `up`. A service is only provisioned for a project
+# when its setting is non-empty (no MUSIFY_RABBITMQ_PASSWORD, no RabbitMQ
+# user for Musify). Passwords are re-applied on every run, so changing one
+# in the env file and running `up` again rotates it.
+set -eu -o pipefail
+
+apk add --no-cache curl jq >/dev/null
+
+# Value of the <PROJECT>_<suffix> setting, empty when unset.
+setting() { eval "printf '%s' \"\${$(echo "$1" | tr '[:lower:]' '[:upper:]')_$2:-}\""; }
+
+# --- Postgres ----------------------------------------------------------------
+export PGHOST=postgres PGUSER="$POSTGRES_USER" PGPASSWORD="$POSTGRES_PASSWORD"
+
+# By default every role may connect to every database (PUBLIC has CONNECT),
+# including `zitadel` and `postgres`. Only the owner, roles granted it
+# explicitly (Zitadel's own user on `zitadel`) and the superuser keep it.
+lock_down_postgres() {
+  psql -q -v ON_ERROR_STOP=1 -d postgres <<'SQL'
+SELECT format('REVOKE CONNECT, TEMPORARY ON DATABASE %I FROM PUBLIC', datname)
+FROM pg_database WHERE NOT datistemplate \gexec
+SQL
+  echo "  postgres: no database open to PUBLIC"
+}
+
+provision_postgres() {
+  local project=$1 password=$2 db="$1_db"
+  psql -q -v ON_ERROR_STOP=1 -d postgres -v role="$project" -v pw="$password" -v db="$db" <<'SQL'
+SELECT format('CREATE ROLE %I LOGIN', :'role')
+WHERE NOT EXISTS (SELECT FROM pg_roles WHERE rolname = :'role') \gexec
+ALTER ROLE :"role" WITH LOGIN NOSUPERUSER NOCREATEDB NOCREATEROLE PASSWORD :'pw';
+SELECT format('CREATE DATABASE %I OWNER %I', :'db', :'role')
+WHERE NOT EXISTS (SELECT FROM pg_database WHERE datname = :'db') \gexec
+ALTER DATABASE :"db" OWNER TO :"role";
+REVOKE CONNECT, TEMPORARY ON DATABASE :"db" FROM PUBLIC;
+SQL
+
+  # A database from before this (created by the superuser, as were the
+  # tables its migrations made) is handed over: the project's migrations
+  # have to own what they alter.
+  psql -q -v ON_ERROR_STOP=1 -d "$db" -v role="$project" <<'SQL' >/dev/null
+SELECT set_config('provision.role', :'role', false);
+DO $$
+DECLARE
+  target text := current_setting('provision.role');
+  r record;
+BEGIN
+  FOR r IN
+    SELECT n.nspname FROM pg_namespace n
+    WHERE n.nspname NOT LIKE 'pg\_%' AND n.nspname <> 'information_schema'
+      AND pg_get_userbyid(n.nspowner) NOT IN (target, 'pg_database_owner')
+      AND NOT EXISTS (SELECT FROM pg_depend d WHERE d.classid = 'pg_namespace'::regclass AND d.objid = n.oid AND d.deptype = 'e')
+  LOOP
+    EXECUTE format('ALTER SCHEMA %I OWNER TO %I', r.nspname, target);
+  END LOOP;
+
+  -- Indexes and the sequences behind serial/identity columns follow their
+  -- table; extension members stay with the extension.
+  FOR r IN
+    SELECT c.oid::regclass AS obj, c.relkind FROM pg_class c
+    JOIN pg_namespace n ON n.oid = c.relnamespace
+    WHERE n.nspname NOT LIKE 'pg\_%' AND n.nspname <> 'information_schema'
+      AND c.relkind IN ('r', 'p', 'v', 'm', 'S', 'f')
+      AND pg_get_userbyid(c.relowner) <> target
+      AND NOT EXISTS (SELECT FROM pg_depend d WHERE d.classid = 'pg_class'::regclass AND d.objid = c.oid AND d.deptype IN ('a', 'i', 'e'))
+  LOOP
+    EXECUTE format('ALTER %s %s OWNER TO %I',
+      CASE r.relkind WHEN 'v' THEN 'VIEW' WHEN 'm' THEN 'MATERIALIZED VIEW'
+                     WHEN 'S' THEN 'SEQUENCE' WHEN 'f' THEN 'FOREIGN TABLE' ELSE 'TABLE' END,
+      r.obj, target);
+  END LOOP;
+
+  -- Enums, domains, composite and range types (not a table's row type nor
+  -- array types, which follow their element type).
+  FOR r IN
+    SELECT t.oid::regtype AS obj, t.typtype FROM pg_type t
+    JOIN pg_namespace n ON n.oid = t.typnamespace
+    WHERE n.nspname NOT LIKE 'pg\_%' AND n.nspname <> 'information_schema'
+      AND t.typtype IN ('e', 'd', 'c', 'r', 'm') AND t.typelem = 0
+      AND (t.typrelid = 0 OR (SELECT relkind FROM pg_class WHERE oid = t.typrelid) = 'c')
+      AND pg_get_userbyid(t.typowner) <> target
+      AND NOT EXISTS (SELECT FROM pg_depend d WHERE d.classid = 'pg_type'::regclass AND d.objid = t.oid AND d.deptype IN ('i', 'e'))
+  LOOP
+    EXECUTE format('ALTER %s %s OWNER TO %I', CASE r.typtype WHEN 'd' THEN 'DOMAIN' ELSE 'TYPE' END, r.obj, target);
+  END LOOP;
+
+  FOR r IN
+    SELECT p.oid::regprocedure AS obj FROM pg_proc p
+    JOIN pg_namespace n ON n.oid = p.pronamespace
+    WHERE n.nspname NOT LIKE 'pg\_%' AND n.nspname <> 'information_schema'
+      AND pg_get_userbyid(p.proowner) <> target
+      AND NOT EXISTS (SELECT FROM pg_depend d WHERE d.classid = 'pg_proc'::regclass AND d.objid = p.oid AND d.deptype = 'e')
+  LOOP
+    EXECUTE format('ALTER ROUTINE %s OWNER TO %I', r.obj, target);
+  END LOOP;
+END $$;
+SQL
+  echo "  postgres: role $project owns database $db"
+}
+
+# --- RabbitMQ ----------------------------------------------------------------
+rabbitmq_put() {
+  curl -fsS -o /dev/null -X PUT "http://rabbitmq:15672/api$1" \
+    -u "$RABBITMQ_DEFAULT_USER:$RABBITMQ_DEFAULT_PASS" \
+    -H "Content-Type: application/json" --data-raw "$2"
+}
+
+provision_rabbitmq() {
+  local project=$1 password=$2
+  rabbitmq_put "/vhosts/$project" '{}'
+  rabbitmq_put "/users/$project" "$(jq -n --arg pw "$password" '{password: $pw, tags: ""}')"
+  rabbitmq_put "/permissions/$project/$project" '{"configure": ".*", "write": ".*", "read": ".*"}'
+  echo "  rabbitmq: user $project, only on vhost $project"
+}
+
+# --- Zitadel -----------------------------------------------------------------
+# zitadel <token> <method> <path> [body]: prints the response body; on a
+# non-2xx answer also reports it on stderr and returns non-zero. It calls
+# Zitadel on infra-net, so it presents the public Host/scheme for Zitadel to
+# find the instance (same as zitadel-login).
+zitadel() {
+  local response code
+  response=$(curl -sS -w '\n%{http_code}' -X "$2" "http://zitadel:8080$3" \
+    -H "Authorization: Bearer $1" -H "Content-Type: application/json" \
+    -H "Host: $ZITADEL_EXTERNAL_DOMAIN" -H "X-Forwarded-Proto: $ZITADEL_PUBLIC_SCHEME" \
+    ${4+--data-raw "$4"})
+  code=$(printf '%s\n' "$response" | tail -n1)
+  response=$(printf '%s\n' "$response" | sed '$d')
+  printf '%s\n' "$response"
+  case "$code" in
+    2??) return 0 ;;
+    *) printf 'zitadel: %s %s -> %s %s\n' "$2" "$3" "$code" "$response" >&2; return 1 ;;
+  esac
+}
+
+# Stops the run when a lookup or creation didn't produce a value.
+need() { [ -n "$2" ] && [ "$2" != null ] || { echo "zitadel: got no $1" >&2; exit 1; }; }
+
+provision_zitadel() {
+  local project=$1 zitadel_project=$2 username="$1-provisioner"
+  local project_id user_id member_roles pat_file token state
+
+  project_id=$(zitadel "$ADMIN_PAT" POST /management/v1/projects/_search \
+    "$(jq -n --arg n "$zitadel_project" '{queries: [{nameQuery: {name: $n, method: "TEXT_QUERY_METHOD_EQUALS"}}]}')" \
+    | jq -r '.result // [] | first | .id // empty')
+  if [ -z "$project_id" ]; then
+    project_id=$(zitadel "$ADMIN_PAT" POST /management/v1/projects "$(jq -n --arg n "$zitadel_project" '{name: $n}')" | jq -r .id)
+  fi
+  need "project id for $zitadel_project" "$project_id"
+
+  user_id=$(zitadel "$ADMIN_PAT" POST /management/v1/users/_search \
+    "$(jq -n --arg u "$username" '{queries: [{userNameQuery: {userName: $u, method: "TEXT_QUERY_METHOD_EQUALS"}}]}')" \
+    | jq -r '.result // [] | first | .id // empty')
+  if [ -z "$user_id" ]; then
+    user_id=$(zitadel "$ADMIN_PAT" POST /management/v1/users/machine \
+      "$(jq -n --arg u "$username" --arg p "$zitadel_project" \
+        '{userName: $u, name: ($p + " provisioner"), description: ("zitadel-init of " + $p + ": manages that project only"), accessTokenType: "ACCESS_TOKEN_TYPE_BEARER"}')" \
+      | jq -r .userId)
+  fi
+  need "user id for $username" "$user_id"
+
+  # Owner of this one project (its apps, roles and grants) and nothing else.
+  member_roles=$(zitadel "$ADMIN_PAT" POST "/management/v1/projects/$project_id/members/_search" \
+    "$(jq -n --arg u "$user_id" '{queries: [{userIdQuery: {userId: $u}}]}')" \
+    | jq -c '.result // [] | first | .roles // empty')
+  if [ -z "$member_roles" ]; then
+    zitadel "$ADMIN_PAT" POST "/management/v1/projects/$project_id/members" \
+      "$(jq -n --arg u "$user_id" '{userId: $u, roles: ["PROJECT_OWNER"]}')" >/dev/null
+  elif [ "$member_roles" != '["PROJECT_OWNER"]' ]; then
+    zitadel "$ADMIN_PAT" PUT "/management/v1/projects/$project_id/members/$user_id" '{"roles": ["PROJECT_OWNER"]}' >/dev/null
+  fi
+
+  # Keep a PAT that still works. A new one (they can't be read back) is only
+  # issued when there is none, or it was revoked or lost with a volume reset.
+  pat_file="/output/$project/zitadel.pat"
+  mkdir -p "/output/$project"
+  chmod 700 "/output/$project"
+  if [ -s "$pat_file" ] \
+    && zitadel "$(tr -d '\r\n' <"$pat_file")" GET /auth/v1/users/me 2>/dev/null \
+      | jq -e --arg u "$username" '.user.userName == $u' >/dev/null 2>&1; then
+    state="kept"
+  else
+    token=$(zitadel "$ADMIN_PAT" POST "/management/v1/users/$user_id/pats" '{"expirationDate": "2099-01-01T00:00:00Z"}' | jq -r .token)
+    need "PAT for $username" "$token"
+    (umask 077 && printf '%s\n' "$token" >"$pat_file.tmp")
+    mv "$pat_file.tmp" "$pat_file"
+    state="issued"
+  fi
+  echo "  zitadel: $username owns project \"$zitadel_project\", PAT $state"
+}
+
+# -----------------------------------------------------------------------------
+echo "==> waiting for Zitadel's admin PAT"
+deadline=$(( $(date +%s) + 180 ))
+until [ -s /output/admin-sa.pat ]; do
+  [ "$(date +%s)" -lt "$deadline" ] || { echo "no /output/admin-sa.pat; check 'docker logs infra-zitadel'" >&2; exit 1; }
+  sleep 3
+done
+ADMIN_PAT=$(tr -d '\r\n' </output/admin-sa.pat)
+
+echo "==> shared"
+lock_down_postgres
+
+for project in $PROJECTS; do
+  echo "==> $project"
+  value=$(setting "$project" DB_PASSWORD)
+  [ -z "$value" ] || provision_postgres "$project" "$value"
+  value=$(setting "$project" RABBITMQ_PASSWORD)
+  [ -z "$value" ] || provision_rabbitmq "$project" "$value"
+  value=$(setting "$project" ZITADEL_PROJECT)
+  [ -z "$value" ] || provision_zitadel "$project" "$value"
+done
+echo "==> done"

provision/seaweedfs-s3-config.sh

@@ -0,0 +1,60 @@
+#!/bin/sh
+# Prints SeaweedFS's S3 identity config (JSON) from the environment:
+#
+#   infra      S3_ACCESS_KEY / S3_SECRET_KEY, admin on every bucket (for
+#              operating the stack, not for projects)
+#   <project>  <PROJECT>_S3_ACCESS_KEY / _SECRET_KEY, limited to the bucket
+#              <PROJECT>_S3_BUCKET (it may create it, nothing else)
+#
+# A project without keys gets no identity. Runs in the SeaweedFS container on
+# every start, so the credentials can't drift from the env file.
+set -eu
+
+upper() { echo "$1" | tr '[:lower:]' '[:upper:]'; }
+setting() { eval "printf '%s' \"\${$(upper "$1")_$2:-}\""; }
+
+# The values go into JSON as they are: keep them to characters that need no
+# escaping (what `openssl rand -hex` / -base64 produce).
+check() {
+  case "$2" in
+    '' | *[!A-Za-z0-9_+/=.-]*) echo "seaweedfs-s3-config: $1 must be non-empty and only use A-Z a-z 0-9 _ + / = . -" >&2; exit 1 ;;
+  esac
+}
+
+identity() {
+  printf '    {"name": "%s", "credentials": [{"accessKey": "%s", "secretKey": "%s"}], "actions": [%s]}' "$1" "$2" "$3" "$4"
+}
+
+# SeaweedFS refuses to start (and keeps restarting) on a repeated access key,
+# e.g. two values left at CHANGE_ME: say which ones instead.
+seen=" "
+unique() {
+  case "$seen" in
+    *" $2 "*) echo "seaweedfs-s3-config: $1 repeats an access key already used above; every identity needs its own" >&2; exit 1 ;;
+  esac
+  seen="$seen$2 "
+}
+
+check S3_ACCESS_KEY "$S3_ACCESS_KEY"
+unique S3_ACCESS_KEY "$S3_ACCESS_KEY"
+check S3_SECRET_KEY "$S3_SECRET_KEY"
+printf '{\n  "identities": [\n'
+identity infra "$S3_ACCESS_KEY" "$S3_SECRET_KEY" '"Admin", "Read", "Write"'
+
+for project in $PROJECTS; do
+  access=$(setting "$project" S3_ACCESS_KEY)
+  [ -n "$access" ] || continue
+  secret=$(setting "$project" S3_SECRET_KEY)
+  bucket=$(setting "$project" S3_BUCKET)
+  prefix=$(upper "$project")
+  check "${prefix}_S3_ACCESS_KEY" "$access"
+  check "${prefix}_S3_SECRET_KEY" "$secret"
+  check "${prefix}_S3_BUCKET" "$bucket"
+  unique "${prefix}_S3_ACCESS_KEY" "$access"
+  # Admin:<bucket> is what lets it create (and configure) that one bucket.
+  printf ',\n'
+  identity "$project" "$access" "$secret" \
+    "\"Admin:$bucket\", \"Read:$bucket\", \"Write:$bucket\", \"List:$bucket\", \"Tagging:$bucket\""
+  echo "seaweedfs-s3-config: $project -> bucket $bucket" >&2
+done
+printf '\n  ]\n}\n'