SharedServices/README.md

Infrastructure

The technology services shared across projects (currently Musify, Ping and docker-manager, each its own git repository): PostgreSQL, Zitadel (OIDC), SeaweedFS (S3-compatible storage), RabbitMQ and Jaeger. None of these have any business logic of their own, so instead of each project running (and provisioning, and keeping in sync) its own copy, they run once, here.

This repo doesn't assume anything about where the other projects are checked out. Each one just needs a local path to this repo's checkout (see "The Zitadel PAT" below), which you set once in that project's own .env.

Each project keeps its own independent, fast docker compose up for the things that actually are its own: API/worker/gateway containers, database migrations, its own OIDC apps and storage bucket. Those get created against the shared Zitadel/SeaweedFS below by the project's own zitadel-init and seaweedfs-init one-shot jobs (see e.g. Musify/MusifyM/deploy/compose.yml). A project's dev stack only needs this one already running; nothing here needs to know a consuming project exists.

What's here vs. what stays in each project

Here (shared)	Stays in each project
PostgreSQL (one instance, one database and role per project)	The project's own EF Core migrations, connection pooling config
Zitadel (OIDC provider)	The project's own OIDC app registration (zitadel-init), JWT validation config
SeaweedFS (S3 storage)	The project's own bucket (seaweedfs-init), key layout
RabbitMQ (broker)	The project's own exchanges/queues/consumers
Jaeger (OTLP collector + UI)	The project's own instrumentation
(not shared)	LiveKit (Ping only, a single-project SFU with its own port range)

Running it

cp .env.example .env
docker compose -f compose.yml -f compose.dev.yml up -d

That starts Postgres, Zitadel (and its login UI), SeaweedFS, RabbitMQ and Jaeger, with their ports published on 127.0.0.1 only (DEV_BIND_ADDRESS), so the dev credentials aren't reachable from the LAN. Add --profile tools to also start pgAdmin.

In production, the base compose.yml file is the production shape already. There are no published ports, and everything is reachable only on the internal infra-net network, so no overlay is needed:

cp .env.prod.example .env.prod
# fill in real values, then: chmod 600 .env.prod
docker compose --env-file .env.prod -f compose.yml up -d

A consuming project's own production compose joins infra-net as an external network to reach postgres, zitadel, seaweedfs, rabbitmq and jaeger by name (see Musify's deploy/compose.prod.yml).

Every project on infra-net shares one DNS namespace, and Docker answers a service name with every container that has it, round-robin. Two projects each with a service called api would get each other's traffic. So anything that crosses projects (the edge, or one project calling into this stack) should use the unique container_name (infra-postgres, musify-api, ...), never a generic service name.

RabbitMQ and its hostname

RabbitMQ keeps its data under its node name, rabbit@<hostname>, so the container has a fixed hostname: rabbitmq. Without it every recreated container (an image update, an .env change) came up with a new hostname and an empty broker. A deployment that ran before this was fixed has its data under the old random name; carry the definitions (users, vhosts, permissions, queues, exchanges) over when applying the change, ideally with the queues drained, since queued messages aren't part of the export:

docker exec infra-rabbitmq rabbitmqctl export_definitions /var/lib/rabbitmq/defs.json
docker compose --env-file .env.prod -f compose.yml up -d rabbitmq
docker exec infra-rabbitmq rabbitmqctl import_definitions /var/lib/rabbitmq/defs.json

Ports (development)

Service	Port	Notes
PostgreSQL	5432	one instance, one database and role per project
Zitadel	8080	OIDC/OAuth2, console at /ui/console
Zitadel login	3900	Login V2 UI at /ui/v2/login (under 8080's hostname in prod)
SeaweedFS	8333 / 8888 / 9333	S3 API / filer / master
RabbitMQ	5672 / 15672	AMQP / management UI
Jaeger	16686 / 4317 / 4318	UI / OTLP gRPC / OTLP HTTP
pgAdmin (--profile tools)	5050	Postgres UI

A consuming project reaches all of these at host.docker.internal:<port> from inside its own containers. That's the same address the browser and the host use, so it stays valid whichever way that project's own apps are running (in Docker or from the IDE). No shared Docker network is needed in development, which is what keeps every project's own up independent and fast, with this stack as the only prerequisite.

Per-project credentials

No project gets an admin account. provision (a one-shot that runs on every up, see provision/) gives each project listed in compose.yml's x-projects its own credentials, from the <PROJECT>_* settings in the env file:

Service	What the project gets	Setting
PostgreSQL	role <project>, owner of database <project>_db; no other role can connect to it, and it can't connect to any other	<PROJECT>_DB_PASSWORD
RabbitMQ	user <project>, only on vhost <project>	<PROJECT>_RABBITMQ_PASSWORD
SeaweedFS	S3 identity limited to the project's bucket (it may create that bucket, nothing else)	<PROJECT>_S3_ACCESS_KEY / _SECRET_KEY
Zitadel	service account <project>-provisioner, owner of the project's own Zitadel project only; its PAT is in zitadel/.output/<project>/zitadel.pat	(the project name, in compose.yml)

An empty setting means that project doesn't use that service. Each project's own .env.example copies its values from here and says so; this repo's env file is the source of truth. Passwords are re-applied on every up, so rotating one is: change it here and in the project, up both.

The admin accounts (POSTGRES_*, RABBITMQ_DEFAULT_*, S3_ACCESS_KEY / S3_SECRET_KEY, zitadel/.output/admin-sa.pat) stay with this stack.

Two limits worth knowing: the Zitadel PROJECT_OWNER role can still read the organization's users (every project's users are in the one organization); and S3 keys must only use A-Z a-z 0-9 _ + / = . - (they go into SeaweedFS's JSON config as they are).

Adding a project: add it to x-projects, add its <PROJECT>_* lines to provision's and seaweedfs's environment in compose.yml (with its bucket and Zitadel project name), put its values in .env.example / .env.prod.example, and up.

Moving an existing deployment over: fill in the new <PROJECT>_* values and up. provision hands over the existing databases (and every table, sequence, type, view and function in them) to each project's role, reuses the existing Zitadel projects and apps, and leaves the buckets as they are. Then switch each project's env file to its own credentials. The only thing that doesn't carry over is RabbitMQ: a project now uses its own vhost, so drain its queues on / first; its consumers recreate their exchanges and queues in the new vhost.

The Zitadel PAT

Each project's zitadel-init mounts its own folder, zitadel/.output/<project>/, read-only (via that project's ZITADEL_PAT_DIR setting) and uses the PAT in it to create its OIDC apps in its own Zitadel project. It can't see or change any other project's.

That setting has no universally correct value. It's a path (relative or absolute) to wherever you cloned this repo on your machine, from the consuming project's deploy/ folder. Each project's .env.example ships with an example value that assumes these repos happen to sit as sibling folders. If yours don't, change it. Not versioned; provision issues a new PAT whenever the one there stops working (e.g. after a Zitadel volume reset).

The Zitadel login (Login V2)

Zitadel v4 serves its sign-in pages from a separate container, zitadel-login (Next.js), under /ui/v2/login. Zitadel redirects every login there, so it has to be running for anyone to sign in. It calls the Zitadel API with its own service account (login-client), whose PAT Zitadel writes on init to the zitadel_bootstrap volume, which only zitadel-login mounts.

• Dev: no proxy, so the login has its own port (ZITADEL_LOGIN_HOST_PORT, 3900) and ZITADEL_LOGIN_BASE_URL points the browser at it.
• Prod: the edge routes /ui/v2/login on Zitadel's hostname to infra-zitadel-login, and everything else to Zitadel.
• zitadel and zitadel-login are pinned to the same version; bump both together.

ZITADEL_LOGIN_BASE_URL and the login-client PAT are only applied when the instance is created. On an instance created before zitadel-login existed:

1. In the console, create a service user login-client, give it the instance role IAM Login Client, and create a PAT for it.

2. Copy it into the volume and restart the login:

   docker cp login-client.pat infra-zitadel:/bootstrap/login-client.pat
   docker restart infra-zitadel-login
   

3. If the instance still uses the old login, set Login V2 as required with base URI ${ZITADEL_LOGIN_BASE_URL}/ in the instance's feature settings.

The public edge

edge/compose.yml is the only thing that publishes host ports 80/443: one nginx that terminates TLS for every project's domains (Gitea, Musify, ...) and proxies to their containers by name, plus a certbot container that renews the Let's Encrypt certificates (webroot) and lets nginx reload them every 6 hours. It joins infra-net and the gitea_gitea network (both external, created by the stacks that own them).

docker compose -f edge/compose.yml up -d

• Vhosts live in edge/conf.d/ (one file per project; 00-common.conf has the shared resolver). Every upstream is a set $upstream ..., resolved per request, so a project's stack being down never stops the edge from starting. Upstreams use the target's container_name (see "Running it" above for why). Shared bits are in edge/snippets/. Check a change with docker exec edge-nginx nginx -t, then docker exec edge-nginx nginx -s reload.

• Unknown hostnames hit 01-default.conf: port 80 closes the connection (ACME challenges still work), port 443 refuses the TLS handshake. Nothing reaches a project unless its vhost names the host.

• TLS settings (snippets/tls.conf) follow Mozilla's intermediate profile and send HSTS with a 1-day max-age. Raise it once every hostname is confirmed to work over https: browsers remember it, so a long value can't be taken back quickly.

• Certificates are state, not versioned (edge/certbot/). To issue one for new hostnames (their DNS has to already point at the host):

  docker compose -f edge/compose.yml run --rm --entrypoint certbot certbot \
    certonly --webroot -w /var/www/certbot -d host1.example.com -d host2.example.com \
    --non-interactive --agree-tos -m you@example.com
  

  Then add the server blocks pointing at /etc/letsencrypt/live/<first -d name>/.

Resetting

docker compose -f compose.yml -f compose.dev.yml down          # stop, keep data
docker compose -f compose.yml -f compose.dev.yml down -v       # stop, wipe volumes

Wiping the volumes resets Zitadel (all projects/apps/users) and SeaweedFS (all buckets). Every consuming project's own zitadel-init/seaweedfs-init re-creates its piece on the next up, since both are idempotent.

Layout

Infrastructure/
├─ compose.yml            # postgres, zitadel (+ login), seaweedfs, rabbitmq, jaeger: the prod shape
├─ edge/                  # shared public nginx + certbot (prod only): compose.yml, conf.d/, snippets/
├─ compose.dev.yml        # dev overlay: publishes ports, adds pgAdmin
├─ .env.example / .env.prod.example
├─ provision/            # per-project credentials: Postgres, RabbitMQ, Zitadel, S3 identities
└─ zitadel/.output/       # admin PAT + one folder per project with its own PAT (not versioned)