Translate docs to English, rewrite README, drop design handoff folder
Translate all of docs/ from Spanish to English and rewrite the root README as a short project overview instead of a technical run guide. Remove web-player/design_handoff_musify_player, an old design-tool export that was no longer needed.
This commit is contained in:
+32
-28
@@ -1,47 +1,51 @@
|
||||
# Musify — Documentación
|
||||
# Musify documentation
|
||||
|
||||
Musify es un backend de streaming de música: subir pistas, procesarlas (transcodificar el audio a `.m4a` y generar miniaturas), organizarlas en playlists y reproducirlas en streaming sin que los bytes pasen por la API.
|
||||
Musify is a music streaming backend: upload tracks, process them (transcode
|
||||
the audio to `.m4a`, generate thumbnails), organize them into playlists, and
|
||||
stream them without the bytes ever passing through the API.
|
||||
|
||||
Esta carpeta explica **qué hace cada pieza, su propósito y por qué está diseñada así**.
|
||||
This folder explains **what each piece does, its purpose, and why it's built
|
||||
this way**.
|
||||
|
||||
## Índice
|
||||
## Index
|
||||
|
||||
- [architecture.md](architecture.md) — Visión global: componentes, cómo encajan y los flujos principales.
|
||||
- [projects.md](projects.md) — Qué es y para qué sirve cada proyecto/solución.
|
||||
- [authentication.md](authentication.md) — Zitadel (OIDC), validación de JWT, provisioning/sync de usuarios.
|
||||
- [storage.md](storage.md) — SeaweedFS (S3), estructura de claves, URLs prefirmadas, qué es público y qué privado.
|
||||
- [streaming.md](streaming.md) — Streaming del audio (`.m4a` por HTTP range) con stream-ticket (RS256) y el reverse proxy (StreamingGateway).
|
||||
- [media-processing.md](media-processing.md) — El worker y los workflows de MassTransit (transcode de audio, miniaturas).
|
||||
- [development.md](development.md) — Cómo levantar todo en local: contenedores, puertos y comandos.
|
||||
- [../deploy/README.md](../deploy/README.md) — El deploy en detalle: compose, scripts y producción.
|
||||
- [architecture.md](architecture.md): overview, components, how they fit together, and the main flows.
|
||||
- [projects.md](projects.md): what each project/solution is and what it's for.
|
||||
- [authentication.md](authentication.md): Zitadel (OIDC), JWT validation, user provisioning/sync.
|
||||
- [storage.md](storage.md): SeaweedFS (S3), key layout, presigned URLs, what's public vs. private.
|
||||
- [streaming.md](streaming.md): audio streaming (`.m4a` over HTTP range) with stream tickets (RS256) and the reverse proxy (StreamingGateway).
|
||||
- [media-processing.md](media-processing.md): the worker and its MassTransit workflows (audio transcoding, thumbnails).
|
||||
- [development.md](development.md): running everything locally, containers, ports, commands.
|
||||
- [../deploy/README.md](../deploy/README.md): deployment in detail, compose, scripts, and production.
|
||||
|
||||
## Mapa de componentes (alto nivel)
|
||||
## Component map (high level)
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────┐
|
||||
Cliente ───► │ Musify.Api (:5111) API REST + emite tickets │
|
||||
Client ───► │ Musify.Api (:5111) REST API + issues tickets│
|
||||
└─────────────────────────────────────────────┘
|
||||
│ EF Core │ presigned (subidas) │ MassTransit (eventos)
|
||||
│ EF Core │ presigned (uploads) │ MassTransit (events)
|
||||
▼ ▼ ▼
|
||||
PostgreSQL SeaweedFS S3 (:8333) RabbitMQ ──► Worker (procesa)
|
||||
(metadatos) (objetos) audio→.m4a, imágenes
|
||||
PostgreSQL SeaweedFS S3 (:8333) RabbitMQ ──► Worker (processes)
|
||||
(metadata) (objects) audio→.m4a, images
|
||||
▲
|
||||
Cliente ───► StreamingGateway (:8081) ┘ sirve audio .m4a (valida ticket, proxy al filer)
|
||||
Client ───► StreamingGateway (:8081) ┘ serves .m4a audio (validates ticket, proxies to the filer)
|
||||
|
||||
Login: Cliente ◄──► Zitadel (:8080, OIDC) → Musify.Api valida el JWT
|
||||
Login: Client ◄──► Zitadel (:8080, OIDC) → Musify.Api validates the JWT
|
||||
```
|
||||
|
||||
## Stack
|
||||
|
||||
- **.NET 10**, C#. Minimal APIs (`Musify.Api`), Worker Service (`Musify.Worker`), ASP.NET + YARP (`Musify.StreamingGateway`).
|
||||
- **CQRS** con [Mediator] (source-generator) y resultados con **ErrorOr**.
|
||||
- **EF Core 10** + **PostgreSQL** (metadatos).
|
||||
- **MassTransit 8** + **RabbitMQ** (procesado asíncrono con routing slips/sagas).
|
||||
- **SeaweedFS** (almacenamiento de objetos compatible con S3, vía AWSSDK.S3).
|
||||
- **ffmpeg** (transcode de audio a AAC/`.m4a`), **SixLabors.ImageSharp** (miniaturas).
|
||||
- **Zitadel** (OIDC/OAuth2) para identidad.
|
||||
- **CQRS** with [Mediator] (source generator) and results via **ErrorOr**.
|
||||
- **EF Core 10** + **PostgreSQL** (metadata).
|
||||
- **MassTransit 8** + **RabbitMQ** (async processing with routing slips/sagas).
|
||||
- **SeaweedFS** (S3-compatible object storage, via AWSSDK.S3).
|
||||
- **ffmpeg** (audio transcoding to AAC/`.m4a`), **SixLabors.ImageSharp** (thumbnails).
|
||||
- **Zitadel** (OIDC/OAuth2) for identity.
|
||||
|
||||
## Estructura del repositorio
|
||||
## Repository layout
|
||||
|
||||
Un solo repo: `backend/` (los seis proyectos .NET), `web-player/` (cliente web
|
||||
SvelteKit), `deploy/` (stacks de Docker Compose y scripts) y `docs/`.
|
||||
A single repo: `backend/` (the six .NET projects), `web-player/` (the
|
||||
SvelteKit web client), `deploy/` (Docker Compose stacks and scripts), and
|
||||
`docs/`.
|
||||
|
||||
+41
-41
@@ -1,55 +1,55 @@
|
||||
# Arquitectura
|
||||
# Architecture
|
||||
|
||||
## Principios
|
||||
## Principles
|
||||
|
||||
- **Clean architecture / capas** en `backend/`: el dominio y la aplicación no dependen de infraestructura; la infraestructura implementa las interfaces (`Application/Contracts`).
|
||||
- **CQRS**: cada operación es un `Command` o `Query` con su handler (Mediator). Los handlers devuelven `Ardalis.Result` (Ok/NotFound/Unauthorized/Conflict/Invalid…), que la API traduce a códigos HTTP.
|
||||
- **Los bytes pesados no pasan por la API**: subidas con URLs prefirmadas (cliente → S3) y reproducción vía un reverse proxy dedicado (gateway → S3). La API solo mueve metadatos y emite permisos.
|
||||
- **Procesado asíncrono**: crear una pista no transcodifica en caliente; publica un evento y un worker hace el trabajo pesado con compensación (sagas/routing slips de MassTransit).
|
||||
- **Clean architecture / layers** in `backend/`: the domain and application layers don't depend on infrastructure; infrastructure implements their interfaces (`Application/Contracts`).
|
||||
- **CQRS**: every operation is a `Command` or `Query` with its own handler (Mediator). Handlers return `Ardalis.Result` (Ok/NotFound/Unauthorized/Conflict/Invalid…), which the API translates into HTTP status codes.
|
||||
- **Heavy bytes never touch the API**: uploads use presigned URLs (client → S3) and playback goes through a dedicated reverse proxy (gateway → S3). The API only moves metadata and issues permissions.
|
||||
- **Async processing**: creating a track doesn't transcode it on the spot. It publishes an event, and a worker does the heavy lifting, with compensation (MassTransit sagas/routing slips).
|
||||
|
||||
## Componentes
|
||||
## Components
|
||||
|
||||
| Componente | Responsabilidad | Por qué separado |
|
||||
| Component | Responsibility | Why it's separate |
|
||||
|---|---|---|
|
||||
| **Musify.Api** | API REST (usuarios, pistas, playlists), autoriza, emite presigned URLs y stream-tickets | Punto de entrada fino; no procesa medios ni sirve bytes |
|
||||
| **Domain/Application/Infrastructure** | Dominio + casos de uso + infraestructura compartida (EF, S3, MassTransit, ffmpeg) | Lógica reutilizada por la API y el Worker |
|
||||
| **StreamingGateway** | Reverse proxy (YARP) que valida el ticket y sirve el audio (`.m4a`) desde SeaweedFS | Escala con el tráfico de bytes, independiente de la API; SeaweedFS queda privado detrás |
|
||||
| **Worker** | Consume eventos de MassTransit: transcodifica audio a `.m4a` y genera miniaturas | Trabajo CPU/IO intensivo fuera del request HTTP |
|
||||
| **PostgreSQL** | Metadatos (usuarios, pistas, playlists, upload intents) | — |
|
||||
| **RabbitMQ** | Cola de eventos + outbox transaccional | Desacopla creación de procesado |
|
||||
| **SeaweedFS** | Almacenamiento de objetos (S3 + filer) | Guarda originales y derivados (audio `.m4a`, miniaturas) |
|
||||
| **Zitadel** | Identidad (OIDC/OAuth2) | Login y emisión de JWT |
|
||||
| **Musify.Api** | REST API (users, tracks, playlists), authorizes requests, issues presigned URLs and stream tickets | A thin entry point, it doesn't process media or serve bytes |
|
||||
| **Domain/Application/Infrastructure** | Domain + use cases + shared infrastructure (EF, S3, MassTransit, ffmpeg) | Logic shared between the API and the Worker |
|
||||
| **StreamingGateway** | Reverse proxy (YARP) that validates the ticket and serves the audio (`.m4a`) from SeaweedFS | Scales with byte traffic independently of the API, and keeps SeaweedFS private behind it |
|
||||
| **Worker** | Consumes MassTransit events: transcodes audio to `.m4a` and generates thumbnails | Keeps CPU/IO-heavy work off the HTTP request path |
|
||||
| **PostgreSQL** | Metadata (users, tracks, playlists, upload intents) | n/a |
|
||||
| **RabbitMQ** | Event queue + transactional outbox | Decouples creation from processing |
|
||||
| **SeaweedFS** | Object storage (S3 + filer) | Stores originals and derivatives (`.m4a` audio, thumbnails) |
|
||||
| **Zitadel** | Identity (OIDC/OAuth2) | Login and JWT issuance |
|
||||
|
||||
## Flujos principales
|
||||
## Main flows
|
||||
|
||||
### 1. Subida de una pista
|
||||
1. Cliente pide `POST /tracks/upload-urls` → la API crea **UploadIntents** (reservas) y devuelve **URLs prefirmadas PUT** (imagen + audio).
|
||||
2. Cliente sube los ficheros **directo a SeaweedFS** con esas URLs (no pasan por la API).
|
||||
3. Cliente llama `POST /tracks` con los `intentId`s → la API valida los intents, crea el `Track` (+ `UserHasTrack`) y publica `CreateTrackResourcesEvent`.
|
||||
4. El **Worker** consume el evento: mueve los originales a su sitio, genera miniaturas y transcodifica el audio a `.m4a`, y actualiza el estado del track.
|
||||
### 1. Uploading a track
|
||||
1. The client requests `POST /tracks/upload-urls`. The API creates **UploadIntents** (reservations) and returns **presigned PUT URLs** (image + audio).
|
||||
2. The client uploads the files **directly to SeaweedFS** using those URLs (never through the API).
|
||||
3. The client calls `POST /tracks` with the `intentId`s. The API validates the intents, creates the `Track` (+ `UserHasTrack`), and publishes `CreateTrackResourcesEvent`.
|
||||
4. The **Worker** consumes the event: moves the originals into place, generates thumbnails, transcodes the audio to `.m4a`, and updates the track's status.
|
||||
|
||||
Ver detalle en [media-processing.md](media-processing.md) y [storage.md](storage.md).
|
||||
See [media-processing.md](media-processing.md) and [storage.md](storage.md) for details.
|
||||
|
||||
### 2. Reproducción (streaming)
|
||||
1. Cliente pide `GET /tracks/{id}/stream` (autenticado) → la API autoriza y devuelve `{ manifestUrl, ticket }` (URL al `.m4a` + ticket RS256 acotado a la carpeta de esa pista).
|
||||
2. El reproductor abre la URL del audio añadiendo `?t=<ticket>` (o cabecera `X-Stream-Ticket`) y descarga por HTTP range.
|
||||
3. El **StreamingGateway** valida el ticket y reenvía al filer de SeaweedFS; los bytes van storage → cliente.
|
||||
### 2. Playback (streaming)
|
||||
1. The client requests `GET /tracks/{id}/stream` (authenticated). The API authorizes it and returns `{ manifestUrl, ticket }` (a URL to the `.m4a` plus an RS256 ticket scoped to that track's folder).
|
||||
2. The player opens the audio URL with `?t=<ticket>` appended (or an `X-Stream-Ticket` header) and downloads it via HTTP range requests.
|
||||
3. The **StreamingGateway** validates the ticket and forwards the request to the SeaweedFS filer. The bytes flow storage → client.
|
||||
|
||||
Ver [streaming.md](streaming.md).
|
||||
See [streaming.md](streaming.md).
|
||||
|
||||
### 3. Autenticación
|
||||
1. Cliente hace login OIDC contra **Zitadel** y obtiene un access token (JWT).
|
||||
2. La Musify.Api valida la firma/iss/aud del JWT.
|
||||
3. En cada token validado, se **sincroniza** el usuario local (crea/actualiza nombre y foto) — provisioning just-in-time.
|
||||
### 3. Authentication
|
||||
1. The client logs in via OIDC against **Zitadel** and gets an access token (JWT).
|
||||
2. Musify.Api validates the JWT's signature, issuer and audience.
|
||||
3. On every validated token, the local user is **synced** (name and picture created or updated). This is just-in-time provisioning.
|
||||
|
||||
Ver [authentication.md](authentication.md).
|
||||
See [authentication.md](authentication.md).
|
||||
|
||||
## Modelo de datos (entidades)
|
||||
## Data model (entities)
|
||||
|
||||
- **User** — `Id` es el `sub` numérico de Zitadel (`long`), más nombre y `ProfilePictureUrl`.
|
||||
- **Track** — pista: nombres de originales/derivados, `AudioFolderName` (carpeta del `.m4a` transcodificado), estados de procesado (`ProcessingStatus`), `LifeCycleStatus`.
|
||||
- **PlayList** — lista del usuario (`UserId`).
|
||||
- **PlayListHasTrack** — pistas dentro de una playlist (con `Position`).
|
||||
- **UserHasTrack** — propiedad: qué usuario posee qué pista.
|
||||
- **UploadIntent** — reserva de subida (clave, bucket, tamaño esperado, estado, expiración) para validar y aplicar cuota.
|
||||
- **Upload** — registro de objetos subidos/transferidos al storage.
|
||||
- **User**: `Id` is Zitadel's numeric `sub` (`long`), plus a name and `ProfilePictureUrl`.
|
||||
- **Track**: original/derivative file names, `AudioFolderName` (the transcoded `.m4a`'s folder), processing state (`ProcessingStatus`), `LifeCycleStatus`.
|
||||
- **PlayList**: a user's playlist (`UserId`).
|
||||
- **PlayListHasTrack**: the tracks inside a playlist (with `Position`).
|
||||
- **UserHasTrack**: ownership, which user owns which track.
|
||||
- **UploadIntent**: an upload reservation (key, bucket, expected size, status, expiration) used for validation and quota.
|
||||
- **Upload**: a record of objects uploaded or transferred to storage.
|
||||
|
||||
+26
-26
@@ -1,40 +1,40 @@
|
||||
# Autenticación e identidad
|
||||
# Authentication and identity
|
||||
|
||||
## Proveedor: Zitadel (OIDC/OAuth2)
|
||||
## Provider: Zitadel (OIDC/OAuth2)
|
||||
|
||||
El login y la emisión de tokens los hace **Zitadel** (no la API). La API es un **resource server**: solo **valida** el access token (JWT) que le llega en `Authorization: Bearer`.
|
||||
Login and token issuance are handled by **Zitadel**, not the API. The API is a **resource server**: it only **validates** the access token (JWT) it receives in `Authorization: Bearer`.
|
||||
|
||||
- Config en la sección `Authentication` (`AuthenticationConfiguration`): metadata address, issuer, audience, client id, endpoints de authorize/token, scopes.
|
||||
- `JwtBearerOptionsSetup` configura la validación (issuer, audience, claves vía metadata OIDC).
|
||||
- En dev, `RequireHttpsMetadata = false`.
|
||||
- Configured under the `Authentication` section (`AuthenticationConfiguration`): metadata address, issuer, audience, client id, authorize/token endpoints, scopes.
|
||||
- `JwtBearerOptionsSetup` sets up validation (issuer, audience, keys via OIDC metadata).
|
||||
- In dev, `RequireHttpsMetadata = false`.
|
||||
|
||||
### Requisito clave en la app de Zitadel
|
||||
- **Auth Token Type = JWT** (si no, Zitadel emite un token **opaco** sin puntos y `JwtBearer` no lo puede validar → 401 en todo lo protegido).
|
||||
- **User Info inside Token** activado si quieres que el access token traiga claims de perfil (`name`, `email`, `picture`). Por defecto el access token es mínimo (`sub`, `aud`, `iss`, `exp`…), sin perfil.
|
||||
### Key requirement in the Zitadel app
|
||||
- **Auth Token Type = JWT** (otherwise Zitadel issues an **opaque** token with no dots, which `JwtBearer` can't validate → 401 on everything protected).
|
||||
- **User Info inside Token** enabled if you want the access token to carry profile claims (`name`, `email`, `picture`). By default the access token is minimal (`sub`, `aud`, `iss`, `exp`…), with no profile info.
|
||||
|
||||
## El `sub` es numérico → `User.Id` es `long`
|
||||
## `sub` is numeric → `User.Id` is `long`
|
||||
|
||||
El `sub` de Zitadel es un entero de 64 bits en string (p. ej. `371953080444977155`), **no un GUID**. Por eso:
|
||||
- `User.Id` y todas las FKs de usuario (`PlayList.UserId`, `UserHasTrack.UserId`, `UploadIntent.UserId`) son **`long`**.
|
||||
- `User.Id` lleva `[DatabaseGenerated(None)]` para que EF **no** lo autogenere: se asigna el `sub`.
|
||||
- `CurrentUser.Id` es `long?` (parsea con `long.TryParse`); `RequiredId` lanza si falta.
|
||||
Zitadel's `sub` is a 64-bit integer encoded as a string (e.g. `371953080444977155`), **not a GUID**. Because of that:
|
||||
- `User.Id` and every user-related FK (`PlayList.UserId`, `UserHasTrack.UserId`, `UploadIntent.UserId`) are **`long`**.
|
||||
- `User.Id` has `[DatabaseGenerated(None)]` so EF doesn't autogenerate it: it's assigned from the `sub`.
|
||||
- `CurrentUser.Id` is `long?` (parsed with `long.TryParse`); `RequiredId` throws if it's missing.
|
||||
|
||||
**Por qué**: el código antiguo hacía `Guid.Parse(sub)` y reventaba (FormatException) → 401 en cada endpoint protegido. Se decidió guardar el `sub` tal cual como número.
|
||||
**Why**: the original code did `Guid.Parse(sub)`, which threw a `FormatException` → 401 on every protected endpoint. The fix was to store the `sub` as-is, as a number.
|
||||
|
||||
## Provisioning / sincronización de usuarios
|
||||
## User provisioning / sync
|
||||
|
||||
En cada token validado, `JwtBearerEventsHandler.TokenValidated`:
|
||||
1. Lee `sub` (id), `name`/`preferred_username`/`email` (con fallback) y el claim OIDC `picture`.
|
||||
2. Envía un `SyncUserCommand`.
|
||||
On every validated token, `JwtBearerEventsHandler.TokenValidated`:
|
||||
1. Reads `sub` (id), `name`/`preferred_username`/`email` (with fallback), and the OIDC `picture` claim.
|
||||
2. Sends a `SyncUserCommand`.
|
||||
|
||||
`SyncUserCommandHandler` hace un **upsert idempotente**:
|
||||
- Si el usuario no existe → lo crea.
|
||||
- Si existe → actualiza nombre/apellidos/`ProfilePictureUrl` **solo si cambiaron** (escribe en DB únicamente cuando hay cambios).
|
||||
`SyncUserCommandHandler` does an **idempotent upsert**:
|
||||
- If the user doesn't exist yet → creates it.
|
||||
- If it exists → updates the name and `ProfilePictureUrl` **only if they changed** (a DB write only happens when something is different).
|
||||
|
||||
**Por qué upsert con "escribe solo si cambió"**: `TokenValidated` se ejecuta en **cada request**; sin ese guardado condicional estaríamos escribiendo en cada petición. Y debe ser resiliente: si el provisioning falla, se loguea pero **no** tumba la autenticación.
|
||||
**Why "write only if changed"**: `TokenValidated` runs on **every request**; without that guard we'd be writing to the DB on every single call. It also needs to be resilient: if provisioning fails, it's logged but authentication **doesn't** fail because of it.
|
||||
|
||||
> Nota: `CreateUserCommand` (endpoint manual `POST /users`) sigue siendo create-only; el sync del login usa `SyncUserCommand`.
|
||||
> Note: `CreateUserCommand` (the manual `POST /users` endpoint) is still create-only; login sync uses `SyncUserCommand`.
|
||||
|
||||
## Foto de perfil
|
||||
## Profile picture
|
||||
|
||||
`User.ProfilePictureUrl` se rellena desde el claim `picture` del token. Si Zitadel no mete la info de perfil en el access token (ver arriba), llega `null` — el código es correcto, pero no hay nada que sincronizar hasta activar esa opción.
|
||||
`User.ProfilePictureUrl` is filled from the token's `picture` claim. If Zitadel doesn't include profile info in the access token (see above), it arrives as `null`. The code is correct, there's just nothing to sync until that option is turned on.
|
||||
|
||||
+82
-81
@@ -1,21 +1,21 @@
|
||||
# Entorno de desarrollo
|
||||
# Development environment
|
||||
|
||||
Cómo levantar todo en local y probarlo. El detalle completo del deploy (y el de
|
||||
producción) está en [deploy/README.md](../deploy/README.md).
|
||||
How to run everything locally and test it. The full deployment details (dev
|
||||
and prod) live in [deploy/README.md](../deploy/README.md).
|
||||
|
||||
## Arranque
|
||||
## Startup
|
||||
|
||||
```sh
|
||||
docker compose -f deploy/compose.yml -f deploy/compose.dev.yml up -d
|
||||
```
|
||||
|
||||
Ese único comando levanta la infraestructura y corre los jobs *one-shot* de
|
||||
bootstrap: genera las claves RS256 de stream-ticket en `deploy/keys/`, aplica
|
||||
las migraciones EF Core y provisiona Zitadel (proyecto + apps OIDC), dejando
|
||||
los client ids en `deploy/.env`. Detalle completo, incluido cómo arrancar las
|
||||
apps en Docker con `--profile apps`, en [deploy/README.md](../deploy/README.md).
|
||||
That one command brings up the infrastructure and runs the one-shot bootstrap
|
||||
jobs: generates the RS256 stream-ticket keys in `deploy/keys/`, applies the
|
||||
EF Core migrations, and provisions Zitadel (project + OIDC apps), writing the
|
||||
client ids into `deploy/.env`. Full details, including how to run the apps in
|
||||
Docker with `--profile apps`, are in [deploy/README.md](../deploy/README.md).
|
||||
|
||||
## Aplicaciones
|
||||
## Applications
|
||||
|
||||
```powershell
|
||||
dotnet run --project backend/Hosts/Musify.Api # :5111
|
||||
@@ -24,45 +24,46 @@ dotnet run --project backend/Hosts/Musify.Worker # background
|
||||
npm --prefix web-player run dev # :5173
|
||||
```
|
||||
|
||||
## Mapa de puertos
|
||||
## Port map
|
||||
|
||||
| Servicio | Puerto |
|
||||
| Service | Port |
|
||||
|---|---|
|
||||
| Musify.Api | 5111 (`/scalar/v1`, `/openapi/v1.json`) |
|
||||
| Musify.StreamingGateway | 8081 (`/health`, `/media/...`) |
|
||||
| web-player | 5173 (`npm run dev`) o 3000 (contenedor) |
|
||||
| web-player | 5173 (`npm run dev`) or 3000 (container) |
|
||||
| SeaweedFS S3 / filer / master | 8333 / 8888 / 9333 |
|
||||
| Zitadel | 8080 |
|
||||
| PostgreSQL | 59000 (→ 5432 en el contenedor) |
|
||||
| PostgreSQL | 59000 (→ 5432 in the container) |
|
||||
| RabbitMQ AMQP / management | 5672 / 15672 |
|
||||
| Jaeger UI / OTLP | 16686 / 4317 |
|
||||
| pgAdmin (`-Tools`) | 5050 |
|
||||
|
||||
DB: `musify_db`, usuario `postgres`/`postgres`. Bucket S3: `webapi-storage`
|
||||
(credenciales `admin_access_key`/`admin_secret_key`). Todos los valores salen de
|
||||
DB: `musify_db`, user `postgres`/`postgres`. S3 bucket: `webapi-storage`
|
||||
(credentials `admin_access_key`/`admin_secret_key`). All of these come from
|
||||
`deploy/.env`.
|
||||
|
||||
## Migraciones
|
||||
## Migrations
|
||||
|
||||
El servicio `migrate` de `deploy/compose.yml` las aplica en cada `docker
|
||||
compose up -d` (no-op si el esquema ya está al día). A mano, tras cambios de
|
||||
esquema, contra la base local:
|
||||
The `migrate` service in `deploy/compose.yml` applies them on every `docker
|
||||
compose up -d` (a no-op if the schema is already current). To run them by
|
||||
hand after a schema change, against the local database:
|
||||
|
||||
```powershell
|
||||
dotnet ef database update --project backend/Core/Musify.Infrastructure --startup-project backend/Core/Musify.Infrastructure --context Database
|
||||
```
|
||||
|
||||
`Musify.Infrastructure` es a la vez el proyecto de migraciones y el de arranque:
|
||||
tiene el `IDesignTimeDbContextFactory` y el paquete `EFCore.Design`. La cadena
|
||||
de conexión se lee de sus user-secrets, o de la variable de entorno
|
||||
`Database__ConnectionString` si no hay user-secrets (así es como la usa el
|
||||
contenedor `migrate`, ver `backend/Musify.Migrator.Dockerfile`).
|
||||
`Musify.Infrastructure` is both the migrations project and the startup
|
||||
project: it has the `IDesignTimeDbContextFactory` and the `EFCore.Design`
|
||||
package. The connection string is read from its user-secrets, or from the
|
||||
`Database__ConnectionString` environment variable if there are no
|
||||
user-secrets (which is how the `migrate` container uses it, see
|
||||
`backend/Musify.Migrator.Dockerfile`).
|
||||
|
||||
## Stream-tickets (claves RS256)
|
||||
## Stream tickets (RS256 keys)
|
||||
|
||||
La API firma los stream-tickets con la clave privada y el gateway los valida con
|
||||
la pública. El servicio `keys-init` de `deploy/compose.yml` las genera en
|
||||
`deploy/keys/` si no existen; a mano:
|
||||
The API signs stream tickets with the private key, and the gateway validates
|
||||
them with the public one. The `keys-init` service in `deploy/compose.yml`
|
||||
generates them in `deploy/keys/` if they don't exist yet; to do it by hand:
|
||||
|
||||
```powershell
|
||||
openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:2048 -out deploy/keys/stream_private.pem
|
||||
@@ -78,74 +79,74 @@ openssl rsa -in deploy/keys/stream_private.pem -pubout -out deploy/keys/stream_p
|
||||
dotnet test backend/Musify.slnx
|
||||
```
|
||||
|
||||
`backend/Tests/` (carpeta `/Tests/` en el `.slnx`) tiene cinco proyectos xUnit:
|
||||
`backend/Tests/` (the `/Tests/` folder in the `.slnx`) has five xUnit projects:
|
||||
|
||||
| Proyecto | Cubre | Docker |
|
||||
| Project | Covers | Docker |
|
||||
|---|---|---|
|
||||
| `Musify.Domain.Tests` | Lógica de los value objects (`TrackAudio`/`TrackPictures`: `IsProcessed`, `IsFailed`, `IsInProgress`). Las entidades son en su mayoría anémicas — sin comportamiento propio, nada más que testear ahí. | No |
|
||||
| `Musify.Application.Tests` | Los ~45 handlers y servicios de `Musify.Application` (Albums, Tracks, PlayLists, Users, Mixes). | No |
|
||||
| `Musify.Infrastructure.Tests` | `Database`/migraciones, `StorageService`, `StreamTicketService`, `AuditableEntityInterceptor`, `PictureService`, `SingleFlightCache`, `PlayListPresetSeeder`. | **Sí** |
|
||||
| `Musify.Api.Tests` | Endpoints HTTP reales (routing, auth, `ValidationFilter`, mapeo `ErrorOr`→HTTP) vía `WebApplicationFactory`, más `ErrorOrHttpExtensions`/`CurrentUser`/validators FluentValidation en aislado. | **Sí** |
|
||||
| `Musify.StreamingGateway.Tests` | `TicketValidator` (RS256) y `TicketValidationMiddleware` (traversal, límites de prefijo, extracción de ticket). | No |
|
||||
| `Musify.Domain.Tests` | Value object logic (`TrackAudio`/`TrackPictures`: `IsProcessed`, `IsFailed`, `IsInProgress`). Entities are mostly anemic, with no behavior of their own beyond that. | No |
|
||||
| `Musify.Application.Tests` | The ~45 handlers and services in `Musify.Application` (Albums, Tracks, PlayLists, Users, Mixes). | No |
|
||||
| `Musify.Infrastructure.Tests` | `Database`/migrations, `StorageService`, `StreamTicketService`, `AuditableEntityInterceptor`, `PictureService`, `SingleFlightCache`, `PlayListPresetSeeder`. | **Yes** |
|
||||
| `Musify.Api.Tests` | Real HTTP endpoints (routing, auth, `ValidationFilter`, `ErrorOr`→HTTP mapping) via `WebApplicationFactory`, plus `ErrorOrHttpExtensions`/`CurrentUser`/FluentValidation validators in isolation. | **Yes** |
|
||||
| `Musify.StreamingGateway.Tests` | `TicketValidator` (RS256) and `TicketValidationMiddleware` (traversal, prefix limits, ticket extraction). | No |
|
||||
|
||||
`Musify.Worker` no tiene proyecto de test: su `Program.cs` es puro *wiring* de
|
||||
DI/host (Hangfire, MassTransit, jobs) — la lógica real que ejecuta vive en
|
||||
`Musify.Infrastructure`, ya cubierta ahí.
|
||||
`Musify.Worker` has no test project: its `Program.cs` is pure DI/host wiring
|
||||
(Hangfire, MassTransit, jobs). The actual logic it runs lives in
|
||||
`Musify.Infrastructure`, already covered there.
|
||||
|
||||
### Application.Tests: por qué SQLite en memoria y no mocks
|
||||
### Application.Tests: why in-memory SQLite instead of mocks
|
||||
|
||||
`IDatabase` expone `DbSet<T>` directamente, así que los handlers arman LINQ
|
||||
real (`Where`, `Include`, `Select`...) contra él — no se puede mockear eso con
|
||||
NSubstitute, no hay `IQueryable` detrás de un mock. En su lugar,
|
||||
`TestSupport/TestDatabase.cs` es un `DbContext` real sobre SQLite en memoria
|
||||
(una conexión `:memory:` nueva por test, con foreign keys reales — a diferencia
|
||||
del proveedor InMemory de EF Core, SQLite además soporta transacciones, que dos
|
||||
handlers usan). `TestSupport/TestEntities.cs` y `TestConfigurations.cs`
|
||||
construyen entidades y configuración con valores por defecto sensatos.
|
||||
`IStorageService`, `IEventBus`, etc. sí se mockean con NSubstitute, al ser
|
||||
interfaces normales.
|
||||
`IDatabase` exposes `DbSet<T>` directly, so handlers build real LINQ
|
||||
(`Where`, `Include`, `Select`...) against it. That can't be mocked with
|
||||
NSubstitute, since there's no `IQueryable` behind a mock. Instead,
|
||||
`TestSupport/TestDatabase.cs` is a real `DbContext` on top of in-memory
|
||||
SQLite (a fresh `:memory:` connection per test, with real foreign keys,
|
||||
unlike EF Core's InMemory provider, SQLite also supports transactions,
|
||||
which two handlers rely on). `TestSupport/TestEntities.cs` and
|
||||
`TestConfigurations.cs` build entities and configuration with sensible
|
||||
defaults. `IStorageService`, `IEventBus`, etc. are mocked with NSubstitute,
|
||||
since those are plain interfaces.
|
||||
|
||||
### Infrastructure.Tests y Api.Tests: Testcontainers
|
||||
### Infrastructure.Tests and Api.Tests: Testcontainers
|
||||
|
||||
Estos dos sí necesitan Docker corriendo. Levantan un Postgres real
|
||||
(`Testcontainers.PostgreSql`) y, en `Infrastructure.Tests`, también un
|
||||
SeaweedFS real (mismo `chrislusf/seaweedfs:latest` e imagen/comando que
|
||||
`deploy/compose.yml`, no MinIO) — así `StorageService` se prueba contra el
|
||||
mismo backend S3-compatible que usa producción, no contra AWS S3. Los
|
||||
contenedores se levantan una vez por corrida (`ICollectionFixture`), con
|
||||
migraciones EF Core reales aplicadas contra el Postgres efímero.
|
||||
These two need Docker running. They spin up a real Postgres
|
||||
(`Testcontainers.PostgreSql`) and, in `Infrastructure.Tests`, a real
|
||||
SeaweedFS too (the same `chrislusf/seaweedfs:latest` image/command as
|
||||
`deploy/compose.yml`, not MinIO), so `StorageService` is tested against the
|
||||
same S3-compatible backend production uses, not against AWS S3. Containers
|
||||
are started once per run (`ICollectionFixture`), with real EF Core
|
||||
migrations applied against the ephemeral Postgres.
|
||||
|
||||
`Api.Tests` usa `WebApplicationFactory<Program>` contra ese mismo Postgres —
|
||||
routing, auth y validación reales — pero sustituye por fakes los servicios que
|
||||
hablan por red (`IStorageService`, `IEventBus`, `IStreamTicketService`) y el
|
||||
esquema de autenticación JWT real por un handler
|
||||
de prueba (ver `TestSupport/FakeAuthenticationHandler.cs`) que autentica según
|
||||
un header, sin necesitar un Zitadel real.
|
||||
`Api.Tests` uses `WebApplicationFactory<Program>` against that same
|
||||
Postgres, with real routing, auth, and validation, but replaces the services
|
||||
that talk over the network with fakes (`IStorageService`, `IEventBus`,
|
||||
`IStreamTicketService`), and swaps the real JWT auth scheme for a test
|
||||
handler (see `TestSupport/FakeAuthenticationHandler.cs`) that authenticates
|
||||
based on a header, with no real Zitadel needed.
|
||||
|
||||
Escribir estos tests contra el backend real encontró y arregló dos bugs reales
|
||||
(no simulados): `StorageService.RemoveFolderAsync` reventaba con
|
||||
`NullReferenceException` contra SeaweedFS (su respuesta `DeleteObjects` omite
|
||||
la lista `<Error>` cuando no hay errores, a diferencia de AWS S3), y
|
||||
`AuditableEntityInterceptor` nunca actualizaba `UpdatedAt` en una actualización
|
||||
normal (carga → modifica → guarda) porque corría antes de que EF detectara los
|
||||
cambios pendientes.
|
||||
Writing these tests against the real backend found and fixed two actual
|
||||
bugs (not staged ones): `StorageService.RemoveFolderAsync` threw a
|
||||
`NullReferenceException` against SeaweedFS (its `DeleteObjects` response
|
||||
omits the `<Error>` list when there are no errors, unlike AWS S3), and
|
||||
`AuditableEntityInterceptor` never updated `UpdatedAt` on a normal update
|
||||
(load → modify → save) because it ran before EF had detected the pending
|
||||
changes.
|
||||
|
||||
## Config (AppSettings)
|
||||
|
||||
Convención: `AppSettings.json` es la plantilla con valores vacíos o neutros;
|
||||
`AppSettings.Development.json` tiene los valores reales de dev; en Docker todo
|
||||
se sobreescribe con variables de entorno (`Seccion__Clave`) desde
|
||||
`deploy/compose.yml`. Para correr `Musify.Api` desde el IDE, los valores de
|
||||
`Authentication` van en user-secrets (no en el repo) — cópialos de
|
||||
`deploy/.env` una vez que `zitadel-init` los haya generado (ver
|
||||
Convention: `AppSettings.json` is the template, with empty or neutral
|
||||
values; `AppSettings.Development.json` has the real dev values; in Docker
|
||||
everything is overridden with environment variables (`Section__Key`) from
|
||||
`deploy/compose.yml`. To run `Musify.Api` from the IDE, the `Authentication`
|
||||
values go into user-secrets (not the repo). Copy them from `deploy/.env`
|
||||
once `zitadel-init` has generated them (see
|
||||
[deploy/README.md](../deploy/README.md)).
|
||||
|
||||
Secciones clave: `Authentication` (Zitadel), `Database`, `MassTransit`,
|
||||
Key sections: `Authentication` (Zitadel), `Database`, `MassTransit`,
|
||||
`InfrastructureStorage` (S3), `ApplicationStorage` (bucket), `Track`, `PlayList`,
|
||||
`UploadIntent`, `AudioTranscoder`, `Workers`, `StreamGateway` / `StreamTicket`.
|
||||
|
||||
## Requisitos del host
|
||||
## Host requirements
|
||||
|
||||
- .NET 10 SDK y Node 22.
|
||||
- **ffmpeg** en el PATH (transcode de audio del Worker).
|
||||
- .NET 10 SDK and Node 22.
|
||||
- **ffmpeg** on the PATH (for the Worker's audio transcoding).
|
||||
- Docker Desktop.
|
||||
|
||||
+25
-25
@@ -1,46 +1,46 @@
|
||||
# Procesado de medios (Worker + MassTransit)
|
||||
# Media processing (Worker + MassTransit)
|
||||
|
||||
El procesado pesado (transcode de audio, miniaturas, transferencias en el bucket) ocurre **fuera del request HTTP**, en el **Worker**, orquestado con **MassTransit** sobre RabbitMQ.
|
||||
Heavy processing (audio transcoding, thumbnails, bucket transfers) happens **outside the HTTP request**, in the **Worker**, orchestrated with **MassTransit** on top of RabbitMQ.
|
||||
|
||||
## Por qué asíncrono y con routing slips
|
||||
## Why async, and why routing slips
|
||||
|
||||
- Transcodificar audio y redimensionar imágenes es CPU/IO intensivo: no debe bloquear una llamada a la API.
|
||||
- Es un proceso **multi-paso con efectos secundarios** (descargar de S3 → transcodificar → subir resultados → actualizar DB). Si un paso falla, hay que **deshacer** los anteriores. Para eso se usan **routing slips** (patrón Saga de MassTransit): cada paso es una *activity* con su **compensación**.
|
||||
- **Outbox transaccional** (EF + MassTransit): el evento se publica de forma consistente con el cambio en la DB (no se pierde ni se duplica).
|
||||
- Transcoding audio and resizing images is CPU/IO-intensive: it shouldn't block an API call.
|
||||
- It's a **multi-step process with side effects** (download from S3 → transcode → upload results → update the DB). If a step fails, the previous ones need to be **undone**. That's what **routing slips** are for (MassTransit's Saga pattern): each step is an *activity* with its own **compensation**.
|
||||
- **Transactional outbox** (EF + MassTransit): the event is published consistently with the DB change (never lost, never duplicated).
|
||||
|
||||
## Disparo
|
||||
## Trigger
|
||||
|
||||
Al crear/actualizar una pista o playlist, el handler publica un evento (p. ej. `CreateTrackResourcesEvent`). Un **consumer** (`CreateTrackConsumer`, `UpdateTrackPictureConsumer`, …) construye un **routing slip** con la secuencia de activities y lo lanza.
|
||||
When a track or playlist is created or updated, the handler publishes an event (e.g. `CreateTrackResourcesEvent`). A **consumer** (`CreateTrackConsumer`, `UpdateTrackPictureConsumer`, …) builds a **routing slip** with the sequence of activities and executes it.
|
||||
|
||||
## Builders de routing slip
|
||||
## Routing slip builders
|
||||
|
||||
- `CreateTrackRoutingSlipBuilder` — alta de pista (consume intents, mueve ficheros, dispara procesado).
|
||||
- `PictureWorkflowRoutingSlipBuilder` — pipeline de imágenes (descargar → redimensionar → subir → actualizar).
|
||||
- `AudioWorkflowRoutingSlipBuilder` — pipeline de audio (descargar → transcodificar a `.m4a` → subir carpeta → actualizar).
|
||||
- `CreateTrackRoutingSlipBuilder`: creating a track (consumes intents, moves files, kicks off processing).
|
||||
- `PictureWorkflowRoutingSlipBuilder`: image pipeline (download → resize → upload → update).
|
||||
- `AudioWorkflowRoutingSlipBuilder`: audio pipeline (download → transcode to `.m4a` → upload folder → update).
|
||||
- `DeleteTrackRoutingSlipBuilder`, `DeletePlayListRoutingSlipBuilder`, `PlayListPictureSourceRoutingSlipBuilder`.
|
||||
|
||||
## Activities (pasos)
|
||||
## Activities (steps)
|
||||
|
||||
Agrupadas por área:
|
||||
Grouped by area:
|
||||
- **Files**: `DownloadFileFromBucketActivity`, `UploadFileToBucketActivity`, `TransferFilesToBucketActivity`, `CopyFileInBucketActivity`, `RemoveFileFromBucketActivity`.
|
||||
- **Audio**: `GenerateAudioWorkflowPathsActivity`, `TranscodeAudioActivity` (ffmpeg → `.m4a`), `UpdateTrackAudioActivity` (guarda `AudioFolderName` y marca `Completed`).
|
||||
- **Audio**: `GenerateAudioWorkflowPathsActivity`, `TranscodeAudioActivity` (ffmpeg → `.m4a`), `UpdateTrackAudioActivity` (saves `AudioFolderName` and marks it `Completed`).
|
||||
- **Pictures**: `GeneratePictureWorkflowPathsActivity`, `ResizePictureActivity` (ImageSharp), `UpdateTrackPictureActivity`, `UpdatePlayListPictureActivity`.
|
||||
- **Tracks/PlayLists**: `MarkTrackAsRemovingActivity`, `DeleteTrackFromDbActivity`, `PublishTrackProcessingEventsActivity`, y equivalentes de playlist.
|
||||
- **Tracks/Playlists**: `MarkTrackAsRemovingActivity`, `DeleteTrackFromDbActivity`, `PublishTrackProcessingEventsActivity`, and their playlist equivalents.
|
||||
- **UploadIntents**: `ConsumeUploadIntentsActivity`.
|
||||
|
||||
`RoutingSlipCleanUpConsumer` reacciona a `Completed`/`Faulted` para limpiar (p. ej. borrar el directorio temporal de trabajo).
|
||||
`RoutingSlipCleanUpConsumer` reacts to `Completed`/`Faulted` to clean up (e.g. deleting the temporary working directory).
|
||||
|
||||
## Transcode de audio
|
||||
## Audio transcoding
|
||||
|
||||
`AudioTranscoderService` invoca **ffmpeg** leyendo el original por stdin y produciendo en un directorio de trabajo un único `audio.m4a`: AAC con codec/bitrate/sample-rate/profile configurables y `-movflags +faststart` (moov al inicio → seek por range). Luego `TransferFilesToBucketActivity` sube esa carpeta a `Tracks/ProcessedAudios/{folder}/` y `UpdateTrackAudioActivity` guarda el `AudioFolderName` (la carpeta **de destino en S3**, no la local) y pone el estado en `Completed`.
|
||||
`AudioTranscoderService` calls **ffmpeg**, reading the original from stdin and producing a single `audio.m4a` in a working directory: AAC with a configurable codec/bitrate/sample-rate/profile, and `-movflags +faststart` (moov at the start, so range-based seeking works). `TransferFilesToBucketActivity` then uploads that folder to `Tracks/ProcessedAudios/{folder}/`, and `UpdateTrackAudioActivity` saves the `AudioFolderName` (the **destination folder in S3**, not the local one) and sets the status to `Completed`.
|
||||
|
||||
## Configuración que necesita el Worker
|
||||
## What the Worker needs configured
|
||||
|
||||
El Worker registra: storage (S3), transcoder (ffmpeg), pictures, DB, los consumers de MassTransit y los jobs de upload intents. Su `AppSettings` debe tener las secciones correctas: `Bucket` (= `webapi-storage`, el mismo que la API), `PlayList`, `Track`, `MassTransit`, `AudioTranscoder`, `UploadIntent`, `Workers` (directorio temporal), `InfrastructureStorage`, `Database`.
|
||||
The Worker registers: storage (S3), transcoder (ffmpeg), pictures, the DB, the MassTransit consumers, and the upload-intent jobs. Its `AppSettings` needs the right sections: `Bucket` (= `webapi-storage`, same as the API), `PlayList`, `Track`, `MassTransit`, `AudioTranscoder`, `UploadIntent`, `Workers` (temp directory), `InfrastructureStorage`, `Database`.
|
||||
|
||||
## Jobs de fondo
|
||||
## Background jobs
|
||||
|
||||
- `UploadIntentExpirationJob` — marca intents caducados como expirados (libera cuota) cada `ExpirationJobIntervalSeconds`.
|
||||
- `TemporalUploadsCleanUpJob` — limpia subidas temporales antiguas.
|
||||
- `UploadIntentExpirationJob`: marks expired intents as such (freeing quota) every `ExpirationJobIntervalSeconds`.
|
||||
- `TemporalUploadsCleanUpJob`: cleans up old temporary uploads.
|
||||
|
||||
Corren en el host del Worker (`AddUploadIntentJobs`). Si no se registran en ningún proceso, los intents colgados no se limpian solos.
|
||||
Both run on the Worker host (`AddUploadIntentJobs`). If they aren't registered anywhere, stale intents never get cleaned up on their own.
|
||||
|
||||
+37
-37
@@ -1,74 +1,74 @@
|
||||
# Proyectos
|
||||
# Projects
|
||||
|
||||
Los proyectos .NET viven en `backend/` (solución `Musify.slnx`, carpetas
|
||||
`Core/`, `Hosts/` y `Tests/`); el cliente web en `web-player/`.
|
||||
The .NET projects live under `backend/` (solution `Musify.slnx`, folders
|
||||
`Core/`, `Hosts/` and `Tests/`); the web client lives under `web-player/`.
|
||||
|
||||
## Musify.Api (`:5111`)
|
||||
|
||||
API REST pública (Minimal APIs, .NET 10). Es la fachada fina del sistema.
|
||||
The public REST API (Minimal APIs, .NET 10). The thin public face of the system.
|
||||
|
||||
- **Endpoints**: `UserEndpoints`, `TrackEndpoints`, `PlayListEndpoints`.
|
||||
- **Autenticación**: JWT Bearer contra Zitadel (`Authentication/`), con `JwtBearerEventsHandler` que sincroniza el usuario en cada token validado.
|
||||
- **`CurrentUser`**: abstracción que se enlaza como parámetro de endpoint (`IBindableFromHttpContext`) y expone los claims básicos (`Id` como `long?`, `RequiredId`, nombre, etc.) sin tener que leer `ClaimsPrincipal` a mano.
|
||||
- **Emite**: presigned URLs de subida (vía `IStorageService`) y **stream-tickets** (vía `IStreamTicketService`, RS256).
|
||||
- **OpenAPI + Scalar** para documentación/prueba; OAuth2 (PKCE) configurado para login desde Scalar.
|
||||
- **CORS** totalmente abierto **solo en Development** (para el front local).
|
||||
- Traduce `ErrorOr` → HTTP en `ResultHttpExtensions`.
|
||||
- **Authentication**: JWT Bearer against Zitadel (`Authentication/`), with a `JwtBearerEventsHandler` that syncs the user on every validated token.
|
||||
- **`CurrentUser`**: an abstraction bound as an endpoint parameter (`IBindableFromHttpContext`) that exposes the basic claims (`Id` as `long?`, `RequiredId`, name, etc.) without reading `ClaimsPrincipal` by hand.
|
||||
- **Issues**: presigned upload URLs (via `IStorageService`) and **stream tickets** (via `IStreamTicketService`, RS256).
|
||||
- **OpenAPI + Scalar** for docs/testing; OAuth2 (PKCE) set up for logging in from Scalar.
|
||||
- **CORS** wide open, but **only in Development** (for the local frontend).
|
||||
- Translates `ErrorOr` into HTTP in `ResultHttpExtensions`.
|
||||
|
||||
**Propósito/porqué**: separar la cara pública (autorización + metadatos) del procesado y del servido de bytes.
|
||||
**Purpose**: keep the public surface (authorization + metadata) separate from processing and byte-serving.
|
||||
|
||||
## Musify.Domain + Musify.Application + Musify.Infrastructure
|
||||
|
||||
El núcleo, en tres proyectos (clean architecture):
|
||||
The core, split into three projects (clean architecture):
|
||||
|
||||
### Domain
|
||||
Entidades (`Entities/`) y value objects/enums (`ValueObjects/`: `ProcessingStatus`, `LifeCycleStatus`, `UploadIntentStatus`, …). Sin dependencias de infraestructura.
|
||||
Entities (`Entities/`) and value objects/enums (`ValueObjects/`: `ProcessingStatus`, `LifeCycleStatus`, `UploadIntentStatus`, …). No infrastructure dependencies.
|
||||
|
||||
### Application
|
||||
Casos de uso (CQRS con Mediator):
|
||||
Use cases (CQRS with Mediator):
|
||||
- `Tracks/`, `PlayLists/`, `Users/` → `Commands`, `Queries`, `Handlers`, `Responses`.
|
||||
- `Contracts/` → interfaces que implementa la infraestructura (`IDatabase`, `IStorageService`, `IAudioTranscoderService`, `IPictureService`, `IEventBus`, `IStreamTicketService`).
|
||||
- `Configuration/` → opciones tipadas (Track, PlayList, ApplicationStorage, UploadIntent…) bindeadas y validadas al arranque.
|
||||
- `Shared/` → helpers transversales (`StorageKey.Combine`, `ImageSize`).
|
||||
- `Contracts/` → interfaces that infrastructure implements (`IDatabase`, `IStorageService`, `IAudioTranscoderService`, `IPictureService`, `IEventBus`, `IStreamTicketService`).
|
||||
- `Configuration/` → typed options (Track, PlayList, ApplicationStorage, UploadIntent…) bound and validated at startup.
|
||||
- `Shared/` → cross-cutting helpers (`StorageKey.Combine`, `ImageSize`).
|
||||
- `Pagination/` → `PaginatedResponse<T>`.
|
||||
|
||||
**Porqué**: la lógica de negocio no conoce EF, S3 ni RabbitMQ; solo interfaces. Reutilizable por la API y el Worker.
|
||||
**Why**: business logic doesn't know about EF, S3, or RabbitMQ, only interfaces. Shared by both the API and the Worker.
|
||||
|
||||
### Infrastructure
|
||||
Implementaciones concretas:
|
||||
- `Persistence/` → `Database` (EF Core/Npgsql), migraciones.
|
||||
- `Services/` → `StorageService` (AWSSDK.S3 contra SeaweedFS), `AudioTranscoderService` (ffmpeg → `.m4a`), `PictureService` (ImageSharp), `StreamTicketService` (firma RS256).
|
||||
- `MassTransit/` → consumers, activities y routing slips (ver [media-processing.md](media-processing.md)).
|
||||
Concrete implementations:
|
||||
- `Persistence/` → `Database` (EF Core/Npgsql), migrations.
|
||||
- `Services/` → `StorageService` (AWSSDK.S3 against SeaweedFS), `AudioTranscoderService` (ffmpeg → `.m4a`), `PictureService` (ImageSharp), `StreamTicketService` (RS256 signing).
|
||||
- `MassTransit/` → consumers, activities, and routing slips (see [media-processing.md](media-processing.md)).
|
||||
- `Jobs/` → `UploadIntentExpirationJob`, `TemporalUploadsCleanUpJob`.
|
||||
|
||||
## Musify.StreamingGateway (`:8081`)
|
||||
|
||||
Reverse proxy (ASP.NET + **YARP**) que sirve el audio (`.m4a`) desde SeaweedFS. Valida el stream-ticket (RS256, clave pública) y, si el objeto pedido cae bajo el prefijo autorizado, reenvía al filer.
|
||||
A reverse proxy (ASP.NET + **YARP**) that serves audio (`.m4a`) from SeaweedFS. Validates the stream ticket (RS256, public key) and, if the requested object falls under the authorized prefix, forwards it to the filer.
|
||||
|
||||
**Porqué un proyecto aparte**: escala con el tráfico de bytes independientemente de la API, tiene superficie mínima (solo sabe validar un JWT y proxiar) y permite que SeaweedFS quede en red privada. Ver [streaming.md](streaming.md).
|
||||
**Why a separate project**: it scales with byte traffic independently of the API, has a minimal surface (it only knows how to validate a JWT and proxy), and lets SeaweedFS stay on a private network. See [streaming.md](streaming.md).
|
||||
|
||||
## Musify.Worker
|
||||
|
||||
Worker Service (host de fondo) que consume los eventos de MassTransit y ejecuta el procesado pesado: transcode de audio a `.m4a` (ffmpeg), generación de miniaturas (ImageSharp), transferencias de ficheros en el bucket, y los jobs de expiración/limpieza de upload intents.
|
||||
A background Worker Service that consumes MassTransit events and runs the heavy processing: audio transcoding to `.m4a` (ffmpeg), thumbnail generation (ImageSharp), file transfers within the bucket, and the upload-intent expiration/cleanup jobs.
|
||||
|
||||
**Porqué aparte**: el trabajo CPU/IO no debe ocurrir dentro de un request HTTP; se escala y reinicia por separado. (Antes se llamaba `Musify.PictureWorker`; se renombró porque hace más que imágenes.)
|
||||
**Why separate**: CPU/IO-heavy work shouldn't happen inside an HTTP request; it scales and restarts independently. (Previously named `Musify.PictureWorker`; renamed once it started doing more than just images.)
|
||||
|
||||
## web-player (`:5173` en dev, `:3000` en contenedor)
|
||||
## web-player (`:5173` in dev, `:3000` in a container)
|
||||
|
||||
Cliente web (SvelteKit 2 + Svelte 5 + Tailwind 4). La sesión OIDC vive en el
|
||||
servidor (cookie cifrada); las llamadas al backend salen del servidor con el
|
||||
access token, y las subidas se hacen contra las URLs prefirmadas de S3.
|
||||
The web client (SvelteKit 2 + Svelte 5 + Tailwind 4). The OIDC session lives
|
||||
on the server (an encrypted cookie); backend calls go out from the server
|
||||
with the access token, and uploads go straight to the presigned S3 URLs.
|
||||
|
||||
## Tests
|
||||
|
||||
`backend/Tests/`, carpeta `/Tests/` en el `.slnx`: cinco proyectos xUnit —
|
||||
`Musify.Domain.Tests`, `Musify.Application.Tests` (SQLite en memoria, sin
|
||||
Docker), `Musify.Infrastructure.Tests` y `Musify.Api.Tests` (Testcontainers:
|
||||
Postgres + SeaweedFS reales) y `Musify.StreamingGateway.Tests`. Detalle en
|
||||
`backend/Tests/` (the `/Tests/` folder in the `.slnx`) has five xUnit projects:
|
||||
`Musify.Domain.Tests`, `Musify.Application.Tests` (in-memory SQLite, no
|
||||
Docker), `Musify.Infrastructure.Tests` and `Musify.Api.Tests` (Testcontainers:
|
||||
real Postgres + SeaweedFS), and `Musify.StreamingGateway.Tests`. Details in
|
||||
[development.md](development.md#tests).
|
||||
|
||||
## Otras carpetas
|
||||
## Other folders
|
||||
|
||||
- **`deploy/`** — stacks de Docker Compose (dev y prod), autocontenidos. Ver
|
||||
- **`deploy/`**: self-contained Docker Compose stacks (dev and prod). See
|
||||
[../deploy/README.md](../deploy/README.md).
|
||||
- **`docs/`** — esta documentación.
|
||||
- **`docs/`**: this documentation.
|
||||
|
||||
+30
-30
@@ -1,48 +1,48 @@
|
||||
# Almacenamiento (SeaweedFS / S3)
|
||||
# Storage (SeaweedFS / S3)
|
||||
|
||||
## Qué es
|
||||
## What it is
|
||||
|
||||
**SeaweedFS** es el almacén de objetos, compatible con la API S3. Guarda los originales subidos y los derivados (audio `.m4a`, miniaturas). Se accede vía `AWSSDK.S3` desde `StorageService` (`IStorageService`).
|
||||
**SeaweedFS** is the object store, S3-API compatible. It holds the uploaded originals and their derivatives (`.m4a` audio, thumbnails). Accessed via `AWSSDK.S3` from `StorageService` (`IStorageService`).
|
||||
|
||||
Expone:
|
||||
- **S3 API** (`:8333`) — operaciones S3 (presigned URLs, put/get/list…).
|
||||
- **filer HTTP** (`:8888`) — sirve los ficheros por ruta; lo usa el **StreamingGateway**.
|
||||
- master (`:9333`) y volume server — internos.
|
||||
It exposes:
|
||||
- **S3 API** (`:8333`): S3 operations (presigned URLs, put/get/list…).
|
||||
- **filer HTTP** (`:8888`): serves files by path, used by the **StreamingGateway**.
|
||||
- master (`:9333`) and volume server, internal only.
|
||||
|
||||
Bucket de la app: **`webapi-storage`** (`ApplicationStorage:Bucket`). Importante: la API y el Worker usan **el mismo bucket** (el worker procesa lo que sube la API).
|
||||
App bucket: **`webapi-storage`** (`ApplicationStorage:Bucket`). Important: the API and the Worker use **the same bucket** (the worker processes whatever the API uploads).
|
||||
|
||||
## Estructura de claves (layout)
|
||||
## Key layout
|
||||
|
||||
- `temp/{userId}/Tracks/{guid}.{ext}` — subidas temporales (objetivo de las presigned PUT, antes de consumir el intent).
|
||||
- `uploads/{userId}/Tracks/OriginalPictures|OriginalAudios/...` — originales ya consumidos.
|
||||
- `Tracks/SmallPictures|MediumPictures|LargePictures/...` — miniaturas generadas.
|
||||
- `Tracks/ProcessedAudios/{AudioFolderName}/` — audio transcodificado (`audio.m4a`, AAC con faststart).
|
||||
- `temp/{userId}/Tracks/{guid}.{ext}`: temporary uploads, the target of presigned PUTs, before the intent is consumed.
|
||||
- `uploads/{userId}/Tracks/OriginalPictures|OriginalAudios/...`: originals that have already been consumed.
|
||||
- `Tracks/SmallPictures|MediumPictures|LargePictures/...`: generated thumbnails.
|
||||
- `Tracks/ProcessedAudios/{AudioFolderName}/`: transcoded audio (`audio.m4a`, AAC with faststart).
|
||||
|
||||
Las claves se construyen **siempre** con el helper `StorageKey.Combine(...)` (en `Application/Shared`), que une segmentos con `/` limpiando espacios y barras. Está centralizado para estandarizar y no repetir lógica.
|
||||
Keys are always built with the `StorageKey.Combine(...)` helper (in `Application/Shared`), which joins segments with `/` and trims stray spaces and slashes. It's centralized to keep the logic consistent and avoid repeating it.
|
||||
|
||||
> En el filer, los objetos S3 viven bajo `/buckets/{bucket}/{key}` — de ahí que el gateway reescriba `/media/{key}` → `/buckets/webapi-storage/{key}`.
|
||||
> In the filer, S3 objects live under `/buckets/{bucket}/{key}`, which is why the gateway rewrites `/media/{key}` → `/buckets/webapi-storage/{key}`.
|
||||
|
||||
## URLs prefirmadas (presigned)
|
||||
## Presigned URLs
|
||||
|
||||
Para **no** hacer pasar ficheros grandes por la API, el cliente habla directo con S3 mediante URLs firmadas de corta vida:
|
||||
To **avoid** routing large files through the API, the client talks to S3 directly using short-lived signed URLs:
|
||||
|
||||
- **Subidas (PUT)**: `RequestTrackUploadUrls` / `RequestPlayListPictureUpload` generan presigned PUT (con `Content-Type` y `If-None-Match: *` firmados, expiración ~120s). El cliente sube directo a S3.
|
||||
- **Imágenes (GET)**: las miniaturas se sirven con presigned GET (un objeto = una URL).
|
||||
- **Uploads (PUT)**: `RequestTrackUploadUrls` / `RequestPlayListPictureUpload` generate presigned PUTs (with signed `Content-Type` and `If-None-Match: *`, ~120s expiry). The client uploads straight to S3.
|
||||
- **Images (GET)**: thumbnails are served via presigned GETs (one object = one URL).
|
||||
|
||||
**Por qué presigned y no proxiar**: subir un audio de ~100 MB a través de la API consumiría memoria/ancho de banda de los servidores de aplicación y escalaría mal. La presigned PUT directa a S3 es el patrón estándar y correcto.
|
||||
**Why presigned instead of proxying**: routing a ~100 MB audio file through the API would burn memory and bandwidth on the application servers and wouldn't scale well. A direct presigned PUT to S3 is the standard, correct pattern here.
|
||||
|
||||
### Upload intents y cuota
|
||||
Cada `upload-urls` crea **UploadIntents** (reservas en estado `Issued`) y devuelve las URLs. Al crear la pista/playlist se **consumen**. Hay cuota por usuario (`MaxActiveUploadIntentsPerUser`, bytes…) para evitar abuso. El `UploadIntentExpirationJob` (en el Worker) marca como expirados los intents caducados y libera la cuota; si ese job no corre, los intents colgados se acumulan y bloquean nuevas subidas.
|
||||
### Upload intents and quota
|
||||
Every `upload-urls` call creates **UploadIntents** (reservations in the `Issued` state) and returns the URLs. They're **consumed** when the track/playlist is created. There's a per-user quota (`MaxActiveUploadIntentsPerUser`, byte limits…) to prevent abuse. The `UploadIntentExpirationJob` (in the Worker) marks expired intents and frees up quota. If that job isn't running, stale intents pile up and block new uploads.
|
||||
|
||||
## Qué es público y qué privado (despliegue)
|
||||
## What's public and what's private (deployment)
|
||||
|
||||
| Exponer a internet | Mantener privado |
|
||||
| Expose to the internet | Keep private |
|
||||
|---|---|
|
||||
| S3 API `:8333` (tras TLS) — necesario para presigned PUT/GET del cliente | filer `:8888` (solo el gateway, en red interna) |
|
||||
| StreamingGateway `:8081` | master `:9333`, volume server, UI de admin |
|
||||
| S3 API `:8333` (behind TLS), needed for the client's presigned PUT/GET | filer `:8888` (internal network, gateway only) |
|
||||
| StreamingGateway `:8081` | master `:9333`, volume server, admin UI |
|
||||
|
||||
- **El audio** se sirve por el gateway → SeaweedFS puede quedar privado para ese flujo.
|
||||
- **Subidas e imágenes** usan presigned → el cliente toca el endpoint S3 directamente, así que `:8333` debe ser alcanzable (pero inútil sin firma válida; es el modelo de cualquier bucket S3/MinIO).
|
||||
- En **producción**: `InfrastructureStorage:Address` debe ser el **endpoint S3 público** (la firma se calcula para ese host); si los servidores no lo alcanzan internamente, se usa config dividida (host público para firmar / interno para operaciones server-side).
|
||||
- **Audio** is served through the gateway, so SeaweedFS can stay private for that flow.
|
||||
- **Uploads and images** use presigned URLs, so the client talks to the S3 endpoint directly and `:8333` has to be reachable (though useless without a valid signature, this is the standard model for any S3/MinIO bucket).
|
||||
- In **production**: `InfrastructureStorage:Address` must be the **public S3 endpoint** (the signature is computed for that host). If the servers can't reach it internally, use split config: public host for signing, internal host for server-side operations.
|
||||
|
||||
**Regla**: el gateway y, como mucho, el `:8333` firmado son la única superficie pública; master/volume/filer/admin **nunca** se exponen.
|
||||
**Rule of thumb**: the gateway, and at most the signed `:8333`, are the only public surface. Master, volume, filer and admin are **never** exposed.
|
||||
|
||||
+31
-31
@@ -1,55 +1,55 @@
|
||||
# Streaming de audio
|
||||
# Audio streaming
|
||||
|
||||
## Objetivo
|
||||
## Goal
|
||||
|
||||
Reproducir el audio **sin que los bytes pasen por la API**, pero con control de acceso. La API solo emite un permiso corto; el reverse proxy sirve los bytes desde SeaweedFS.
|
||||
Play audio **without the bytes passing through the API**, while still controlling access. The API only issues a short-lived permission; the reverse proxy serves the bytes from SeaweedFS.
|
||||
|
||||
## Formato: fichero único AAC (.m4a) por HTTP range
|
||||
## Format: a single AAC file (.m4a), served via HTTP range
|
||||
|
||||
El audio se transcodifica a **un único `audio.m4a`** (AAC, contenedor MP4 con `+faststart` para que el átomo `moov` quede al inicio y el seek por range funcione), dentro de la carpeta `Tracks/ProcessedAudios/{AudioFolderName}/`. No hay manifest ni segmentos: el reproductor descarga el fichero por **HTTP range requests** (modelo tipo Spotify para VOD musical).
|
||||
Audio is transcoded into **one `audio.m4a`** (AAC, MP4 container with `+faststart` so the `moov` atom sits at the start and range-based seeking works), stored under `Tracks/ProcessedAudios/{AudioFolderName}/`. There's no manifest or segments: the player downloads the file via **HTTP range requests** (the same model Spotify uses for on-demand music).
|
||||
|
||||
Multi-bitrate no está implementado hoy; el camino para añadirlo es transcodificar varias calidades (`audio_96.m4a`, …) y que el cliente elija una al iniciar (sin ABR fluido a mitad de pista, igual que Spotify).
|
||||
Multi-bitrate isn't implemented yet; the path to add it is transcoding several qualities (`audio_96.m4a`, …) and letting the client pick one at playback start (no seamless mid-track ABR, same as Spotify).
|
||||
|
||||
## El acceso: stream-ticket de prefijo
|
||||
## Access control: a prefix-scoped stream ticket
|
||||
|
||||
Se autoriza **el prefijo de la carpeta** de la pista con un *stream-ticket* de corta vida, y el proxy lo valida en cada petición. Aunque hoy sea un único objeto, el ticket sigue acotado al prefijo `Tracks/ProcessedAudios/<folder>/` (cubre el fichero actual y cualquier calidad futura).
|
||||
Access is authorized for **the track's folder prefix** via a short-lived *stream ticket*, validated by the proxy on every request. Even though there's only one object today, the ticket is still scoped to the prefix `Tracks/ProcessedAudios/<folder>/` (covering the current file and any future quality variants).
|
||||
|
||||
## Flujo
|
||||
## Flow
|
||||
|
||||
1. `GET /tracks/{id}/stream` (autenticado) → la API autoriza y devuelve:
|
||||
1. `GET /tracks/{id}/stream` (authenticated) → the API authorizes the request and returns:
|
||||
```json
|
||||
{ "manifestUrl": "http://<gateway>/media/Tracks/ProcessedAudios/<folder>/audio.m4a",
|
||||
"ticket": "<JWT RS256>", "expiresInSeconds": 3600 }
|
||||
```
|
||||
El ticket lleva el claim `prefix = "Tracks/ProcessedAudios/<folder>/"`.
|
||||
2. El reproductor abre la URL del audio añadiéndole `?t=<ticket>` (o la cabecera `X-Stream-Ticket`).
|
||||
3. El **StreamingGateway** (YARP) valida el ticket y reenvía al filer; los bytes van SeaweedFS → cliente (con range/seek).
|
||||
The ticket carries the claim `prefix = "Tracks/ProcessedAudios/<folder>/"`.
|
||||
2. The player opens the audio URL with `?t=<ticket>` appended (or the `X-Stream-Ticket` header).
|
||||
3. The **StreamingGateway** (YARP) validates the ticket and forwards the request to the filer; the bytes flow SeaweedFS → client (with range/seek support).
|
||||
|
||||
## El ticket (RS256)
|
||||
## The ticket (RS256)
|
||||
|
||||
- Lo firma la API con clave **privada** (`StreamTicketService`); el gateway valida con la **pública** (`TicketValidator`). Así el gateway **no puede emitir** tickets, solo verificarlos.
|
||||
- Claims: `sub`, `prefix`, `aud = media-gateway`, `iss = musify-webapi`, `exp` (TTL ~1h).
|
||||
- Signed by the API with a **private** key (`StreamTicketService`); validated by the gateway with the **public** one (`TicketValidator`). That way the gateway **can't issue** tickets, only verify them.
|
||||
- Claims: `sub`, `prefix`, `aud = media-gateway`, `iss = musify-webapi`, `exp` (~1h TTL).
|
||||
|
||||
**Por qué RS256 (asimétrica)**: separa responsabilidades — emisor (la API) y verificador (gateway) no comparten secreto; el gateway, aunque se comprometa, no puede crear permisos.
|
||||
**Why RS256 (asymmetric)**: it separates responsibilities. The issuer (API) and the verifier (gateway) don't share a secret, so even if the gateway is compromised, it can't mint permissions.
|
||||
|
||||
## El gateway (validación + proxy)
|
||||
## The gateway (validation + proxy)
|
||||
|
||||
`TicketValidationMiddleware` en cada petición a `/media/**`:
|
||||
1. Lee `?t=`, valida firma/aud/iss/exp.
|
||||
2. Comprueba que la clave pedida **empieza por** el `prefix` del ticket (`StartsWith`). Si no → 403.
|
||||
3. Quita el `?t=` y deja pasar a YARP, que reescribe `/media/{key}` → `/buckets/webapi-storage/{key}` del filer.
|
||||
`TicketValidationMiddleware`, on every request to `/media/**`:
|
||||
1. Reads `?t=`, validates its signature/aud/iss/exp.
|
||||
2. Checks that the requested key **starts with** the ticket's `prefix`. If not → 403.
|
||||
3. Strips the `?t=` and passes the request on to YARP, which rewrites `/media/{key}` → `/buckets/webapi-storage/{key}` for the filer.
|
||||
|
||||
- Sin ticket → **401**; fuera de prefijo → **403**; ok → **200** (y **206** en range requests, así que el seek funciona).
|
||||
- Pasa `Range`/`Accept-Ranges` y expone esos headers por CORS.
|
||||
- No ticket → **401**; outside the prefix → **403**; ok → **200** (and **206** for range requests, so seeking works).
|
||||
- Passes through `Range`/`Accept-Ranges` and exposes those headers via CORS.
|
||||
|
||||
## ¿Un ticket por canción?
|
||||
## One ticket per track?
|
||||
|
||||
Sí. Cada ticket está acotado a **una** carpeta (una pista). Para reproducir otra, se pide otro ticket. Es barato (la API solo firma un JWT) y es lo más seguro: cada permiso abre solo lo que vas a reproducir.
|
||||
Yes. Each ticket is scoped to **one** folder (one track). Playing a different one means requesting a new ticket. It's cheap (the API just signs a JWT) and it's the safest option: every permission only unlocks what you're about to play.
|
||||
|
||||
- Si algún día molesta (muchísimos cambios de pista), el mismo mecanismo del claim `prefix` permite ampliar el alcance: a nivel de playlist o de toda la biblioteca (`Tracks/ProcessedAudios/`). Trade-off seguridad ↔ comodidad. Por defecto, **por canción**.
|
||||
- If that ever becomes a problem (very frequent track switching), the same `prefix` claim mechanism can widen the scope to a whole playlist or the whole library (`Tracks/ProcessedAudios/`). It's a security/convenience trade-off. The default is **per track**.
|
||||
|
||||
## Pendiente / notas de seguridad
|
||||
## Open items / security notes
|
||||
|
||||
- La política de quién puede pedir el ticket es hoy "cualquier autenticado" (TODO: restringir con `UserHasTrack`/visibilidad).
|
||||
- El check de prefijo es `StartsWith` literal; conviene endurecer rechazando claves con `..` (path traversal), aunque navegadores y ASP.NET normalizan `..`.
|
||||
- El ticket viaja en la query (`?t=`); el gateway lo elimina antes de reenviar (no llega a los logs del filer).
|
||||
- Today, the policy for who can request a ticket is "any authenticated user" (TODO: restrict it with `UserHasTrack`/visibility).
|
||||
- The prefix check is a literal `StartsWith`; it would be worth hardening it against `..` path traversal, even though browsers and ASP.NET normalize `..`.
|
||||
- The ticket travels in the query string (`?t=`); the gateway strips it before forwarding, so it never reaches the filer's logs.
|
||||
|
||||
Reference in New Issue
Block a user