Update docs for the refactored backend and add backend conventions guide
This commit is contained in:
+13
-12
@@ -3,7 +3,7 @@
|
||||
## Principles
|
||||
|
||||
- **Clean architecture / layers** in `backend/`: the domain and application layers don't depend on infrastructure; infrastructure implements their interfaces (`Application/Contracts`).
|
||||
- **CQRS**: every operation is a `Command` or `Query` with its own handler (Mediator). Handlers return `Ardalis.Result` (Ok/NotFound/Unauthorized/Conflict/Invalid…), which the API translates into HTTP status codes.
|
||||
- **CQRS**: every operation is a `Command` or `Query` with its own handler (Mediator). Handlers that can fail return `ErrorOr<T>` (built with `AppErrors`: NotFound, Forbidden, Conflict, Validation; `Unauthorized` only for "not signed in"), which the API translates into HTTP status codes and ProblemDetails with `ToHttpResult()`.
|
||||
- **Heavy bytes never touch the API**: uploads use presigned URLs (client → S3) and playback goes through a dedicated reverse proxy (gateway → S3). The API only moves metadata and issues permissions.
|
||||
- **Async processing**: creating a track doesn't transcode it on the spot. It publishes an event, and a worker does the heavy lifting, with compensation (MassTransit sagas/routing slips).
|
||||
|
||||
@@ -11,11 +11,11 @@
|
||||
|
||||
| Component | Responsibility | Why it's separate |
|
||||
|---|---|---|
|
||||
| **Musify.Api** | REST API (users, tracks, playlists), authorizes requests, issues presigned URLs and stream tickets | A thin entry point, it doesn't process media or serve bytes |
|
||||
| **Musify.Api** | REST API (users, tracks, albums, playlists, likes, mixes, genres), authorizes requests, issues presigned URLs and stream tickets | A thin entry point, it doesn't process media or serve bytes |
|
||||
| **Domain/Application/Infrastructure** | Domain + use cases + shared infrastructure (EF, S3, MassTransit, ffmpeg) | Logic shared between the API and the Worker |
|
||||
| **StreamingGateway** | Reverse proxy (YARP) that validates the ticket and serves the audio (`.m4a`) from SeaweedFS | Scales with byte traffic independently of the API, and keeps SeaweedFS private behind it |
|
||||
| **Worker** | Consumes MassTransit events: transcodes audio to `.m4a` and generates thumbnails | Keeps CPU/IO-heavy work off the HTTP request path |
|
||||
| **PostgreSQL** *(shared, runs in the separate `Infrastructure` repo)* | Metadata (users, tracks, playlists, upload intents) | n/a |
|
||||
| **Worker** | Consumes MassTransit events (transcodes audio to `.m4a`, generates thumbnails, deletes entities) and runs the recurring Hangfire jobs | Keeps CPU/IO-heavy work off the HTTP request path |
|
||||
| **PostgreSQL** *(shared, runs in the separate `Infrastructure` repo)* | Metadata (users, tracks, albums, playlists, upload intents, listening history) and the Hangfire job store | n/a |
|
||||
| **RabbitMQ** *(shared, runs in Infrastructure)* | Event queue + transactional outbox | Decouples creation from processing |
|
||||
| **SeaweedFS** *(shared, runs in Infrastructure)* | Object storage (S3 + filer) | Stores originals and derivatives (`.m4a` audio, thumbnails) |
|
||||
| **Zitadel** *(shared, runs in Infrastructure)* | Identity (OIDC/OAuth2) | Login and JWT issuance |
|
||||
@@ -25,13 +25,13 @@
|
||||
### 1. Uploading a track
|
||||
1. The client requests `POST /tracks/upload-urls`. The API creates **UploadIntents** (reservations) and returns **presigned PUT URLs** (image + audio).
|
||||
2. The client uploads the files **directly to SeaweedFS** using those URLs (never through the API).
|
||||
3. The client calls `POST /tracks` with the `intentId`s. The API validates the intents, creates the `Track` (+ `UserHasTrack`), and publishes `CreateTrackResourcesEvent`.
|
||||
3. The client calls `POST /tracks` with the `intentId`s. The API validates the intents, creates the `Track` (owned by the user through `OwnerUserId`), and publishes `CreateTrackResourcesEvent`. Albums and playlists follow the same pattern with `POST /albums/upload-picture` / `POST /playlists/upload-picture` and `POST /albums` / `POST /playlists`.
|
||||
4. The **Worker** consumes the event: moves the originals into place, generates thumbnails, transcodes the audio to `.m4a`, and updates the track's status.
|
||||
|
||||
See [media-processing.md](media-processing.md) and [storage.md](storage.md) for details.
|
||||
|
||||
### 2. Playback (streaming)
|
||||
1. The client requests `GET /tracks/{id}/stream` (authenticated). The API authorizes it and returns `{ manifestUrl, ticket }` (a URL to the `.m4a` plus an RS256 ticket scoped to that track's folder).
|
||||
1. The client requests `GET /tracks/{id}/stream` (authenticated, or anonymous when `Playback:AllowAnonymousListening` is on). The API records the listen (`StartListeningCommand`) and issues the ticket. The API authorizes it and returns `{ manifestUrl, ticket }` (a URL to the `.m4a` plus an RS256 ticket scoped to that track's folder).
|
||||
2. The player opens the audio URL with `?t=<ticket>` appended (or an `X-Stream-Ticket` header) and downloads it via HTTP range requests.
|
||||
3. The **StreamingGateway** validates the ticket and forwards the request to the SeaweedFS filer. The bytes flow storage → client.
|
||||
|
||||
@@ -46,9 +46,10 @@ See [authentication.md](authentication.md).
|
||||
|
||||
## Data model (entities)
|
||||
|
||||
- **User**: `Id` is Zitadel's numeric `sub` (`long`), plus a name and `ProfilePictureUrl`.
|
||||
- **Track**: original/derivative file names, `AudioFolderName` (the transcoded `.m4a`'s folder), processing state (`ProcessingStatus`), `LifeCycleStatus`.
|
||||
- **PlayList**: a user's playlist (`UserId`).
|
||||
- **PlayListHasTrack**: the tracks inside a playlist (with `Position`).
|
||||
- **UserHasTrack**: ownership, which user owns which track.
|
||||
- **UploadIntent**: an upload reservation (key, bucket, expected size, status, expiration) used for validation and quota.
|
||||
- **User**: `Id` is Zitadel's numeric `sub` (`long`), plus a name and `ProfilePictureUrl`. **UserFollow** links follower and followed.
|
||||
- **Track**: owned by `OwnerUserId`; `Pictures` (`TrackPictures`) and `Audio` (`TrackAudio`, with `FolderName` = the transcoded `.m4a`'s folder) are owned types with their own processing status; `LifeCycleStatus` (Active, Removing, Failed). **TrackTag** holds its genres, **TrackLike** the likes, **ListeningHistory** the listens.
|
||||
- **Album** / **AlbumHasTrack**: an album with 1-based track numbers. **PlayList** / **PlayListHasTrack**: a playlist with 0-based positions and a `Visibility` (Private or Public). Both use `EntityPictures` (resized names stay null until processed).
|
||||
- **Mix** / **MixItem**: server-generated daily mixes, identified by `MixKind` (the web-player renders the title).
|
||||
- **UploadIntent**: an upload reservation (key, bucket, purpose, expected size, status, expiration) used for validation and quota.
|
||||
|
||||
Every entity that can be listed or removed implements `IHasLifeCycle`; public reads return only `Active` rows, and the ownership checks (`FindOwnedAsync`) treat non-active entities as not found.
|
||||
|
||||
@@ -15,7 +15,7 @@ Login and token issuance are handled by **Zitadel**, not the API. The API is a *
|
||||
## `sub` is numeric → `User.Id` is `long`
|
||||
|
||||
Zitadel's `sub` is a 64-bit integer encoded as a string (e.g. `371953080444977155`), **not a GUID**. Because of that:
|
||||
- `User.Id` and every user-related FK (`PlayList.UserId`, `UserHasTrack.UserId`, `UploadIntent.UserId`) are **`long`**.
|
||||
- `User.Id` and every user-related FK (`PlayList.OwnerUserId`, `Track.OwnerUserId`, `UploadIntent.UserId`) are **`long`**.
|
||||
- `User.Id` has `[DatabaseGenerated(None)]` so EF doesn't autogenerate it: it's assigned from the `sub`.
|
||||
- `CurrentUser.Id` is `long?` (parsed with `long.TryParse`); `RequiredId` throws if it's missing.
|
||||
|
||||
|
||||
+4
-3
@@ -87,13 +87,14 @@ dotnet test backend/Musify.slnx
|
||||
| Project | Covers | Docker |
|
||||
|---|---|---|
|
||||
| `Musify.Domain.Tests` | Value object logic (`TrackAudio`/`TrackPictures`: `IsProcessed`, `IsFailed`, `IsInProgress`). Entities are mostly anemic, with no behavior of their own beyond that. | No |
|
||||
| `Musify.Application.Tests` | The ~45 handlers and services in `Musify.Application` (Albums, Tracks, PlayLists, Users, Mixes). | No |
|
||||
| `Musify.Infrastructure.Tests` | `Database`/migrations, `StorageService`, `StreamTicketService`, `AuditableEntityInterceptor`, `PictureService`, `PlayListPresetSeeder`. | **Yes** |
|
||||
| `Musify.Application.Tests` | The handlers and services in `Musify.Application` (Albums, Tracks, PlayLists, Users, Mixes...). Its `TestDatabase` builds the model with `ModelConfiguration.Apply` from Infrastructure, so it can't drift from the real `Database`. | No |
|
||||
| `Musify.Infrastructure.Tests` | `Database`/migrations, `StorageService`, `StreamTicketService`, `AuditableEntityInterceptor`, `PictureService`, plus the MassTransit layer (lifecycle and picture-update activities, routing slip builders, `ProcessingSlipFaultConsumer`, the saga state machines on the MassTransit test harness) and `RecurringJobsRegistrar`. | **Yes** |
|
||||
| `Musify.Api.Tests` | Real HTTP endpoints (routing, auth, `ValidationFilter`, `ErrorOr`→HTTP mapping) via `WebApplicationFactory`, plus `ErrorOrHttpExtensions`/`CurrentUser`/FluentValidation validators in isolation. | **Yes** |
|
||||
| `Musify.StreamingGateway.Tests` | `TicketValidator` (RS256) and `TicketValidationMiddleware` (traversal, prefix limits, ticket extraction). | No |
|
||||
|
||||
`Musify.Worker` has no test project: its `Program.cs` is pure DI/host wiring
|
||||
(Hangfire, MassTransit, jobs). The actual logic it runs lives in
|
||||
(Hangfire, MassTransit, jobs), and the container is validated on build, so a
|
||||
missing registration fails at startup. The actual logic it runs lives in
|
||||
`Musify.Infrastructure`, already covered there.
|
||||
|
||||
### Application.Tests: why in-memory SQLite instead of mocks
|
||||
|
||||
@@ -10,22 +10,24 @@ Heavy processing (audio transcoding, thumbnails, bucket transfers) happens **out
|
||||
|
||||
## Trigger
|
||||
|
||||
When a track or playlist is created or updated, the handler publishes an event (e.g. `CreateTrackResourcesEvent`). A **consumer** (`CreateTrackConsumer`, `UpdateTrackPictureConsumer`, …) builds a **routing slip** with the sequence of activities and executes it.
|
||||
When a track, album or playlist is created, updated or deleted, the handler publishes an event (e.g. `CreateTrackResourcesEvent`, `UpdateAlbumPictureSourceEvent`, `DeleteAlbumEvent`). A **consumer** (`CreateTrackConsumer`, `UpdateTrackPictureConsumer`, `DeleteAlbumConsumer`, …) builds a **routing slip** with the sequence of activities and executes it. Every slip starts from `RoutingSlips.Create(correlationId)` and adds steps with `AddStep(...)`; `TrackFaults(subjectId, processKind)` makes a failed slip publish a failure event for its subject (`ProcessingSlipFaultConsumer`, one process kind per workflow: `TrackPicture`, `TrackAudio`, `PlayListPicture`, `AlbumPicture` and the three `*Creation` kinds).
|
||||
|
||||
## Routing slip builders
|
||||
|
||||
- `CreateTrackRoutingSlipBuilder`: creating a track (consumes intents, moves files, kicks off processing).
|
||||
- `PictureWorkflowRoutingSlipBuilder`: image pipeline (download → resize → upload → update).
|
||||
- `AudioWorkflowRoutingSlipBuilder`: audio pipeline (download → transcode to `.m4a` → upload folder → update).
|
||||
- `DeleteTrackRoutingSlipBuilder`, `DeletePlayListRoutingSlipBuilder`, `PlayListPictureSourceRoutingSlipBuilder`.
|
||||
- `DeleteRoutingSlipBuilder`: marks a track, album or playlist as `Removing`, removes its files from the bucket and finally deletes the row.
|
||||
- `PictureSourceRoutingSlipBuilder`: album and playlist pictures (copy the uploaded file to its final key, consume the intent, publish the event that starts the picture workflow), for both creation and update.
|
||||
|
||||
## Activities (steps)
|
||||
|
||||
Grouped by area:
|
||||
- **Files**: `DownloadFileFromBucketActivity`, `UploadFileToBucketActivity`, `TransferFilesToBucketActivity`, `CopyFileInBucketActivity`, `RemoveFileFromBucketActivity`.
|
||||
- **Audio**: `GenerateAudioWorkflowPathsActivity`, `TranscodeAudioActivity` (ffmpeg → `.m4a`), `UpdateTrackAudioActivity` (saves `AudioFolderName` and marks it `Completed`).
|
||||
- **Pictures**: `GeneratePictureWorkflowPathsActivity`, `ResizePictureActivity` (ImageSharp), `UpdateTrackPictureActivity`, `UpdatePlayListPictureActivity`.
|
||||
- **Tracks/Playlists**: `MarkTrackAsRemovingActivity`, `DeleteTrackFromDbActivity`, `PublishTrackProcessingEventsActivity`, and their playlist equivalents.
|
||||
- **Pictures**: `GeneratePictureWorkflowPathsActivity`, `ResizePictureActivity` (ImageSharp), `UpdateTrackPictureActivity`, `UpdatePlayListPictureActivity`, `UpdateAlbumPictureActivity` (a shared `UpdatePicturesActivity<T>` base; compensation restores the previous names).
|
||||
- **Life cycle**: `Mark{Track,PlayList,Album}LifeCycleActivity` and `Delete{Track,PlayList,Album}Activity` (generic over the entity; a missing row is skipped).
|
||||
- **Events**: `PublishTrackProcessingEventsActivity`, `PublishPlayListPictureProcessingEventActivity`, `PublishAlbumPictureProcessingEventActivity`.
|
||||
- **UploadIntents**: `ConsumeUploadIntentsActivity`.
|
||||
|
||||
`RoutingSlipCleanUpConsumer` reacts to `Completed`/`Faulted` to clean up (e.g. deleting the temporary working directory).
|
||||
@@ -36,11 +38,15 @@ Grouped by area:
|
||||
|
||||
## What the Worker needs configured
|
||||
|
||||
The Worker registers: storage (S3), transcoder (ffmpeg), pictures, the DB, the MassTransit consumers, and the upload-intent jobs. Its `appsettings` needs the right sections: `Bucket` (= `webapi-storage`, same as the API), `PlayList`, `Track`, `MassTransit`, `AudioTranscoder`, `UploadIntent`, `Workers` (temp directory), `InfrastructureStorage`, `Database`.
|
||||
The Worker calls `AddApplication` (every Application handler is registered, so its container is validated on build), plus observability, the DB, storage (S3), the transcoder (ffmpeg), pictures, stream tickets, the MassTransit consumers and the Hangfire jobs. Its `appsettings.json` mirrors the API for the shared sections: `ApplicationStorage` (the same bucket as the API), `PlayList`, `Album`, `Track`, `Mix`, `UploadIntent`, `StreamGateway`, `StreamTicket` and `Playback` (the handlers need them to be constructed; the Worker never issues tickets, so the private key is never read), and adds `MassTransit`, `AudioTranscoder`, `Workers` (temp directory), `InfrastructureStorage`, `Database` and `OpenTelemetry`.
|
||||
|
||||
## Background jobs
|
||||
|
||||
- `UploadIntentExpirationJob`: marks expired intents as such (freeing quota) every `ExpirationJobIntervalSeconds`.
|
||||
- `TemporalUploadsCleanUpJob`: cleans up old temporary uploads.
|
||||
Recurring **Hangfire** jobs (Postgres storage), registered by `RecurringJobsRegistrar` when the Worker starts:
|
||||
|
||||
Both run on the Worker host (`AddUploadIntentJobs`). If they aren't registered anywhere, stale intents never get cleaned up on their own.
|
||||
- `DailyMixGenerationJob` (`daily-mix-generation`): regenerates the mixes every day.
|
||||
- `ListeningHistoryCleanupJob` (`listening-history-cleanup`): prunes old listens every day at 03:00.
|
||||
- `UploadIntentExpirationJob` (`upload-intent-expiration`): marks expired intents as such (freeing quota) every `ExpirationJobIntervalSeconds`.
|
||||
- `TempUploadsCleanupJob` (`temp-uploads-cleanup`): cleans up old temporary uploads under `TempRootPrefix` (`TempUploadsRetentionDays`) every `TempCleanupJobIntervalSeconds`.
|
||||
|
||||
Hangfire cron has a one-minute resolution, so intervals in seconds are rounded up to whole minutes (`RecurringJobsRegistrar.EveryCron`). If the Worker isn't running, stale intents never get cleaned up on their own.
|
||||
|
||||
+16
-13
@@ -7,13 +7,14 @@ The .NET projects live under `backend/` (solution `Musify.slnx`, folders
|
||||
|
||||
The public REST API (Minimal APIs, .NET 10). The thin public face of the system.
|
||||
|
||||
- **Endpoints**: `UserEndpoints`, `TrackEndpoints`, `PlayListEndpoints`.
|
||||
- **Endpoints**: `AlbumEndpoints`, `ConfigEndpoints`, `GenreEndpoints`, `LikeEndpoints`, `MixEndpoints`, `PlayListEndpoints`, `TrackEndpoints`, `UserEndpoints`. Every endpoint returns `IResult`; routes use `{id:guid}` (sub-resources `{trackId:guid}`) and `{id:long}` / `{userId:long}` for user ids; list endpoints bind `PageQuery` (`pageNumber`, `pageSize` 1..100). The three `/cover` routes share `CoverEndpoint.Stream`.
|
||||
- **Validation**: FluentValidation validators in `Validators/`; every limit lives in `Validators/Limits.cs` (the web-player copies the same values) and the upload allow-lists in `Validators/Uploads.cs`.
|
||||
- **Authentication**: JWT Bearer against Zitadel (`Authentication/`), with a `JwtBearerEventsHandler` that syncs the user on every validated token.
|
||||
- **`CurrentUser`**: an abstraction bound as an endpoint parameter (`IBindableFromHttpContext`) that exposes the basic claims (`Id` as `long?`, `RequiredId`, name, etc.) without reading `ClaimsPrincipal` by hand.
|
||||
- **Issues**: presigned upload URLs (via `IStorageService`) and **stream tickets** (via `IStreamTicketService`, RS256).
|
||||
- **OpenAPI + Scalar** for docs/testing; OAuth2 (PKCE) set up for logging in from Scalar.
|
||||
- **CORS** wide open, but **only in Development** (for the local frontend).
|
||||
- Translates `ErrorOr` into HTTP in `ResultHttpExtensions`.
|
||||
- Translates `ErrorOr` into HTTP (ProblemDetails) in `ErrorOrHttpExtensions`.
|
||||
|
||||
**Purpose**: keep the public surface (authorization + metadata) separate from processing and byte-serving.
|
||||
|
||||
@@ -26,30 +27,31 @@ Entities (`Entities/`) and value objects/enums (`ValueObjects/`: `ProcessingStat
|
||||
|
||||
### Application
|
||||
Use cases (CQRS with Mediator):
|
||||
- `Tracks/`, `PlayLists/`, `Users/` → `Commands`, `Queries`, `Handlers`, `Responses`.
|
||||
- One folder per feature (`Albums/`, `Genres/`, `Likes/`, `Mixes/`, `Pictures/`, `PlayLists/`, `Tracks/`, `Users/`, …): each `<UseCase>.cs` holds the command/query record and its handler; DTOs live in `Responses/`.
|
||||
- `Contracts/` → interfaces that infrastructure implements (`IDatabase`, `IStorageService`, `IAudioTranscoderService`, `IPictureService`, `IEventBus`, `IStreamTicketService`).
|
||||
- `Configuration/` → typed options (Track, PlayList, ApplicationStorage, UploadIntent…) bound and validated at startup.
|
||||
- `Shared/` → cross-cutting helpers (`StorageKey.Combine`, `ImageSize`).
|
||||
- `Pagination/` → `PaginatedResponse<T>`.
|
||||
- `Configuration/` → typed options (Track, PlayList, Album, Mix, ApplicationStorage, UploadIntent, Playback, StreamGateway…), each ending in `Configuration`, bound and validated at startup with `AddValidatedOptions`; `Pictures.cs` holds the shared picture routes and sizes.
|
||||
- `Shared/` → cross-cutting helpers: `StorageKey.Combine`, `ImageSize(s)`, `AppErrors`, `PageRequest` / `PaginatedResponse<T>`, `OwnershipExtensions.FindOwnedAsync`, `TextNormalizer`.
|
||||
- `Services/` → `UploadIntentIssuer` / `UploadIntentValidator` (upload intents) and `TrackStreamIssuer` (stream tickets).
|
||||
- `AddApplication(configuration)` registers all of the above; the API and the Worker both call it.
|
||||
|
||||
**Why**: business logic doesn't know about EF, S3, or RabbitMQ, only interfaces. Shared by both the API and the Worker.
|
||||
|
||||
### Infrastructure
|
||||
Concrete implementations:
|
||||
- `Persistence/` → `Database` (EF Core/Npgsql), migrations.
|
||||
- `Persistence/` → `Database` (EF Core/Npgsql), migrations, and `ModelConfiguration.Apply` (the entity model, shared with the in-memory test database).
|
||||
- `Services/` → `StorageService` (AWSSDK.S3 against SeaweedFS), `AudioTranscoderService` (ffmpeg → `.m4a`), `PictureService` (ImageSharp), `StreamTicketService` (RS256 signing).
|
||||
- `MassTransit/` → consumers, activities, and routing slips (see [media-processing.md](media-processing.md)).
|
||||
- `Jobs/` → `UploadIntentExpirationJob`, `TemporalUploadsCleanUpJob`.
|
||||
- `MassTransit/` → consumers, activities, routing slips and sagas (see [media-processing.md](media-processing.md)).
|
||||
- `Jobs/` → the recurring Hangfire jobs (`DailyMixGenerationJob`, `ListeningHistoryCleanupJob`, `UploadIntentExpirationJob`, `TempUploadsCleanupJob`) and `RecurringJobsRegistrar`.
|
||||
|
||||
## Musify.StreamingGateway (`:8081`)
|
||||
|
||||
A reverse proxy (ASP.NET + **YARP**) that serves audio (`.m4a`) from SeaweedFS. Validates the stream ticket (RS256, public key) and, if the requested object falls under the authorized prefix, forwards it to the filer.
|
||||
A reverse proxy (ASP.NET + **YARP**) that serves audio (`.m4a`) from SeaweedFS. Validates the stream ticket (RS256, public key) and, if the requested object falls under the authorized prefix, forwards it to the filer. It references neither Application nor Infrastructure and registers its own options. The bucket in the default YARP transform is the placeholder `CHANGE_ME` (compose overrides it; `appsettings.Development.json` sets `webapi-storage`).
|
||||
|
||||
**Why a separate project**: it scales with byte traffic independently of the API, has a minimal surface (it only knows how to validate a JWT and proxy), and lets SeaweedFS stay on a private network. See [streaming.md](streaming.md).
|
||||
|
||||
## Musify.Worker
|
||||
|
||||
A background Worker Service that consumes MassTransit events and runs the heavy processing: audio transcoding to `.m4a` (ffmpeg), thumbnail generation (ImageSharp), file transfers within the bucket, and the upload-intent expiration/cleanup jobs.
|
||||
A background Worker Service that consumes MassTransit events and runs the heavy processing: audio transcoding to `.m4a` (ffmpeg), thumbnail generation (ImageSharp), file transfers within the bucket, and the recurring Hangfire jobs (daily mixes, listening-history cleanup, upload-intent expiration, temporary upload cleanup).
|
||||
|
||||
**Why separate**: CPU/IO-heavy work shouldn't happen inside an HTTP request; it scales and restarts independently. (Previously named `Musify.PictureWorker`; renamed once it started doing more than just images.)
|
||||
|
||||
@@ -63,8 +65,9 @@ with the access token, and uploads go straight to the presigned S3 URLs.
|
||||
|
||||
`backend/Tests/` (the `/Tests/` folder in the `.slnx`) has five xUnit projects:
|
||||
`Musify.Domain.Tests`, `Musify.Application.Tests` (in-memory SQLite, no
|
||||
Docker), `Musify.Infrastructure.Tests` and `Musify.Api.Tests` (Testcontainers:
|
||||
real Postgres + SeaweedFS), and `Musify.StreamingGateway.Tests`. Details in
|
||||
Docker), `Musify.Infrastructure.Tests` (Testcontainers: real Postgres +
|
||||
SeaweedFS; also covers the MassTransit activities, routing slips, sagas and
|
||||
jobs), `Musify.Api.Tests` (Testcontainers) and `Musify.StreamingGateway.Tests`. Details in
|
||||
[development.md](development.md#tests).
|
||||
|
||||
## Other folders
|
||||
|
||||
+8
-6
@@ -13,26 +13,28 @@ App bucket: **`webapi-storage`** (`ApplicationStorage:Bucket`). Important: the A
|
||||
|
||||
## Key layout
|
||||
|
||||
- `temp/{userId}/Tracks/{guid}.{ext}`: temporary uploads, the target of presigned PUTs, before the intent is consumed.
|
||||
- `uploads/{userId}/Tracks/OriginalPictures|OriginalAudios/...`: originals that have already been consumed.
|
||||
- `Tracks/SmallPictures|MediumPictures|LargePictures/...`: generated thumbnails.
|
||||
The layout is configured per owner (the `Track`, `PlayList` and `Album` sections, `Routes`); `ParentFolder` is `Tracks`, `PlayLists` or `Albums`.
|
||||
|
||||
- `{TempRootPrefix}/{userId}/{ParentFolder}/{guid}.{ext}`: temporary uploads, the target of presigned PUTs, before the intent is consumed (`UploadIntent:TempRootPrefix`, default `temp`; objects under the old `temporal/` prefix from earlier versions may need a one-off clean-up).
|
||||
- `uploads/{userId}/{ParentFolder}/OriginalPictures|OriginalAudios/...`: originals that have already been consumed.
|
||||
- `{ParentFolder}/SmallPictures|MediumPictures|LargePictures/...`: generated thumbnails.
|
||||
- `Tracks/ProcessedAudios/{AudioFolderName}/`: transcoded audio (`audio.m4a`, AAC with faststart).
|
||||
|
||||
Keys are always built with the `StorageKey.Combine(...)` helper (in `Application/Shared`), which joins segments with `/` and trims stray spaces and slashes. It's centralized to keep the logic consistent and avoid repeating it.
|
||||
|
||||
> In the filer, S3 objects live under `/buckets/{bucket}/{key}`, which is why the gateway rewrites `/media/{key}` → `/buckets/webapi-storage/{key}`.
|
||||
> In the filer, S3 objects live under `/buckets/{bucket}/{key}`, which is why the gateway rewrites `/media/{key}` → `/buckets/{bucket}/{key}` (the bucket comes from the compose override `S3_BUCKET`; the default in the gateway's `appsettings.json` is the placeholder `CHANGE_ME`).
|
||||
|
||||
## Presigned URLs
|
||||
|
||||
To **avoid** routing large files through the API, the client talks to S3 directly using short-lived signed URLs:
|
||||
|
||||
- **Uploads (PUT)**: `RequestTrackUploadUrls` / `RequestPlayListPictureUpload` generate presigned PUTs (with signed `Content-Type` and `If-None-Match: *`, ~120s expiry). The client uploads straight to S3.
|
||||
- **Uploads (PUT)**: `RequestTrackUploadUrls` / `RequestAlbumPictureUpload` / `RequestPlayListPictureUpload` (issued by `UploadIntentIssuer`) generate presigned PUTs (with signed `Content-Type` and `If-None-Match: *`, ~120s expiry). The client uploads straight to S3.
|
||||
- **Images (GET)**: thumbnails are served via presigned GETs (one object = one URL).
|
||||
|
||||
**Why presigned instead of proxying**: routing a ~100 MB audio file through the API would burn memory and bandwidth on the application servers and wouldn't scale well. A direct presigned PUT to S3 is the standard, correct pattern here.
|
||||
|
||||
### Upload intents and quota
|
||||
Every `upload-urls` call creates **UploadIntents** (reservations in the `Issued` state) and returns the URLs. They're **consumed** when the track/playlist is created. There's a per-user quota (`MaxActiveUploadIntentsPerUser`, byte limits…) to prevent abuse. The `UploadIntentExpirationJob` (in the Worker) marks expired intents and frees up quota. If that job isn't running, stale intents pile up and block new uploads.
|
||||
Every `upload-urls` call creates **UploadIntents** (reservations in the `Issued` state) and returns the URLs. They're **consumed** when the track/playlist is created. There's a per-user quota (`MaxActiveUploadIntentsPerUser`, byte limits…) to prevent abuse. The `UploadIntentExpirationJob` (a Hangfire job on the Worker) marks expired intents and frees up quota, and `TempUploadsCleanupJob` deletes abandoned temporary objects. If the Worker isn't running, stale intents pile up and block new uploads. An intent is bound to its purpose (track picture/audio, album picture, playlist picture) and the same intent can't be used for two files.
|
||||
|
||||
## What's public and what's private (deployment)
|
||||
|
||||
|
||||
+3
-3
@@ -37,7 +37,7 @@ Access is authorized for **the track's folder prefix** via a short-lived *stream
|
||||
`TicketValidationMiddleware`, on every request to `/media/**`:
|
||||
1. Reads `?t=`, validates its signature/aud/iss/exp.
|
||||
2. Checks that the requested key **starts with** the ticket's `prefix`. If not → 403.
|
||||
3. Strips the `?t=` and passes the request on to YARP, which rewrites `/media/{key}` → `/buckets/webapi-storage/{key}` for the filer.
|
||||
3. Strips the `?t=` and passes the request on to YARP, which rewrites `/media/{key}` → `/buckets/{bucket}/{key}` for the filer (the bucket is set by the compose override `ReverseProxy__Routes__media__Transforms__0__PathPattern`; the default in `appsettings.json` is the placeholder `CHANGE_ME`, and `appsettings.Development.json` uses `webapi-storage`).
|
||||
|
||||
- No ticket → **401**; outside the prefix → **403**; ok → **200** (and **206** for range requests, so seeking works).
|
||||
- Passes through `Range`/`Accept-Ranges` and exposes those headers via CORS.
|
||||
@@ -50,6 +50,6 @@ Yes. Each ticket is scoped to **one** folder (one track). Playing a different on
|
||||
|
||||
## Open items / security notes
|
||||
|
||||
- Today, the policy for who can request a ticket is "any authenticated user" (TODO: restrict it with `UserHasTrack`/visibility).
|
||||
- The prefix check is a literal `StartsWith`; it would be worth hardening it against `..` path traversal, even though browsers and ASP.NET normalize `..`.
|
||||
- Who can request a ticket is decided by `StartListeningCommand`: any authenticated user, or anonymous users when `Playback:AllowAnonymousListening` is on (the ticket then carries `maxBytes`, an anonymous preview limit). TODO: restrict it by track visibility.
|
||||
- The prefix check is a literal `StartsWith` on the key, and the gateway rejects any `.`/`..` segment with 400. For tickets with `maxBytes` it clamps the `Range` header to the preview size, replaces a missing or invalid range, and answers 416 to suffix ranges or ranges that start beyond the limit.
|
||||
- The ticket travels in the query string (`?t=`); the gateway strips it before forwarding, so it never reaches the filer's logs.
|
||||
|
||||
Reference in New Issue
Block a user