Admin
23e30d96fc
Translate all of docs/ from Spanish to English and rewrite the root README as a short project overview instead of a technical run guide. Remove web-player/design_handoff_musify_player, an old design-tool export that was no longer needed.
49 lines
3.5 KiB
Markdown
49 lines
3.5 KiB
Markdown
# Storage (SeaweedFS / S3)
|
|
|
|
## What it is
|
|
|
|
**SeaweedFS** is the object store, S3-API compatible. It holds the uploaded originals and their derivatives (`.m4a` audio, thumbnails). Accessed via `AWSSDK.S3` from `StorageService` (`IStorageService`).
|
|
|
|
It exposes:
|
|
- **S3 API** (`:8333`): S3 operations (presigned URLs, put/get/list…).
|
|
- **filer HTTP** (`:8888`): serves files by path, used by the **StreamingGateway**.
|
|
- master (`:9333`) and volume server, internal only.
|
|
|
|
App bucket: **`webapi-storage`** (`ApplicationStorage:Bucket`). Important: the API and the Worker use **the same bucket** (the worker processes whatever the API uploads).
|
|
|
|
## Key layout
|
|
|
|
- `temp/{userId}/Tracks/{guid}.{ext}`: temporary uploads, the target of presigned PUTs, before the intent is consumed.
|
|
- `uploads/{userId}/Tracks/OriginalPictures|OriginalAudios/...`: originals that have already been consumed.
|
|
- `Tracks/SmallPictures|MediumPictures|LargePictures/...`: generated thumbnails.
|
|
- `Tracks/ProcessedAudios/{AudioFolderName}/`: transcoded audio (`audio.m4a`, AAC with faststart).
|
|
|
|
Keys are always built with the `StorageKey.Combine(...)` helper (in `Application/Shared`), which joins segments with `/` and trims stray spaces and slashes. It's centralized to keep the logic consistent and avoid repeating it.
|
|
|
|
> In the filer, S3 objects live under `/buckets/{bucket}/{key}`, which is why the gateway rewrites `/media/{key}` → `/buckets/webapi-storage/{key}`.
|
|
|
|
## Presigned URLs
|
|
|
|
To **avoid** routing large files through the API, the client talks to S3 directly using short-lived signed URLs:
|
|
|
|
- **Uploads (PUT)**: `RequestTrackUploadUrls` / `RequestPlayListPictureUpload` generate presigned PUTs (with signed `Content-Type` and `If-None-Match: *`, ~120s expiry). The client uploads straight to S3.
|
|
- **Images (GET)**: thumbnails are served via presigned GETs (one object = one URL).
|
|
|
|
**Why presigned instead of proxying**: routing a ~100 MB audio file through the API would burn memory and bandwidth on the application servers and wouldn't scale well. A direct presigned PUT to S3 is the standard, correct pattern here.
|
|
|
|
### Upload intents and quota
|
|
Every `upload-urls` call creates **UploadIntents** (reservations in the `Issued` state) and returns the URLs. They're **consumed** when the track/playlist is created. There's a per-user quota (`MaxActiveUploadIntentsPerUser`, byte limits…) to prevent abuse. The `UploadIntentExpirationJob` (in the Worker) marks expired intents and frees up quota. If that job isn't running, stale intents pile up and block new uploads.
|
|
|
|
## What's public and what's private (deployment)
|
|
|
|
| Expose to the internet | Keep private |
|
|
|---|---|
|
|
| S3 API `:8333` (behind TLS), needed for the client's presigned PUT/GET | filer `:8888` (internal network, gateway only) |
|
|
| StreamingGateway `:8081` | master `:9333`, volume server, admin UI |
|
|
|
|
- **Audio** is served through the gateway, so SeaweedFS can stay private for that flow.
|
|
- **Uploads and images** use presigned URLs, so the client talks to the S3 endpoint directly and `:8333` has to be reachable (though useless without a valid signature, this is the standard model for any S3/MinIO bucket).
|
|
- In **production**: `InfrastructureStorage:Address` must be the **public S3 endpoint** (the signature is computed for that host). If the servers can't reach it internally, use split config: public host for signing, internal host for server-side operations.
|
|
|
|
**Rule of thumb**: the gateway, and at most the signed `:8333`, are the only public surface. Master, volume, filer and admin are **never** exposed.
|