Files
AudioManager/README.md
T
TapTap 44b75c4d4d
CI / chart (push) Successful in 3s
CI / test (push) Successful in 2m20s
CI / chart-push (push) Successful in 6s
CI / image (push) Successful in 3m39s
feat: accept, store and expose client diagnostics
- V11 client_logs table with per-user storage and retention/pruning
- POST/GET/DELETE /logs with batch validation, rate limit and admin access
- redact bearer tokens, Authorization headers and query/JSON secrets twice
- add X-Request-Id to responses and include it in the server log pattern
- report client error entries immediately in the server log
- scripts/logs.sh for source-side retrieval and clearing
- chart knobs CLIENT_LOG_RETENTION_DAYS and CLIENT_LOG_MAX_ROWS_PER_USER
2026-11-20 13:43:05 +00:00

389 lines
16 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# AudioManager
A self-hosted, mood-first music server. You create private moods, add tracks
from Deezer or YouTube Music, and the server downloads them into a shared media
store. Recommendations are discovered and downloaded automatically so a mood
keeps playing without a mounted music library.
- Server: Kotlin, Ktor, SQLite (Exposed + Flyway), `yt-dlp`, `streamrip`,
`ffmpeg`.
- Client: Compose Multiplatform (Android, desktop, Wasm).
- No uploads, no mounted library, no likes, no podcasts.
## Contents
- [How it works](#how-it-works)
- [Quick start](#quick-start)
- [Configuration](#configuration)
- [Native API](#native-api)
- [Architecture](#architecture)
- [Media store and GC](#media-store-and-gc)
- [Database](#database)
- [Deployment](#deployment)
- [Client](#client)
- [Testing](#testing)
## How it works
- **Moods are private.** A user sees only their own moods. Songs are shared
server-wide and deduplicated by `(source, externalId)`.
- **Two catalogs.** Every mood has a download source: **Deezer** (default) or
**YouTube Music**. Search, recommendations and downloads follow that source;
permanent songs from either source can live in the same mood.
- **Songs are the collection.** A mood needs at least one song before it can
play. Removing a song only detaches it; the file is deleted when no mood,
recommendation slot or open playback session references it.
- **Recommendations are active slots.** A mood keeps `recommendationCount`
(0–30, default 10) recommendations in `downloading`, `ready` or `failed`
state. Hidden `new` candidates refill the slots automatically.
- **Discovery uses the whole mood.** Every artist (weighted by how many songs
the mood has), every album, the hint and the genres feed one pass. On Deezer
the server also asks Deezer for the artists' **radio** — a Daft Punk mood gets
MGMT, Caribou and Jamiroquai — plus related artists and album deep cuts. On
YouTube Music it follows the song's `RDAMVM` **radio queue**, the same mix the
player builds. Adding a song makes the pool refresh instead of waiting for
`refreshHours`.
- **Add / Not interested are permanent.** Adding promotes a recommendation into
the collection. Rejecting is remembered per mood. Both free a slot and the
server refills it.
- **Playback mixes both pools.** Every queue slot rolls `collectionPercent`;
the rest comes from ready recommendations. An empty pool falls back to the
other one. Small pools replay least-recently-played tracks.
- **Downloads are automatic.** Recommendation downloads run in the background
up to the configured count even if the mood is never played. Manual adds have
higher priority. Deezer needs an ARL cookie; YouTube Music needs no account.
## Quick start
Requirements: JDK 21, `ffmpeg`, `yt-dlp`, and `streamrip` (`pip install
streamrip==2.1.0`, which provides `rip`) for Deezer. The smoke test also needs
`sqlite3`, `curl` and `python3`. `nix-shell` provides everything except
`streamrip`.
```bash
# 1. Build and create local config
./gradlew setupLocal
# edit config/users.yaml (a password is required)
# 2. Run
./gradlew run
# health: http://localhost:8080/health
# API: http://localhost:8080/api/v1
```
The server creates `./data/media` (the media store), `./data/downloads`
(download temp files) and `./data/musicmanager.db`.
Deezer downloads need `DEEZER_ARL`; YouTube Music downloads need `yt-dlp` to be
able to reach YouTube. If YouTube applies bot checks, search and downloads fail
with `502 search_failed` / `failed` recommendations; retry later or update
`yt-dlp`. A Deezer Free ARL is limited to 128 kbps; 320 kbps needs Deezer
Premium (`DEEZER_QUALITY=1`).
## Configuration
Everything is environment variables:
| Variable | Default | Meaning |
|---|---|---|
| `PORT` | `8080` | HTTP port |
| `DATA_DIR` | `./data` | Database, cover cache, default media root |
| `MEDIA_DIR` | `<DATA_DIR>/media` | Server-owned song store |
| `DOWNLOAD_DIR` | `<DATA_DIR>/downloads` | yt-dlp temp/work directory |
| `USERS_FILE` | `./config/users.yaml` | User accounts |
| `FFMPEG_PATH` | `ffmpeg` | Transcoder binary |
| `YTDLP_PATH` | `yt-dlp` | YouTube Music downloader / search binary |
| `STREAMRIP_PATH` | `rip` | Deezer downloader (`streamrip`) |
| `DEFAULT_SOURCE` | `deezer` | Source for new moods: `deezer` or `youtube` |
| `DEEZER_ARL` | empty | Deezer account cookie used for downloads (chart: `deezer.arl` or `deezer.existingSecret`) |
| `DEEZER_QUALITY` | `0` | `0` = 128 kbps MP3, `1` = 320 kbps MP3 |
| `DEEZER_API_BASE` | `https://api.deezer.com` | Public Deezer API base |
| `LOUDNESS_ENABLED` | `true` | Measure each file and expose a per-track gain |
| `LOUDNESS_TARGET_LUFS` | `-16` | Integrated loudness target (`-30` to `-5`) |
| `CLIENT_LOG_RETENTION_DAYS` | `14` | Days uploaded client diagnostics are kept |
| `CLIENT_LOG_MAX_ROWS_PER_USER` | `20000` | Newest diagnostic rows kept per user |
### `users.yaml`
```yaml
users:
- name: admin
password: "change-me"
admin: true
```
Passwords live only in this file (use a Vault-managed Secret in Kubernetes).
`admin` is only needed for the maintenance scan endpoints. Old `folders:` keys
are accepted and ignored.
## Client diagnostics
The apps keep a small local ring buffer (500 entries) of warnings, errors and
crashes. Warnings and errors are flushed immediately, so the line that explains
a crash survives it; info entries are batched. When signed in, the client uploads
up to 100 entries per minute to `POST /api/v1/logs`, tagged with a random install
id, platform, app version and per-run session id.
Privacy and safety:
- `Authorization` headers, `Bearer` tokens, `token=`/`password=`/`arl=` query
parameters and JSON secret fields are replaced with `***` on the client
**and** again on the server.
- The server stores rows per user; a user can only read their own logs. An
administrator can pass `user=<name>`.
- Retention defaults to 14 days and 20,000 rows per user; both are configurable.
- A rejected batch (HTTP 400/413/422) is dropped so it cannot block later logs.
Inspect them with the helper:
```bash
scripts/logs.sh --server https://audiomanager.example.com --user alice \
--level warn --limit 100
# administrators can read another account
scripts/logs.sh --server ... --user admin --target alice
# clear after investigating
scripts/logs.sh --server ... --user alice --clear
```
The password is read from `LOGS_PASSWORD` or prompted; it is never stored.
## Native API
All endpoints are under `/api/v1` and require a bearer token, except login.
Media URLs may use `?token=` because HTML audio and images cannot send headers.
### Authentication
| Method | Path | Purpose |
|---|---|---|
| `POST` | `/auth/login` | `{username,password,deviceName,platform}` → token |
| `POST` | `/auth/logout` | Revoke the current token |
| `GET` | `/me` | Current user |
| `GET` | `/info` | Default source, Deezer quality, loudness and log retention |
| `GET` | `/auth/devices` | List device tokens |
| `DELETE` | `/auth/devices/{id}` | Revoke a device |
### Diagnostics
| Method | Path | Purpose |
|---|---|---|
| `POST` | `/logs` | Upload a client diagnostics batch |
| `GET` | `/logs` | List diagnostics (`level`, `device`, `since`, `before`, `limit`; admins may add `user`) |
| `DELETE` | `/logs` | Delete the current user's (or, for admins, a named user's) diagnostics |
### Moods
| Method | Path | Purpose |
|---|---|---|
| `GET` | `/moods` | List the user's moods |
| `POST` | `/moods` | Create `{name,description,query,genres,source,collectionPercent,recommendationCount,refreshHours}` |
| `GET` | `/moods/{id}` | Mood detail: songs, active recommendations, download status |
| `PATCH` | `/moods/{id}` | Update options |
| `DELETE` | `/moods/{id}` | Delete mood, detach songs and run GC |
| `GET` | `/moods/{id}/preview` | Unpersisted radio preview |
| `POST` | `/moods/{id}/songs` | Add `{songId}`, `{source,externalId}` or `{url}` |
| `DELETE` | `/moods/{id}/songs/{songId}` | Detach a song |
| `GET` | `/moods/{id}/recommendations` | Active recommendation slots |
| `POST` | `/moods/{id}/recommendations/refresh` | Force discovery + top-up |
| `POST` | `/moods/{id}/recommendations/{externalId}/add` | Add to the collection |
| `POST` | `/moods/{id}/recommendations/{externalId}/reject` | Hide permanently |
| `POST` | `/moods/{id}/recommendations/{externalId}/retry` | Retry a failed download |
Moods accept `source` on create/update; switching it clears the active
recommendation pool and refills it from the new catalog.
Adding an already downloaded song attaches instantly (`200`). A new track is
queued for download (`202`, `downloading: true`) and is attached when the job
finishes. `/moods/{id}/songs` accepts `{"songId":...}`, `{"source":"deezer",
"externalId":"..."}`, or `{"url":"..."}` (Deezer or YouTube).
### Search
| Method | Path | Purpose |
|---|---|---|
| `GET` | `/search?q=...&limit=...&source=...&moodId=...` | Deezer or YouTube Music search |
Results longer than 10 minutes (or shorter than 30 seconds) are hidden. Each
result reports `source`, `externalId`, `downloaded`, `songId` and `inMood`.
`source` defaults to the target mood's source, then to `DEFAULT_SOURCE`.
Deezer search uses the public API and returns canonical tracks with a square
album cover. YouTube Music search uses the "Songs" filter and resolves every
hit, which takes roughly 10–15 seconds for ten results; if Music search fails,
it falls back to plain YouTube search.
### Playback
| Method | Path | Purpose |
|---|---|---|
| `POST` | `/moods/{id}/sessions` | Start a queue (`400 empty_mood` without songs) |
| `GET` | `/sessions/{id}` | Resume a session |
| `POST` | `/sessions/{id}/next` | Fetch more tracks |
| `POST` | `/sessions/{id}/events` | Report `started`, `completed`, `skipped` |
| `POST` | `/sessions/{id}/add-to-mood` | Add the playing track to a mood |
| `DELETE` | `/sessions/{id}` | Close the session |
### Media and maintenance
| Method | Path | Purpose |
|---|---|---|
| `GET` | `/songs/{id}/stream` | Range-aware stream; `maxBitRate`, `timeOffset`, `download` |
| `GET` | `/songs/{id}/cover?size=` | Embedded artwork |
| `GET` | `/scan` | Scan status |
| `POST` | `/scan` | Start a scan (admin only) |
## Architecture
```
Ktor routes
└── MoodRepository ── moods, mood_songs, recommendation lifecycle
└── SongRepository ── shared song store keyed by (source, externalId)
└── RadioService ── mixes collection + ready recommendations
└── MusicCatalogs ── Deezer public API + yt-dlp YouTube Music search
└── DownloadService── single worker: yt-dlp/streamrip → media store → index
└── DiscoveryService── mood source search → hidden `new` candidates
└── TopUpService ── keeps active slots at recommendationCount
└── MediaScanner ── rebuilds the index from the media root
└── MediaGcService ── refcounted file deletion
```
Background loops: download worker (continuous), recommendation top-up (hourly),
media GC (hourly). A scan runs at startup.
## Media store and GC
Layout:
```
<MEDIA_DIR>/<source>-<externalId>/<sanitized-title>.mp3
```
Examples: `deezer-67238732/Instant Crush.mp3`, `youtube-dQw4w9WgXcQ/...`.
Older YouTube downloads in `<MEDIA_DIR>/<videoId>/` are still read and treated
as YouTube. One row and one file exist per track, no matter how many moods or
users reference it. Covers are cached under `<DATA_DIR>/covers` keyed by song
id.
Deezer is downloaded with `streamrip` (MP3 128/320) and YouTube Music with
`yt-dlp` (`-x --audio-format mp3 --audio-quality 0`). Both embed the artist,
album and cover art, so the indexer can derive the metadata from the file.
A file is deleted only when it has no references from `mood_songs`, an active
recommendation (`downloading`, `ready`, `added`, `failed`), or an open playback
session. GC runs after removals, after rejections, after mood deletion, and
hourly.
## Database
SQLite in WAL mode. Migrations are Flyway scripts under
`src/main/resources/db/migration`.
- **V8 is destructive.** It drops every table from the mounted-library design
(folders, artists, albums, songs, moods, seeds, sessions, longform progress,
downloads) and creates the mood-first schema. `users` and `api_tokens` are
kept.
- **V9 adds sources.** It renames `video_id` to `external_id` and adds a
`source` column to songs, moods, recommendations and download jobs. Existing
rows become `youtube`.
- There is no import from an old media directory. Back up externally and start
with an empty `MEDIA_DIR`.
## Deployment
The Helm chart is in `charts/audiomanager`. It renders one PVC at `/data`:
```
/data/musicmanager.db
/data/media/<source>-<externalId>/<title>.mp3
/data/covers/
/data/downloads/
```
No hostPath, no `library.yaml`, no init container. `fsGroup: 10001` makes the
PVC writable by the non-root container.
```bash
kubectl create secret generic audiomanager-users \
--from-file=users.yaml=config/users.yaml
# Only needed for Deezer. Leave it out to use YouTube Music only.
kubectl create secret generic audiomanager-deezer \
--from-literal=arl='<deezer-arl>'
helm upgrade --install audiomanager charts/audiomanager \
--namespace audiomanager --create-namespace \
--set users.existingSecret=audiomanager-users \
--set deezer.existingSecret=audiomanager-deezer \
--set deezer.quality=0 \
--set defaultSource=deezer \
--set image.tag=<sha>
```
`scripts/release.sh --deploy` builds/pushes the image and runs the same Helm
upgrade. CI builds `:latest` and `:sha-<commit>` on every push to `main`, and
publishes the packaged chart.
### Setting the Deezer ARL in a values file
`deezer.arl` takes the cookie directly, so no Secret is needed:
```yaml
# values.local.yaml (gitignored)
image:
tag: sha-<commit>
users:
existingSecret: audiomanager-users
defaultSource: deezer
deezer:
arl: "<arl>"
quality: 0
```
```bash
helm upgrade --install audiomanager charts/audiomanager \
--namespace audiomanager --create-namespace \
-f values.local.yaml
```
Set only one of `deezer.arl` and `deezer.existingSecret`; the chart fails
otherwise. A plain value is stored in clear text in your values file, in
`helm get values` and in the release Secret, so use `deezer.existingSecret` on a
shared cluster or an untracked `values.local.yaml` here.
Before deploying the mood-first version over an old instance, wipe the old
database and media: V8 drops the content, but the old mounted files are no
longer referenced and should be removed by the operator. The old chart used the
same PVC name with a smaller size, and PVCs cannot shrink or be repopulated by
an upgrade, so recreate it:
```bash
kubectl -n audiomanager scale deployment/audiomanager --replicas=0
kubectl -n audiomanager delete pvc audiomanager-data
helm upgrade --install audiomanager charts/audiomanager \
--namespace audiomanager --reset-values \
--set users.existingSecret=audiomanager-users --set image.tag=<sha>
```
Old hostPath music/longform files can be deleted manually. There is no import.
## Client
Compose Multiplatform client: `/home/taptap/AudioManagerClient`.
- Android, desktop and Wasm from `commonMain`.
- Screens: Moods, Mood Detail, Search, Downloads, Settings.
- Every mood has a **Download source** selector (Deezer or YouTube Music);
search, recommendations and downloads follow it.
- Android background playback and media notifications are kept.
## Testing
```bash
./gradlew test # server unit/integration tests
./scripts/smoke.sh # end-to-end with a fake yt-dlp
scripts/check-chart.sh # lint + render the Helm chart
```
The smoke test seeds a tiny media store, puts deterministic fake `yt-dlp` and
`rip` binaries plus a fake Deezer API on localhost, and exercises both sources:
auth, search filtering, adding songs, URL resolve, downloads, recommendations,
radio sessions, streaming, covers and logout.