- V11 client_logs table with per-user storage and retention/pruning - POST/GET/DELETE /logs with batch validation, rate limit and admin access - redact bearer tokens, Authorization headers and query/JSON secrets twice - add X-Request-Id to responses and include it in the server log pattern - report client error entries immediately in the server log - scripts/logs.sh for source-side retrieval and clearing - chart knobs CLIENT_LOG_RETENTION_DAYS and CLIENT_LOG_MAX_ROWS_PER_USER
389 lines
16 KiB
Markdown
389 lines
16 KiB
Markdown
# AudioManager
|
||
|
||
A self-hosted, mood-first music server. You create private moods, add tracks
|
||
from Deezer or YouTube Music, and the server downloads them into a shared media
|
||
store. Recommendations are discovered and downloaded automatically so a mood
|
||
keeps playing without a mounted music library.
|
||
|
||
- Server: Kotlin, Ktor, SQLite (Exposed + Flyway), `yt-dlp`, `streamrip`,
|
||
`ffmpeg`.
|
||
- Client: Compose Multiplatform (Android, desktop, Wasm).
|
||
- No uploads, no mounted library, no likes, no podcasts.
|
||
|
||
## Contents
|
||
|
||
- [How it works](#how-it-works)
|
||
- [Quick start](#quick-start)
|
||
- [Configuration](#configuration)
|
||
- [Native API](#native-api)
|
||
- [Architecture](#architecture)
|
||
- [Media store and GC](#media-store-and-gc)
|
||
- [Database](#database)
|
||
- [Deployment](#deployment)
|
||
- [Client](#client)
|
||
- [Testing](#testing)
|
||
|
||
## How it works
|
||
|
||
- **Moods are private.** A user sees only their own moods. Songs are shared
|
||
server-wide and deduplicated by `(source, externalId)`.
|
||
- **Two catalogs.** Every mood has a download source: **Deezer** (default) or
|
||
**YouTube Music**. Search, recommendations and downloads follow that source;
|
||
permanent songs from either source can live in the same mood.
|
||
- **Songs are the collection.** A mood needs at least one song before it can
|
||
play. Removing a song only detaches it; the file is deleted when no mood,
|
||
recommendation slot or open playback session references it.
|
||
- **Recommendations are active slots.** A mood keeps `recommendationCount`
|
||
(0–30, default 10) recommendations in `downloading`, `ready` or `failed`
|
||
state. Hidden `new` candidates refill the slots automatically.
|
||
- **Discovery uses the whole mood.** Every artist (weighted by how many songs
|
||
the mood has), every album, the hint and the genres feed one pass. On Deezer
|
||
the server also asks Deezer for the artists' **radio** — a Daft Punk mood gets
|
||
MGMT, Caribou and Jamiroquai — plus related artists and album deep cuts. On
|
||
YouTube Music it follows the song's `RDAMVM` **radio queue**, the same mix the
|
||
player builds. Adding a song makes the pool refresh instead of waiting for
|
||
`refreshHours`.
|
||
- **Add / Not interested are permanent.** Adding promotes a recommendation into
|
||
the collection. Rejecting is remembered per mood. Both free a slot and the
|
||
server refills it.
|
||
- **Playback mixes both pools.** Every queue slot rolls `collectionPercent`;
|
||
the rest comes from ready recommendations. An empty pool falls back to the
|
||
other one. Small pools replay least-recently-played tracks.
|
||
- **Downloads are automatic.** Recommendation downloads run in the background
|
||
up to the configured count even if the mood is never played. Manual adds have
|
||
higher priority. Deezer needs an ARL cookie; YouTube Music needs no account.
|
||
|
||
## Quick start
|
||
|
||
Requirements: JDK 21, `ffmpeg`, `yt-dlp`, and `streamrip` (`pip install
|
||
streamrip==2.1.0`, which provides `rip`) for Deezer. The smoke test also needs
|
||
`sqlite3`, `curl` and `python3`. `nix-shell` provides everything except
|
||
`streamrip`.
|
||
|
||
```bash
|
||
# 1. Build and create local config
|
||
./gradlew setupLocal
|
||
# edit config/users.yaml (a password is required)
|
||
|
||
# 2. Run
|
||
./gradlew run
|
||
# health: http://localhost:8080/health
|
||
# API: http://localhost:8080/api/v1
|
||
```
|
||
|
||
The server creates `./data/media` (the media store), `./data/downloads`
|
||
(download temp files) and `./data/musicmanager.db`.
|
||
|
||
Deezer downloads need `DEEZER_ARL`; YouTube Music downloads need `yt-dlp` to be
|
||
able to reach YouTube. If YouTube applies bot checks, search and downloads fail
|
||
with `502 search_failed` / `failed` recommendations; retry later or update
|
||
`yt-dlp`. A Deezer Free ARL is limited to 128 kbps; 320 kbps needs Deezer
|
||
Premium (`DEEZER_QUALITY=1`).
|
||
|
||
## Configuration
|
||
|
||
Everything is environment variables:
|
||
|
||
| Variable | Default | Meaning |
|
||
|---|---|---|
|
||
| `PORT` | `8080` | HTTP port |
|
||
| `DATA_DIR` | `./data` | Database, cover cache, default media root |
|
||
| `MEDIA_DIR` | `<DATA_DIR>/media` | Server-owned song store |
|
||
| `DOWNLOAD_DIR` | `<DATA_DIR>/downloads` | yt-dlp temp/work directory |
|
||
| `USERS_FILE` | `./config/users.yaml` | User accounts |
|
||
| `FFMPEG_PATH` | `ffmpeg` | Transcoder binary |
|
||
| `YTDLP_PATH` | `yt-dlp` | YouTube Music downloader / search binary |
|
||
| `STREAMRIP_PATH` | `rip` | Deezer downloader (`streamrip`) |
|
||
| `DEFAULT_SOURCE` | `deezer` | Source for new moods: `deezer` or `youtube` |
|
||
| `DEEZER_ARL` | empty | Deezer account cookie used for downloads (chart: `deezer.arl` or `deezer.existingSecret`) |
|
||
| `DEEZER_QUALITY` | `0` | `0` = 128 kbps MP3, `1` = 320 kbps MP3 |
|
||
| `DEEZER_API_BASE` | `https://api.deezer.com` | Public Deezer API base |
|
||
| `LOUDNESS_ENABLED` | `true` | Measure each file and expose a per-track gain |
|
||
| `LOUDNESS_TARGET_LUFS` | `-16` | Integrated loudness target (`-30` to `-5`) |
|
||
| `CLIENT_LOG_RETENTION_DAYS` | `14` | Days uploaded client diagnostics are kept |
|
||
| `CLIENT_LOG_MAX_ROWS_PER_USER` | `20000` | Newest diagnostic rows kept per user |
|
||
|
||
### `users.yaml`
|
||
|
||
```yaml
|
||
users:
|
||
- name: admin
|
||
password: "change-me"
|
||
admin: true
|
||
```
|
||
|
||
Passwords live only in this file (use a Vault-managed Secret in Kubernetes).
|
||
`admin` is only needed for the maintenance scan endpoints. Old `folders:` keys
|
||
are accepted and ignored.
|
||
|
||
## Client diagnostics
|
||
|
||
The apps keep a small local ring buffer (500 entries) of warnings, errors and
|
||
crashes. Warnings and errors are flushed immediately, so the line that explains
|
||
a crash survives it; info entries are batched. When signed in, the client uploads
|
||
up to 100 entries per minute to `POST /api/v1/logs`, tagged with a random install
|
||
id, platform, app version and per-run session id.
|
||
|
||
Privacy and safety:
|
||
|
||
- `Authorization` headers, `Bearer` tokens, `token=`/`password=`/`arl=` query
|
||
parameters and JSON secret fields are replaced with `***` on the client
|
||
**and** again on the server.
|
||
- The server stores rows per user; a user can only read their own logs. An
|
||
administrator can pass `user=<name>`.
|
||
- Retention defaults to 14 days and 20,000 rows per user; both are configurable.
|
||
- A rejected batch (HTTP 400/413/422) is dropped so it cannot block later logs.
|
||
|
||
Inspect them with the helper:
|
||
|
||
```bash
|
||
scripts/logs.sh --server https://audiomanager.example.com --user alice \
|
||
--level warn --limit 100
|
||
# administrators can read another account
|
||
scripts/logs.sh --server ... --user admin --target alice
|
||
# clear after investigating
|
||
scripts/logs.sh --server ... --user alice --clear
|
||
```
|
||
|
||
The password is read from `LOGS_PASSWORD` or prompted; it is never stored.
|
||
|
||
## Native API
|
||
|
||
All endpoints are under `/api/v1` and require a bearer token, except login.
|
||
Media URLs may use `?token=` because HTML audio and images cannot send headers.
|
||
|
||
### Authentication
|
||
|
||
| Method | Path | Purpose |
|
||
|---|---|---|
|
||
| `POST` | `/auth/login` | `{username,password,deviceName,platform}` → token |
|
||
| `POST` | `/auth/logout` | Revoke the current token |
|
||
| `GET` | `/me` | Current user |
|
||
| `GET` | `/info` | Default source, Deezer quality, loudness and log retention |
|
||
| `GET` | `/auth/devices` | List device tokens |
|
||
| `DELETE` | `/auth/devices/{id}` | Revoke a device |
|
||
|
||
### Diagnostics
|
||
|
||
| Method | Path | Purpose |
|
||
|---|---|---|
|
||
| `POST` | `/logs` | Upload a client diagnostics batch |
|
||
| `GET` | `/logs` | List diagnostics (`level`, `device`, `since`, `before`, `limit`; admins may add `user`) |
|
||
| `DELETE` | `/logs` | Delete the current user's (or, for admins, a named user's) diagnostics |
|
||
|
||
### Moods
|
||
|
||
| Method | Path | Purpose |
|
||
|---|---|---|
|
||
| `GET` | `/moods` | List the user's moods |
|
||
| `POST` | `/moods` | Create `{name,description,query,genres,source,collectionPercent,recommendationCount,refreshHours}` |
|
||
| `GET` | `/moods/{id}` | Mood detail: songs, active recommendations, download status |
|
||
| `PATCH` | `/moods/{id}` | Update options |
|
||
| `DELETE` | `/moods/{id}` | Delete mood, detach songs and run GC |
|
||
| `GET` | `/moods/{id}/preview` | Unpersisted radio preview |
|
||
| `POST` | `/moods/{id}/songs` | Add `{songId}`, `{source,externalId}` or `{url}` |
|
||
| `DELETE` | `/moods/{id}/songs/{songId}` | Detach a song |
|
||
| `GET` | `/moods/{id}/recommendations` | Active recommendation slots |
|
||
| `POST` | `/moods/{id}/recommendations/refresh` | Force discovery + top-up |
|
||
| `POST` | `/moods/{id}/recommendations/{externalId}/add` | Add to the collection |
|
||
| `POST` | `/moods/{id}/recommendations/{externalId}/reject` | Hide permanently |
|
||
| `POST` | `/moods/{id}/recommendations/{externalId}/retry` | Retry a failed download |
|
||
|
||
Moods accept `source` on create/update; switching it clears the active
|
||
recommendation pool and refills it from the new catalog.
|
||
|
||
Adding an already downloaded song attaches instantly (`200`). A new track is
|
||
queued for download (`202`, `downloading: true`) and is attached when the job
|
||
finishes. `/moods/{id}/songs` accepts `{"songId":...}`, `{"source":"deezer",
|
||
"externalId":"..."}`, or `{"url":"..."}` (Deezer or YouTube).
|
||
|
||
### Search
|
||
|
||
| Method | Path | Purpose |
|
||
|---|---|---|
|
||
| `GET` | `/search?q=...&limit=...&source=...&moodId=...` | Deezer or YouTube Music search |
|
||
|
||
Results longer than 10 minutes (or shorter than 30 seconds) are hidden. Each
|
||
result reports `source`, `externalId`, `downloaded`, `songId` and `inMood`.
|
||
|
||
`source` defaults to the target mood's source, then to `DEFAULT_SOURCE`.
|
||
Deezer search uses the public API and returns canonical tracks with a square
|
||
album cover. YouTube Music search uses the "Songs" filter and resolves every
|
||
hit, which takes roughly 10–15 seconds for ten results; if Music search fails,
|
||
it falls back to plain YouTube search.
|
||
|
||
### Playback
|
||
|
||
| Method | Path | Purpose |
|
||
|---|---|---|
|
||
| `POST` | `/moods/{id}/sessions` | Start a queue (`400 empty_mood` without songs) |
|
||
| `GET` | `/sessions/{id}` | Resume a session |
|
||
| `POST` | `/sessions/{id}/next` | Fetch more tracks |
|
||
| `POST` | `/sessions/{id}/events` | Report `started`, `completed`, `skipped` |
|
||
| `POST` | `/sessions/{id}/add-to-mood` | Add the playing track to a mood |
|
||
| `DELETE` | `/sessions/{id}` | Close the session |
|
||
|
||
### Media and maintenance
|
||
|
||
| Method | Path | Purpose |
|
||
|---|---|---|
|
||
| `GET` | `/songs/{id}/stream` | Range-aware stream; `maxBitRate`, `timeOffset`, `download` |
|
||
| `GET` | `/songs/{id}/cover?size=` | Embedded artwork |
|
||
| `GET` | `/scan` | Scan status |
|
||
| `POST` | `/scan` | Start a scan (admin only) |
|
||
|
||
## Architecture
|
||
|
||
```
|
||
Ktor routes
|
||
└── MoodRepository ── moods, mood_songs, recommendation lifecycle
|
||
└── SongRepository ── shared song store keyed by (source, externalId)
|
||
└── RadioService ── mixes collection + ready recommendations
|
||
└── MusicCatalogs ── Deezer public API + yt-dlp YouTube Music search
|
||
└── DownloadService── single worker: yt-dlp/streamrip → media store → index
|
||
└── DiscoveryService── mood source search → hidden `new` candidates
|
||
└── TopUpService ── keeps active slots at recommendationCount
|
||
└── MediaScanner ── rebuilds the index from the media root
|
||
└── MediaGcService ── refcounted file deletion
|
||
```
|
||
|
||
Background loops: download worker (continuous), recommendation top-up (hourly),
|
||
media GC (hourly). A scan runs at startup.
|
||
|
||
## Media store and GC
|
||
|
||
Layout:
|
||
|
||
```
|
||
<MEDIA_DIR>/<source>-<externalId>/<sanitized-title>.mp3
|
||
```
|
||
|
||
Examples: `deezer-67238732/Instant Crush.mp3`, `youtube-dQw4w9WgXcQ/...`.
|
||
Older YouTube downloads in `<MEDIA_DIR>/<videoId>/` are still read and treated
|
||
as YouTube. One row and one file exist per track, no matter how many moods or
|
||
users reference it. Covers are cached under `<DATA_DIR>/covers` keyed by song
|
||
id.
|
||
|
||
Deezer is downloaded with `streamrip` (MP3 128/320) and YouTube Music with
|
||
`yt-dlp` (`-x --audio-format mp3 --audio-quality 0`). Both embed the artist,
|
||
album and cover art, so the indexer can derive the metadata from the file.
|
||
|
||
A file is deleted only when it has no references from `mood_songs`, an active
|
||
recommendation (`downloading`, `ready`, `added`, `failed`), or an open playback
|
||
session. GC runs after removals, after rejections, after mood deletion, and
|
||
hourly.
|
||
|
||
## Database
|
||
|
||
SQLite in WAL mode. Migrations are Flyway scripts under
|
||
`src/main/resources/db/migration`.
|
||
|
||
- **V8 is destructive.** It drops every table from the mounted-library design
|
||
(folders, artists, albums, songs, moods, seeds, sessions, longform progress,
|
||
downloads) and creates the mood-first schema. `users` and `api_tokens` are
|
||
kept.
|
||
- **V9 adds sources.** It renames `video_id` to `external_id` and adds a
|
||
`source` column to songs, moods, recommendations and download jobs. Existing
|
||
rows become `youtube`.
|
||
- There is no import from an old media directory. Back up externally and start
|
||
with an empty `MEDIA_DIR`.
|
||
|
||
## Deployment
|
||
|
||
The Helm chart is in `charts/audiomanager`. It renders one PVC at `/data`:
|
||
|
||
```
|
||
/data/musicmanager.db
|
||
/data/media/<source>-<externalId>/<title>.mp3
|
||
/data/covers/
|
||
/data/downloads/
|
||
```
|
||
|
||
No hostPath, no `library.yaml`, no init container. `fsGroup: 10001` makes the
|
||
PVC writable by the non-root container.
|
||
|
||
```bash
|
||
kubectl create secret generic audiomanager-users \
|
||
--from-file=users.yaml=config/users.yaml
|
||
# Only needed for Deezer. Leave it out to use YouTube Music only.
|
||
kubectl create secret generic audiomanager-deezer \
|
||
--from-literal=arl='<deezer-arl>'
|
||
helm upgrade --install audiomanager charts/audiomanager \
|
||
--namespace audiomanager --create-namespace \
|
||
--set users.existingSecret=audiomanager-users \
|
||
--set deezer.existingSecret=audiomanager-deezer \
|
||
--set deezer.quality=0 \
|
||
--set defaultSource=deezer \
|
||
--set image.tag=<sha>
|
||
```
|
||
|
||
`scripts/release.sh --deploy` builds/pushes the image and runs the same Helm
|
||
upgrade. CI builds `:latest` and `:sha-<commit>` on every push to `main`, and
|
||
publishes the packaged chart.
|
||
|
||
### Setting the Deezer ARL in a values file
|
||
|
||
`deezer.arl` takes the cookie directly, so no Secret is needed:
|
||
|
||
```yaml
|
||
# values.local.yaml (gitignored)
|
||
image:
|
||
tag: sha-<commit>
|
||
users:
|
||
existingSecret: audiomanager-users
|
||
defaultSource: deezer
|
||
deezer:
|
||
arl: "<arl>"
|
||
quality: 0
|
||
```
|
||
|
||
```bash
|
||
helm upgrade --install audiomanager charts/audiomanager \
|
||
--namespace audiomanager --create-namespace \
|
||
-f values.local.yaml
|
||
```
|
||
|
||
Set only one of `deezer.arl` and `deezer.existingSecret`; the chart fails
|
||
otherwise. A plain value is stored in clear text in your values file, in
|
||
`helm get values` and in the release Secret, so use `deezer.existingSecret` on a
|
||
shared cluster or an untracked `values.local.yaml` here.
|
||
|
||
Before deploying the mood-first version over an old instance, wipe the old
|
||
database and media: V8 drops the content, but the old mounted files are no
|
||
longer referenced and should be removed by the operator. The old chart used the
|
||
same PVC name with a smaller size, and PVCs cannot shrink or be repopulated by
|
||
an upgrade, so recreate it:
|
||
|
||
```bash
|
||
kubectl -n audiomanager scale deployment/audiomanager --replicas=0
|
||
kubectl -n audiomanager delete pvc audiomanager-data
|
||
helm upgrade --install audiomanager charts/audiomanager \
|
||
--namespace audiomanager --reset-values \
|
||
--set users.existingSecret=audiomanager-users --set image.tag=<sha>
|
||
```
|
||
|
||
Old hostPath music/longform files can be deleted manually. There is no import.
|
||
|
||
## Client
|
||
|
||
Compose Multiplatform client: `/home/taptap/AudioManagerClient`.
|
||
|
||
- Android, desktop and Wasm from `commonMain`.
|
||
- Screens: Moods, Mood Detail, Search, Downloads, Settings.
|
||
- Every mood has a **Download source** selector (Deezer or YouTube Music);
|
||
search, recommendations and downloads follow it.
|
||
- Android background playback and media notifications are kept.
|
||
|
||
## Testing
|
||
|
||
```bash
|
||
./gradlew test # server unit/integration tests
|
||
./scripts/smoke.sh # end-to-end with a fake yt-dlp
|
||
scripts/check-chart.sh # lint + render the Helm chart
|
||
```
|
||
|
||
The smoke test seeds a tiny media store, puts deterministic fake `yt-dlp` and
|
||
`rip` binaries plus a fake Deezer API on localhost, and exercises both sources:
|
||
auth, search filtering, adding songs, URL resolve, downloads, recommendations,
|
||
radio sessions, streaming, covers and logout.
|