Compare commits

...
27 Commits
Author SHA1 Message Date
TapTap 44b75c4d4d feat: accept, store and expose client diagnostics
CI / chart (push) Successful in 3s
CI / test (push) Successful in 2m20s
CI / chart-push (push) Successful in 6s
CI / image (push) Successful in 3m39s
- V11 client_logs table with per-user storage and retention/pruning
- POST/GET/DELETE /logs with batch validation, rate limit and admin access
- redact bearer tokens, Authorization headers and query/JSON secrets twice
- add X-Request-Id to responses and include it in the server log pattern
- report client error entries immediately in the server log
- scripts/logs.sh for source-side retrieval and clearing
- chart knobs CLIENT_LOG_RETENTION_DAYS and CLIENT_LOG_MAX_ROWS_PER_USER
2026-11-20 13:43:05 +00:00
TapTap 91312faa1c feat: per-track loudness normalization and server audio info
CI / chart (push) Successful in 3s
CI / test (push) Successful in 2m6s
CI / chart-push (push) Successful in 3s
CI / image (push) Successful in 3m31s
- measure downloads with ffmpeg's EBU R128 filter and store gain/LUFS
- backfill the existing library one file at a time
- expose gain on songs and a /info endpoint for source quality
- chart knobs LOUDNESS_ENABLED and LOUDNESS_TARGET_LUFS
2026-11-19 16:38:21 +00:00
TapTap 6763a83b03 feat: report download status in the mood detail
CI / chart (push) Successful in 2s
CI / test (push) Successful in 3m32s
CI / chart-push (push) Successful in 10s
CI / image (push) Successful in 3m59s
The client can show a per-track Downloading/Added/Failed state in search
instead of relying on a snackbar under the keyboard.
2026-11-17 16:43:20 +00:00
TapTap de1e95bfc6 feat(chart): set the Deezer ARL from values
CI / chart (push) Successful in 2s
CI / test (push) Successful in 1m59s
CI / chart-push (push) Successful in 5s
CI / image (push) Successful in 3m35s
- new deezer.arl value rendered as DEEZER_ARL
- existingSecret and arl are mutually exclusive
- values.local.yaml is gitignored; README documents both options
- chart 0.4.1 and a chart check for the new value
2026-11-17 16:10:56 +00:00
TapTap 26c41cc7d9 feat: recommend from Deezer and YouTube Music radios
CI / chart (push) Successful in 2s
CI / test (push) Successful in 1m57s
CI / chart-push (push) Successful in 5s
CI / image (push) Successful in 3m39s
- Deezer: /artist/{id}/radio for collection artists, affinity 2.0
- YouTube Music: RDAMVM watch playlist via yt-dlp, affinity 2.0
- seed radios from the 2 newest songs of the mood's own catalog
- dedupe candidates by recording, not only by external id
- clean raw YouTube titles and Topic suffixes when indexing
- tests, smoke marker for the RDAMVM path, docs
2026-11-17 15:22:05 +00:00
TapTap 62bcefad20 feat: base discovery on the whole mood collection
CI / chart (push) Successful in 3s
CI / test (push) Successful in 1m56s
CI / chart-push (push) Successful in 5s
CI / image (push) Successful in 3m34s
- seed from every artist (weighted by song count) and every album, not one track
- walk Deezer's graph: artist top tracks, related artists, album deep cuts
- score by artist frequency, collection album, catalog affinity and hint text
- penalise repeated artists so a 30-slot mood spans artists, not one
- refresh the pool when a song is added instead of waiting for refreshHours
- skip discovery for a mood with no hint, genres or songs
- extend tests and the smoke test with the artist graph
2026-11-17 10:37:47 +00:00
TapTap a78d359d0c docs: update the dual-source test count [skip ci] 2026-11-17 07:37:04 +00:00
TapTap 141698282d feat: add Deezer as the default download source
CI / chart (push) Successful in 3s
CI / test (push) Successful in 2m23s
CI / chart-push (push) Successful in 5s
CI / image (push) Successful in 3m50s
- V9 adds source/external_id to songs, moods, recommendations and jobs
- Deezer public catalog for search, related album covers and metadata
- streamrip downloader configured with DEEZER_ARL, 128/320 MP3
- per-mood source selector; switching clears and refills recommendations
- keep YouTube Music as a full second catalog and downloader
- media layout <source>-<externalId>, legacy YouTube folders still read
- Docker/streamrip install, chart ARL secret, docs and dual-source smoke test
2026-11-17 07:32:34 +00:00
TapTap 81a47f4cd2 docs: describe the YouTube Music search timeout [skip ci] 2026-11-16 18:40:14 +00:00
TapTap 515b9bbe1b feat: search and download from YouTube Music
CI / chart (push) Successful in 3s
CI / test (push) Successful in 2m6s
CI / chart-push (push) Successful in 4s
CI / image (push) Successful in 3m25s
- search music.youtube.com with the Songs filter and resolve each hit so
  results carry canonical artist, album and duration
- fall back to plain ytsearch when the Music search fails
- download the Music watch URL first and the regular YouTube URL second
- parse explicit artist fields and drop channel/album/playlist entries
- update the smoke stub, docs and tests
2026-11-16 18:38:53 +00:00
TapTap 61d8b58223 fix: never replay a client-excluded track in a radio batch
CI / chart (push) Successful in 4s
CI / test (push) Successful in 2m19s
CI / chart-push (push) Successful in 6s
CI / image (push) Successful in 3m24s
2026-11-15 21:53:23 +00:00
TapTap ef536d1882 docs: describe the destructive PVC upgrade [skip ci] 2026-11-15 21:50:32 +00:00
TapTap 0caeecb325 feat: mark search results that are already in the target mood
CI / chart (push) Successful in 4s
CI / test (push) Successful in 2m19s
CI / chart-push (push) Successful in 5s
CI / image (push) Failing after 6m4s
2026-11-15 21:46:57 +00:00
TapTap 0613b8aedc test: cover recommendation conversion when a song is added
CI / chart (push) Successful in 3s
CI / test (push) Successful in 2m29s
CI / chart-push (push) Successful in 7s
CI / image (push) Successful in 3m40s
2026-11-15 21:45:47 +00:00
TapTap de038deb58 fix: top up recommendation slots after adding an existing song
CI / chart (push) Successful in 3s
CI / test (push) Successful in 2m9s
CI / chart-push (push) Successful in 6s
CI / image (push) Failing after 6m8s
2026-11-15 21:45:12 +00:00
TapTap 824eb36dd2 feat: rebuild AudioManager around downloadable moods
CI / chart (push) Successful in 3s
CI / test (push) Successful in 2m2s
CI / chart-push (push) Successful in 4s
CI / image (push) Successful in 3m20s
- V8 wipe migration and the shared song store keyed by YouTube video id
- media store, indexer, downloader, recommendation top-up and refcounted GC
- mood songs plus the recommendation lifecycle and mixed radio pools
- new search, song and recommendation API; remove library/longform/seed routes
- chart PVC/media env, docs, Nix, README and an end-to-end smoke test
2026-11-15 21:37:39 +00:00
TapTap 50d8b6c580 docs: capture the mood-first redesign decisions and plan [skip ci] 2026-11-15 21:05:59 +00:00
TapTap 4be502bb4e fix: harden scans, downloads, sync and radio edge cases
CI / chart (push) Successful in 3s
CI / test (push) Successful in 2m2s
CI / chart-push (push) Successful in 4s
CI / image (push) Successful in 3m13s
Scanner and library:
- Keep the last library when a configured folder cannot be walked; the
  previous code deleted every song, cascading away mood seeds and progress.
- Re-adding a file now clears its tombstone, and a re-deletion refreshes the
  timestamp instead of keeping the old one.
- Tombstone timestamps are unique per row, and delta pages continue from the
  older of the song/tombstone page ends.
- Snapshot pagination accepts a stable revision for later pages so songs
  inserted mid-sync are not skipped forever.
- Reject "%" and "_" searches that normalize to an empty LIKE needle.

Downloads and discovery:
- Download promotion uses the deterministic path id, one folder per track so
  covers do not overwrite each other, removes failed promotions, and restarts
  the scan after a running one finishes.
- Stale downloading jobs are recovered; reject cancels queued/in-flight jobs;
  markDone only owns downloading rows; rejecting no longer seeds the track.
- Reuse an already completed download of the same video for another mood.
- yt-dlp stdout and stderr are separated, empty discovery results no longer
  prune undecided recommendations, and filenames are capped by UTF-8 bytes.

API and data:
- Malformed or missing JSON returns 400 instead of 500; optional session
  bodies use receiveNullable.
- Playback events must belong to the session, known song and open session.
- Radio refills the candidate pool after recent-song pruning.
- Seed percent is required in MoodRepository.update.
- Normalize legacy V4 liked/downloaded recommendation states before V5, with
  an upgrade regression test.

CI: check out without persisting credentials and require tests before chart
publishing.
2026-11-15 18:18:21 +00:00
TapTap c75f2971b4 feat: mix mood radio by seed percentage and embed download artwork
CI / chart (push) Successful in 3s
CI / chart-push (push) Successful in 5s
CI / test (push) Successful in 1m50s
CI / image (push) Successful in 3m16s
Moods now store a seedPercent (0-100, default 50). RadioService rolls that
share independently for every queued slot: heads picks from the seed songs,
tails from the generated recommendation pool. When one pool is empty the other
fills the batch, so a mood never stalls. The DTO exposes both seedPercent and
the derived recommendationPercent, and create/patch accept seedPercent.

Automatic YT downloads now embed the video thumbnail as MP3 cover art so the
scanner publishes a cover for newly imported tracks.

V7 adds moods.seed_percent.
2026-11-15 17:07:04 +00:00
TapTap dd7bbde84d chart: make the prepared import directory owner-writable
CI / chart (push) Successful in 2s
CI / chart-push (push) Successful in 4s
CI / test (push) Successful in 1m47s
CI / image (push) Successful in 3m35s
chown alone keeps the existing mode, so a directory that was not owner-writable
still failed. Apply u+rwX,g+rwX as well.
2026-11-15 14:40:50 +00:00
TapTap e7704c42ea fix(download): explain filesystem failures and prepare the import dir
CI / chart (push) Successful in 3s
CI / chart-push (push) Successful in 4s
CI / test (push) Successful in 1m47s
CI / image (push) Successful in 3m45s
A failing Files.createDirectories only produced the directory path in the log,
which hid whether the music hostPath was read-only or owned by another user.
Errors now include the exception type, cause and a hint about UID 10001, and
permission or read-only failures fail immediately instead of retrying three
times.

The chart gets a prepare-imports init container (downloads.prepare) that
creates "<music>/AudioManager Imports" and gives it to UID 10001, so downloads
work on a hostPath owned by root. Chart version 0.2.4.
2026-11-15 14:40:28 +00:00
TapTap a5af1ee83f feat: download accepted recommendations and track longform progress
CI / chart (push) Successful in 4s
CI / chart-push (push) Successful in 4s
CI / test (push) Successful in 2m5s
CI / image (push) Successful in 3m14s
Accepted recommendations without a library match now enter a download queue.
A worker downloads the MP3 with yt-dlp into "<music>/AudioManager Imports",
runs a scan, matches the new song, links it to the recommendation and adds it
as a seed. Jobs retry three times and expose queued/downloading/done/failed
state (plus the error) on the recommendation DTO, so clients can show progress
and retry. Downloads can be disabled and the target folder chosen.

Long-form gets resume points: GET/PUT /longform/progress, scoped to longform
folders and to the owner, with automatic completion detection. Longform stays
excluded from moods and radio.

The chart mounts the music hostPath writable for imports and passes the new
DOWNLOAD_* settings; version 0.2.3.
2026-11-15 13:26:22 +00:00
TapTap 51d1b20bf6 fix(image): use the standalone yt-dlp_linux asset
CI / chart (push) Successful in 3s
CI / chart-push (push) Successful in 4s
CI / test (push) Successful in 1m46s
CI / image (push) Successful in 3m12s
The plain yt-dlp release asset is a Python zipapp and needs python3 in the
runtime image. yt-dlp_linux is the self-contained PyInstaller binary, so the
image stays minimal and yt-dlp --version verifies it at build time.
2026-11-15 12:50:24 +00:00
TapTap c5aa000d1a feat: discover YT recommendations and tune the radio
CI / chart (push) Successful in 26s
CI / chart-push (push) Successful in 5s
CI / test (push) Successful in 2m6s
CI / image (push) Failing after 2m54s
yt-dlp now feeds mood discovery. Queries come from the free-text hint, seed
artists and genres; they run in parallel, are filtered to music-length videos,
matched against library songs by normalized title and artist, scored, and
stored as review candidates. Refresh, accept and reject endpoints are added:
accepting a library match adds a seed immediately, while accepted tracks that
are not owned wait for the future downloader. Decisions survive refreshes and
stale undecided candidates are pruned.

A background pass refreshes stale moods every hour, and the image ships a
pinned yt-dlp binary (YTDLP_PATH, also added to the Nix shell and chart).

Radio now avoids tracks played in the last 7 days when the pool is large
enough, spaces artists and albums across a batch, and weighs skips three times
as heavily as completions. The chart moves to 0.2.2.
2026-11-15 12:44:30 +00:00
TapTap 67eed8d262 chart: link the README release to the public source repository
CI / chart (push) Successful in 2s
CI / chart-push (push) Successful in 4s
CI / test (push) Successful in 1m53s
CI / image (push) Successful in 3m36s
GITHUB_SERVER_URL in Actions points at the internal gitea-service host, so the
generated commit link was not reachable. Derive the public host from the
published chart URL (override with SOURCE_SERVER_URL).

Also refresh the chart description: the server is no longer Subsonic.
2026-11-14 22:01:30 +00:00
TapTap f1dc60aae5 chart: show the newest chart in the Charts repository README
CI / chart (push) Successful in 2s
CI / chart-push (push) Successful in 4s
CI / test (push) Successful in 1m39s
CI / image (push) Successful in 3m31s
The chart repository README was static, so the published version was only
visible in index.yaml. push-chart.sh now regenerates a marked "Newest chart"
section with the version, app version, publish time and source commit, plus a
table of the newest version of every chart in the repository. Plain markdown
only, so the README stays readable on the Gitea front page.

The section is inserted before the first section on the first publish and
replaced in place afterwards.
2026-11-14 21:59:51 +00:00
TapTap 8e87fed93f chart: refuse to render without an explicit namespace
Helm falls back to the "default" namespace when --namespace is omitted, so a
forgotten flag (or a Kustomize helmCharts entry without namespace) silently
deployed into the wrong place. Render the namespace through one helper that
fails on an empty or implicit "default" namespace, with allowDefaultNamespace
as an explicit opt-in.

scripts/check-chart.sh lints the chart, asserts every namespaced resource
carries the passed namespace, and checks both failure and opt-in paths. CI runs
it instead of the plain template call.

Bump the chart to 0.2.1.
2026-11-14 21:59:48 +00:00
87 changed files with 10090 additions and 3566 deletions

No files matched your search

+8 -3
View File
@@ -23,6 +23,8 @@ jobs:
image: gitea.tap-tap.win/taptap/audiomanager-ci:2
steps:
- uses: actions/checkout@v4
with:
persist-credentials: false
- name: Test
run: gradle test --console=plain
- name: Smoke test
@@ -34,13 +36,14 @@ jobs:
image: gitea.tap-tap.win/taptap/audiomanager-ci:2
steps:
- uses: actions/checkout@v4
with:
persist-credentials: false
- name: Lint and render the chart
run: |
helm lint charts/audiomanager
helm template audiomanager charts/audiomanager --set users.existingSecret=test > /dev/null
run: scripts/check-chart.sh
chart-push:
needs:
- test
- chart
if: github.event_name == 'push'
runs-on: ubuntu-latest
@@ -52,6 +55,8 @@ jobs:
CHARTS_TOKEN: ${{ secrets.AUDIOMANAGER_CHARTS_TOKEN }}
steps:
- uses: actions/checkout@v4
with:
persist-credentials: false
- name: Package and publish the chart
run: scripts/push-chart.sh
+6 -2
View File
@@ -9,8 +9,8 @@ build/
.vscode/
# Local runtime data
data/
media/
/data/
/media/
config/library.yaml
config/users.yaml
*.db
@@ -23,4 +23,8 @@ test-media/
# Packaged Helm charts
dist/
# Local Helm overrides (may contain the Deezer ARL)
values.local.yaml
charts/*/values.local.yaml
.kotlin/
+21 -2
View File
@@ -12,9 +12,23 @@ RUN gradle --no-daemon shadowJar
# glibc (not Alpine/musl): sqlite-jdbc ships no native library for Linux-Musl.
FROM eclipse-temurin:21-jre-jammy
ARG YTDLP_VERSION=2026.10.26
ARG STREAMRIP_VERSION=2.1.0
RUN apt-get update \
&& apt-get install -y --no-install-recommends ffmpeg \
&& apt-get install -y --no-install-recommends \
ca-certificates \
curl \
ffmpeg \
python3 \
python3-pip \
&& rm -rf /var/lib/apt/lists/* \
&& curl -fsSL "https://github.com/yt-dlp/yt-dlp/releases/download/${YTDLP_VERSION}/yt-dlp_linux" \
-o /usr/local/bin/yt-dlp \
&& chmod 0755 /usr/local/bin/yt-dlp \
&& pip3 install --no-cache-dir "streamrip==${STREAMRIP_VERSION}" \
&& yt-dlp --version \
&& rip --version \
&& groupadd -g 10001 app \
&& useradd -u 10001 -g app -M -s /usr/sbin/nologin app
@@ -22,9 +36,14 @@ WORKDIR /app
COPY --from=build /src/build/libs/*-all.jar /app/musicmanager.jar
ENV DATA_DIR=/data \
LIBRARY_CONFIG=/etc/musicmanager/library.yaml \
MEDIA_DIR=/data/media \
DOWNLOAD_DIR=/data/downloads \
USERS_FILE=/etc/musicmanager/users.yaml \
FFMPEG_PATH=ffmpeg \
YTDLP_PATH=yt-dlp \
STREAMRIP_PATH=rip \
DEFAULT_SOURCE=deezer \
DEEZER_QUALITY=0 \
PORT=8080
USER 10001:10001
+310 -240
View File
@@ -1,318 +1,388 @@
# AudioManager
A self-hosted, mood-first music server in Kotlin/Ktor, plus a Kotlin Multiplatform client.
A self-hosted, mood-first music server. You create private moods, add tracks
from Deezer or YouTube Music, and the server downloads them into a shared media
store. Recommendations are discovered and downloaded automatically so a mood
keeps playing without a mounted music library.
AudioManager replaces static playlists with **moods**: curated anchors plus a radio queue that is generated live by the server and learns from what you finish and skip. There is no like button and no OpenSubsonic layer — the native API and the native client are designed together around this model.
- Server: Kotlin, Ktor, SQLite (Exposed + Flyway), `yt-dlp`, `streamrip`,
`ffmpeg`.
- Client: Compose Multiplatform (Android, desktop, Wasm).
- No uploads, no mounted library, no likes, no podcasts.
## Contents
1. [Quick start](#quick-start)
2. [Configuration](#configuration)
3. [The mood model](#the-mood-model)
4. [Native API](#native-api)
5. [Architecture](#architecture)
6. [Database](#database)
7. [Scanner](#scanner)
8. [Deployment](#deployment)
9. [Client](#client)
10. [Testing](#testing)
11. [Roadmap](#roadmap)
- [How it works](#how-it-works)
- [Quick start](#quick-start)
- [Configuration](#configuration)
- [Native API](#native-api)
- [Architecture](#architecture)
- [Media store and GC](#media-store-and-gc)
- [Database](#database)
- [Deployment](#deployment)
- [Client](#client)
- [Testing](#testing)
---
## How it works
- **Moods are private.** A user sees only their own moods. Songs are shared
server-wide and deduplicated by `(source, externalId)`.
- **Two catalogs.** Every mood has a download source: **Deezer** (default) or
**YouTube Music**. Search, recommendations and downloads follow that source;
permanent songs from either source can live in the same mood.
- **Songs are the collection.** A mood needs at least one song before it can
play. Removing a song only detaches it; the file is deleted when no mood,
recommendation slot or open playback session references it.
- **Recommendations are active slots.** A mood keeps `recommendationCount`
(0–30, default 10) recommendations in `downloading`, `ready` or `failed`
state. Hidden `new` candidates refill the slots automatically.
- **Discovery uses the whole mood.** Every artist (weighted by how many songs
the mood has), every album, the hint and the genres feed one pass. On Deezer
the server also asks Deezer for the artists' **radio** — a Daft Punk mood gets
MGMT, Caribou and Jamiroquai — plus related artists and album deep cuts. On
YouTube Music it follows the song's `RDAMVM` **radio queue**, the same mix the
player builds. Adding a song makes the pool refresh instead of waiting for
`refreshHours`.
- **Add / Not interested are permanent.** Adding promotes a recommendation into
the collection. Rejecting is remembered per mood. Both free a slot and the
server refills it.
- **Playback mixes both pools.** Every queue slot rolls `collectionPercent`;
the rest comes from ready recommendations. An empty pool falls back to the
other one. Small pools replay least-recently-played tracks.
- **Downloads are automatic.** Recommendation downloads run in the background
up to the configured count even if the mood is never played. Manual adds have
higher priority. Deezer needs an ARL cookie; YouTube Music needs no account.
## Quick start
Requirements: JDK 21, ffmpeg, SQLite (bundled through JDBC). With Nix, `nix-shell` provides the JDK, Gradle, ffmpeg, sqlite, Python, Helm, Kustomize and kubectl.
Requirements: JDK 21, `ffmpeg`, `yt-dlp`, and `streamrip` (`pip install
streamrip==2.1.0`, which provides `rip`) for Deezer. The smoke test also needs
`sqlite3`, `curl` and `python3`. `nix-shell` provides everything except
`streamrip`.
```bash
# 1. Build
gradle shadowJar
# 1. Build and create local config
./gradlew setupLocal
# edit config/users.yaml (a password is required)
# 2. Point the server at a library
mkdir -p data
cat > config/library.yaml <<'YAML'
library:
folders:
- id: 1
name: Music
type: music
path: /absolute/path/to/music
- id: 10
name: Long-form
type: longform
path: /absolute/path/to/longform
YAML
# 3. Define users (passwords stay outside SQLite)
cat > config/users.yaml <<'YAML'
users:
- name: alice
password: change-me
admin: true
folders: [1]
YAML
# 4. Run
DATA_DIR=./data LIBRARY_CONFIG=./config/library.yaml USERS_FILE=./config/users.yaml \
java -jar build/libs/musicmanager-*-all.jar
# 2. Run
./gradlew run
# health: http://localhost:8080/health
# API: http://localhost:8080/api/v1
```
The server exposes the native API under `/api/v1`; use the Kotlin Multiplatform client for browsing and playback. The first start scans the library automatically because the database is empty. Later scans are manual:
The server creates `./data/media` (the media store), `./data/downloads`
(download temp files) and `./data/musicmanager.db`.
```bash
curl -X POST -H "Authorization: Bearer $TOKEN" http://localhost:8080/api/v1/scan
```
Deezer downloads need `DEEZER_ARL`; YouTube Music downloads need `yt-dlp` to be
able to reach YouTube. If YouTube applies bot checks, search and downloads fail
with `502 search_failed` / `failed` recommendations; retry later or update
`yt-dlp`. A Deezer Free ARL is limited to 128 kbps; 320 kbps needs Deezer
Premium (`DEEZER_QUALITY=1`).
## Configuration
### `library.yaml`
Everything is environment variables:
```yaml
library:
folders:
- id: 1
name: Music
type: music
path: /mnt/media/music
- id: 10
name: Long-form
type: longform
path: /mnt/media/longform
```
- `type` is either `music` or `longform`. They are never mixed in a mood.
- IDs are stable and referenced by songs; never reuse an ID for a different path.
- Long-form folders are shared with every user automatically.
| Variable | Default | Meaning |
|---|---|---|
| `PORT` | `8080` | HTTP port |
| `DATA_DIR` | `./data` | Database, cover cache, default media root |
| `MEDIA_DIR` | `<DATA_DIR>/media` | Server-owned song store |
| `DOWNLOAD_DIR` | `<DATA_DIR>/downloads` | yt-dlp temp/work directory |
| `USERS_FILE` | `./config/users.yaml` | User accounts |
| `FFMPEG_PATH` | `ffmpeg` | Transcoder binary |
| `YTDLP_PATH` | `yt-dlp` | YouTube Music downloader / search binary |
| `STREAMRIP_PATH` | `rip` | Deezer downloader (`streamrip`) |
| `DEFAULT_SOURCE` | `deezer` | Source for new moods: `deezer` or `youtube` |
| `DEEZER_ARL` | empty | Deezer account cookie used for downloads (chart: `deezer.arl` or `deezer.existingSecret`) |
| `DEEZER_QUALITY` | `0` | `0` = 128 kbps MP3, `1` = 320 kbps MP3 |
| `DEEZER_API_BASE` | `https://api.deezer.com` | Public Deezer API base |
| `LOUDNESS_ENABLED` | `true` | Measure each file and expose a per-track gain |
| `LOUDNESS_TARGET_LUFS` | `-16` | Integrated loudness target (`-30` to `-5`) |
| `CLIENT_LOG_RETENTION_DAYS` | `14` | Days uploaded client diagnostics are kept |
| `CLIENT_LOG_MAX_ROWS_PER_USER` | `20000` | Newest diagnostic rows kept per user |
### `users.yaml`
```yaml
users:
- name: alice
password: change-me
- name: admin
password: "change-me"
admin: true
folders: [1]
- name: bob
password: another-secret
admin: false
folders: [1]
```
Passwords are read into memory at startup and never written to SQLite. In production this file is a Vault-managed Secret. Restart the deployment after changing users.
Passwords live only in this file (use a Vault-managed Secret in Kubernetes).
`admin` is only needed for the maintenance scan endpoints. Old `folders:` keys
are accepted and ignored.
### Environment
## Client diagnostics
| Variable | Default | Meaning |
| --- | --- | --- |
| `PORT` | `8080` | HTTP port |
| `DATA_DIR` | `./data` | SQLite database and cover cache |
| `LIBRARY_CONFIG` | `./config/library.yaml` | Library folders |
| `USERS_FILE` | `./config/users.yaml` | Users and passwords |
| `FFMPEG_PATH` | `ffmpeg` | Transcoder binary |
The apps keep a small local ring buffer (500 entries) of warnings, errors and
crashes. Warnings and errors are flushed immediately, so the line that explains
a crash survives it; info entries are batched. When signed in, the client uploads
up to 100 entries per minute to `POST /api/v1/logs`, tagged with a random install
id, platform, app version and per-run session id.
## The mood model
Privacy and safety:
A mood is a private collection owned by one user:
- `Authorization` headers, `Bearer` tokens, `token=`/`password=`/`arl=` query
parameters and JSON secret fields are replaced with `***` on the client
**and** again on the server.
- The server stores rows per user; a user can only read their own logs. An
administrator can pass `user=<name>`.
- Retention defaults to 14 days and 20,000 rows per user; both are configurable.
- A rejected batch (HTTP 400/413/422) is dropped so it cannot block later logs.
- **Seeds** are permanent anchors. Every seed is always eligible for the radio.
- **Genres / free-text hint** widen the candidate pool.
- **Target size** is the desired batch and session size, not a hard playlist length.
- **Radio** is generated by the server for each playback session. It mixes seeds, related genres and artists, text matches, and the rest of the owner's music.
- **Signals** come from playback: completed tracks gain weight, skipped tracks lose weight. There are no likes.
- **Add to mood** is explicit. The client offers *add to current mood* and *choose another mood*, and the server turns the track into a manual seed.
- **Long-form is excluded** at seed validation and radio generation.
Inspect them with the helper:
A playback session belongs to one mood and one user. The server remembers queued tracks so a batch never repeats within the session, and a client can resume after a restart.
```bash
scripts/logs.sh --server https://audiomanager.example.com --user alice \
--level warn --limit 100
# administrators can read another account
scripts/logs.sh --server ... --user admin --target alice
# clear after investigating
scripts/logs.sh --server ... --user alice --clear
```
The password is read from `LOGS_PASSWORD` or prompted; it is never stored.
## Native API
All endpoints live under `/api/v1`. JSON in, JSON out, standard HTTP status codes. Errors use:
```json
{ "error": { "code": "not_found", "message": "Mood not found" } }
```
All endpoints are under `/api/v1` and require a bearer token, except login.
Media URLs may use `?token=` because HTML audio and images cannot send headers.
### Authentication
```http
POST /api/v1/auth/login
Content-Type: application/json
| Method | Path | Purpose |
|---|---|---|
| `POST` | `/auth/login` | `{username,password,deviceName,platform}` → token |
| `POST` | `/auth/logout` | Revoke the current token |
| `GET` | `/me` | Current user |
| `GET` | `/info` | Default source, Deezer quality, loudness and log retention |
| `GET` | `/auth/devices` | List device tokens |
| `DELETE` | `/auth/devices/{id}` | Revoke a device |
{ "username": "alice", "password": "change-me", "deviceName": "Pixel", "platform": "android" }
```
### Diagnostics
```json
{
"token": "…",
"expiresAt": "2027-05-13T…",
"user": { "name": "alice", "admin": true, "musicFolders": [1], "longformFolders": [10] }
}
```
Send `Authorization: Bearer <token>` on every request. Tokens are stored as SHA-256 hashes and expire after 180 days. Media URLs may also use `?token=` because HTML audio and `<img>` tags cannot set headers.
- `POST /auth/logout` revokes the current token.
- `GET /auth/devices` lists devices; `DELETE /auth/devices/{id}` revokes one.
- `GET /me` returns the user and folder IDs.
### Library sync
- `GET /library/songs?cursor=<id>&limit=500&scope=music|longform|all` — snapshot page, ordered by song ID. Loop while `nextCursor` is present.
- `GET /library/songs?since=<revision>` — delta with `songs` and `deletedSongIds`. Loop while `nextSince` is present.
- `GET /library/search?q=&limit=&scope=` — normalized substring search.
- `GET /songs/{id}/stream` — original MP3 with Range support. Optional `maxBitRate`, `timeOffset`, `format`, `download=true`.
- `GET /covers/{albumIdOrSongId}?size=320` — cached JPEG cover.
Song objects contain deterministic IDs (`sg-…`), album/artist names, duration, size, bitrate, genres, `folderId`, `relativePath`, and `updatedAt`, so a client can build an offline library without extra requests.
| Method | Path | Purpose |
|---|---|---|
| `POST` | `/logs` | Upload a client diagnostics batch |
| `GET` | `/logs` | List diagnostics (`level`, `device`, `since`, `before`, `limit`; admins may add `user`) |
| `DELETE` | `/logs` | Delete the current user's (or, for admins, a named user's) diagnostics |
### Moods
- `GET /moods`
- `POST /moods` — `{ name, description?, query?, genres[], targetSize, refreshHours, ytEnabled, seedSongIds[] }`
- `GET /moods/{id}` — mood, seeds and recommendations.
- `PATCH /moods/{id}` — partial settings update; `clearDescription` / `clearQuery` clear fields.
- `DELETE /moods/{id}`
- `POST /moods/{id}/seeds` — `{ songId }`
- `DELETE /moods/{id}/seeds/{songId}`
- `GET /moods/{id}/preview?limit=20` — unpersisted radio preview.
- `GET /moods/{id}/recommendations` — YT Music candidates (see roadmap).
| Method | Path | Purpose |
|---|---|---|
| `GET` | `/moods` | List the user's moods |
| `POST` | `/moods` | Create `{name,description,query,genres,source,collectionPercent,recommendationCount,refreshHours}` |
| `GET` | `/moods/{id}` | Mood detail: songs, active recommendations, download status |
| `PATCH` | `/moods/{id}` | Update options |
| `DELETE` | `/moods/{id}` | Delete mood, detach songs and run GC |
| `GET` | `/moods/{id}/preview` | Unpersisted radio preview |
| `POST` | `/moods/{id}/songs` | Add `{songId}`, `{source,externalId}` or `{url}` |
| `DELETE` | `/moods/{id}/songs/{songId}` | Detach a song |
| `GET` | `/moods/{id}/recommendations` | Active recommendation slots |
| `POST` | `/moods/{id}/recommendations/refresh` | Force discovery + top-up |
| `POST` | `/moods/{id}/recommendations/{externalId}/add` | Add to the collection |
| `POST` | `/moods/{id}/recommendations/{externalId}/reject` | Hide permanently |
| `POST` | `/moods/{id}/recommendations/{externalId}/retry` | Retry a failed download |
### Radio sessions
Moods accept `source` on create/update; switching it clears the active
recommendation pool and refills it from the new catalog.
- `POST /moods/{id}/sessions` — `{ limit?: 15 }` → session plus the first queue.
- `GET /sessions/{id}` — resume: session and queued tracks.
- `POST /sessions/{id}/next` — `{ limit?, excludeSongIds[] }` → the next batch.
- `POST /sessions/{id}/events` — `{ events: [{ songId, type: "started"|"completed"|"skipped", positionSeconds? }] }`
- `POST /sessions/{id}/add-to-mood` — `{ songId, moodId? }`. Without `moodId`, the session's mood is used. This is the replacement for likes.
- `DELETE /sessions/{id}` — close.
Adding an already downloaded song attaches instantly (`200`). A new track is
queued for download (`202`, `downloading: true`) and is attached when the job
finishes. `/moods/{id}/songs` accepts `{"songId":...}`, `{"source":"deezer",
"externalId":"..."}`, or `{"url":"..."}` (Deezer or YouTube).
### System
### Search
- `GET /scan` — `{ scanning, count, error }`
- `POST /scan` — admin only; 202 Accepted.
- `GET /health` — unauthenticated liveness probe.
| Method | Path | Purpose |
|---|---|---|
| `GET` | `/search?q=...&limit=...&source=...&moodId=...` | Deezer or YouTube Music search |
Results longer than 10 minutes (or shorter than 30 seconds) are hidden. Each
result reports `source`, `externalId`, `downloaded`, `songId` and `inMood`.
`source` defaults to the target mood's source, then to `DEFAULT_SOURCE`.
Deezer search uses the public API and returns canonical tracks with a square
album cover. YouTube Music search uses the "Songs" filter and resolves every
hit, which takes roughly 10–15 seconds for ten results; if Music search fails,
it falls back to plain YouTube search.
### Playback
| Method | Path | Purpose |
|---|---|---|
| `POST` | `/moods/{id}/sessions` | Start a queue (`400 empty_mood` without songs) |
| `GET` | `/sessions/{id}` | Resume a session |
| `POST` | `/sessions/{id}/next` | Fetch more tracks |
| `POST` | `/sessions/{id}/events` | Report `started`, `completed`, `skipped` |
| `POST` | `/sessions/{id}/add-to-mood` | Add the playing track to a mood |
| `DELETE` | `/sessions/{id}` | Close the session |
### Media and maintenance
| Method | Path | Purpose |
|---|---|---|
| `GET` | `/songs/{id}/stream` | Range-aware stream; `maxBitRate`, `timeOffset`, `download` |
| `GET` | `/songs/{id}/cover?size=` | Embedded artwork |
| `GET` | `/scan` | Scan status |
| `POST` | `/scan` | Start a scan (admin only) |
## Architecture
```text
src/main/kotlin/win/taptap/musicmanager/
|-- Application.kt # Ktor entry point, plugins, graceful shutdown
|-- config/
| |-- AppConfig.kt # Port, data dir, ffmpeg, env overrides
| |-- LibraryConfig.kt # Folder IDs, types and paths
| `-- UsersConfig.kt # Users, admins and folder assignments
|-- api/
| |-- ApiRoutes.kt # /api/v1 wiring, auth and system endpoints
| |-- ApiModels.kt # Native DTOs
| |-- ApiSupport.kt # Token resolution, errors, folder scoping
| |-- LibraryApi.kt # Sync, search, stream, covers
| |-- MoodApi.kt # Mood CRUD, seeds, preview, recommendations
| `-- SessionApi.kt # Radio sessions, events, add-to-mood
|-- radio/
| `-- RadioService.kt # Candidate pool, scoring, diversity, batches
|-- db/
| |-- DatabaseFactory.kt # SQLite WAL + Flyway
| |-- Tables.kt # Exposed definitions
| |-- ApiTokenRepository.kt # Hashed bearer tokens
| |-- RadioRepository.kt # Sessions, events, affinity
| |-- MoodRepository.kt # Moods, seeds, recommendations
| |-- LibraryRepository.kt # Scoped browsing, sync, search
| `-- MediaRepository.kt # Scanner persistence and tombstones
|-- scanner/ # Recursive MP3 scan, tags, covers
|-- transcoding/ # ffmpeg MP3 transcoding
`-- util/ # Deterministic IDs, sort names
```
Ktor routes
└── MoodRepository ── moods, mood_songs, recommendation lifecycle
└── SongRepository ── shared song store keyed by (source, externalId)
└── RadioService ── mixes collection + ready recommendations
└── MusicCatalogs ── Deezer public API + yt-dlp YouTube Music search
└── DownloadService── single worker: yt-dlp/streamrip → media store → index
└── DiscoveryService── mood source search → hidden `new` candidates
└── TopUpService ── keeps active slots at recommendationCount
└── MediaScanner ── rebuilds the index from the media root
└── MediaGcService ── refcounted file deletion
```
The old OpenSubsonic layer (`/rest`, `subsonic/`, playlist materialization) was removed in 0.2.0. The server and the KMP client are the two halves of the same product; there is no compatibility layer to preserve and no server-rendered UI.
Background loops: download worker (continuous), recommendation top-up (hourly),
media GC (hourly). A scan runs at startup.
## Media store and GC
Layout:
```
<MEDIA_DIR>/<source>-<externalId>/<sanitized-title>.mp3
```
Examples: `deezer-67238732/Instant Crush.mp3`, `youtube-dQw4w9WgXcQ/...`.
Older YouTube downloads in `<MEDIA_DIR>/<videoId>/` are still read and treated
as YouTube. One row and one file exist per track, no matter how many moods or
users reference it. Covers are cached under `<DATA_DIR>/covers` keyed by song
id.
Deezer is downloaded with `streamrip` (MP3 128/320) and YouTube Music with
`yt-dlp` (`-x --audio-format mp3 --audio-quality 0`). Both embed the artist,
album and cover art, so the indexer can derive the metadata from the file.
A file is deleted only when it has no references from `mood_songs`, an active
recommendation (`downloading`, `ready`, `added`, `failed`), or an open playback
session. GC runs after removals, after rejections, after mood deletion, and
hourly.
## Database
SQLite in WAL mode, migrations through Flyway:
SQLite in WAL mode. Migrations are Flyway scripts under
`src/main/resources/db/migration`.
- `V1__init.sql` — users, folders, artists, albums, songs, user state.
- `V2__user_item_ratings.sql`, `V3__song_lyrics.sql`.
- `V4__moods.sql` — the first playlist-based mood prototype.
- `V5__native_api.sql` — native moods, tokens, playback sessions, affinity, tombstones; drops playlists; adds `songs.updated_at`.
Key invariants:
- IDs are deterministic (`ar-`, `al-`, `sg-`, `mo-`) with frozen normalization (NFC, trimmed, collapsed whitespace, lowercase).
- Albums are grouped by album artist, not by track artist.
- Deleted songs leave tombstones so offline clients can remove them.
- Passwords never appear in the database.
## Scanner
- Indexes `.mp3` recursively; other extensions are skipped and logged.
- Tags first, `Artist/Album/Track` path fallback.
- Delta scan: unchanged file mtimes are skipped.
- Orphan cleanup writes song tombstones.
- The first start scans automatically when the songs table is empty.
- Manual scans come from `POST /api/v1/scan` or a client.
- **V8 is destructive.** It drops every table from the mounted-library design
(folders, artists, albums, songs, moods, seeds, sessions, longform progress,
downloads) and creates the mood-first schema. `users` and `api_tokens` are
kept.
- **V9 adds sources.** It renames `video_id` to `external_id` and adds a
`source` column to songs, moods, recommendations and download jobs. Existing
rows become `youtube`.
- There is no import from an old media directory. Back up externally and start
with an empty `MEDIA_DIR`.
## Deployment
Images and charts are published by Gitea Actions to `gitea.tap-tap.win`.
The Helm chart is in `charts/audiomanager`. It renders one PVC at `/data`:
```text
Image: gitea.tap-tap.win/taptap/audiomanager:<tag>
Chart: https://gitea.tap-tap.win/TapTap/Charts/raw/branch/main
```
/data/musicmanager.db
/data/media/<source>-<externalId>/<title>.mp3
/data/covers/
/data/downloads/
```
Kustomize example:
```yaml
helmCharts:
- name: audiomanager
repo: https://gitea.tap-tap.win/TapTap/Charts/raw/branch/main
releaseName: audiomanager
version: 0.2.0-sha.<commit>
namespace: audiomanager
valuesFile: values.yaml
```
The chart expects:
- a `users.yaml` Secret (Vault-managed) referenced by `users.existingSecret`;
- a `library.yaml` ConfigMap;
- read-only hostPaths for music and long-form;
- a registry pull secret for the private image.
Ingress host: `subsonic.tap-tap.win` (the historical name is kept, but the server no longer speaks the Subsonic protocol).
No hostPath, no `library.yaml`, no init container. `fsGroup: 10001` makes the
PVC writable by the non-root container.
```bash
kustomize build --enable-helm . | kubectl apply -f -
kubectl -n audiomanager rollout status deployment/audiomanager
kubectl create secret generic audiomanager-users \
--from-file=users.yaml=config/users.yaml
# Only needed for Deezer. Leave it out to use YouTube Music only.
kubectl create secret generic audiomanager-deezer \
--from-literal=arl='<deezer-arl>'
helm upgrade --install audiomanager charts/audiomanager \
--namespace audiomanager --create-namespace \
--set users.existingSecret=audiomanager-users \
--set deezer.existingSecret=audiomanager-deezer \
--set deezer.quality=0 \
--set defaultSource=deezer \
--set image.tag=<sha>
```
Flyway applies `V5__native_api.sql` automatically. Existing playlist-based moods are migrated to native moods; old playlists and user stars are dropped.
`scripts/release.sh --deploy` builds/pushes the image and runs the same Helm
upgrade. CI builds `:latest` and `:sha-<commit>` on every push to `main`, and
publishes the packaged chart.
### Setting the Deezer ARL in a values file
`deezer.arl` takes the cookie directly, so no Secret is needed:
```yaml
# values.local.yaml (gitignored)
image:
tag: sha-<commit>
users:
existingSecret: audiomanager-users
defaultSource: deezer
deezer:
arl: "<arl>"
quality: 0
```
```bash
helm upgrade --install audiomanager charts/audiomanager \
--namespace audiomanager --create-namespace \
-f values.local.yaml
```
Set only one of `deezer.arl` and `deezer.existingSecret`; the chart fails
otherwise. A plain value is stored in clear text in your values file, in
`helm get values` and in the release Secret, so use `deezer.existingSecret` on a
shared cluster or an untracked `values.local.yaml` here.
Before deploying the mood-first version over an old instance, wipe the old
database and media: V8 drops the content, but the old mounted files are no
longer referenced and should be removed by the operator. The old chart used the
same PVC name with a smaller size, and PVCs cannot shrink or be repopulated by
an upgrade, so recreate it:
```bash
kubectl -n audiomanager scale deployment/audiomanager --replicas=0
kubectl -n audiomanager delete pvc audiomanager-data
helm upgrade --install audiomanager charts/audiomanager \
--namespace audiomanager --reset-values \
--set users.existingSecret=audiomanager-users --set image.tag=<sha>
```
Old hostPath music/longform files can be deleted manually. There is no import.
## Client
The client lives in `TapTap/AudioManagerClient` and is built with Kotlin Multiplatform + Compose Multiplatform:
Compose Multiplatform client: `/home/taptap/AudioManagerClient`.
- **Android**: Media3/ExoPlayer, background playback, offline downloads.
- **Desktop (JVM)**: shared Compose UI and a JVM audio backend.
- **Web (Wasm)**: the same Compose UI, HTML audio, tokenized stream URLs.
The shared module owns the API client, auth store, SQLDelight offline database, mood radio queue, and platform player abstractions. Downloads are original MP3s from `/stream?download=true`. Web is streaming-only (the browser owns its cache).
- Android, desktop and Wasm from `commonMain`.
- Screens: Moods, Mood Detail, Search, Downloads, Settings.
- Every mood has a **Download source** selector (Deezer or YouTube Music);
search, recommendations and downloads follow it.
- Android background playback and media notifications are kept.
## Testing
```bash
gradle test # unit + API integration tests
GRADLE=gradle scripts/smoke.sh # end-to-end server smoke test
./gradlew test # server unit/integration tests
./scripts/smoke.sh # end-to-end with a fake yt-dlp
scripts/check-chart.sh # lint + render the Helm chart
```
The smoke test generates tagged MP3s, starts the server, and verifies login, sync, moods, radio, add-to-mood, streaming, transcoding, covers, scan and logout.
## Roadmap
- [x] MP3 scanner, covers, deterministic IDs, SQLite + Flyway.
- [x] Native bearer-auth API, sync deltas, tombstones.
- [x] Mood CRUD, seeds, server-driven radio, playback events, add-to-mood.
- [x] Kotlin Multiplatform client foundation (Android, desktop, web).
- [ ] Offline downloads and offline playback across clients.
- [ ] YT Music discovery through `yt-dlp`, recommendation list, accept/reject.
- [ ] Optional downloader that writes accepted tracks into an import folder and rescans.
- [ ] Home-screen widgets, Android Auto, and richer radio tuning.
The smoke test seeds a tiny media store, puts deterministic fake `yt-dlp` and
`rip` binaries plus a fake Deezer API on localhost, and exercises both sources:
auth, search filtering, adding songs, URL resolve, downloads, recommendations,
radio sessions, streaming, covers and logout.
+15 -32
View File
@@ -36,6 +36,7 @@ dependencies {
implementation("io.ktor:ktor-server-call-logging-jvm:3.1.2")
implementation("io.ktor:ktor-server-auto-head-response-jvm:3.1.2")
implementation("io.ktor:ktor-server-cors-jvm:3.1.2")
implementation("io.ktor:ktor-server-call-id-jvm:3.1.2")
// Database
implementation("org.jetbrains.exposed:exposed-core:0.60.0")
@@ -69,44 +70,26 @@ tasks.test {
}
/**
* Prepares a local development checkout: creates the media directories and copies
* the example configs when they are missing. Existing files are never overwritten.
* Prepares a local development checkout: creates the data/media directories and
* copies the example users file when it is missing. Existing files are never
* overwritten.
*/
val setupLocal by tasks.registering {
group = "application"
description = "Create local media folders and copy example configs if missing."
description = "Create local data folders and copy the example users config if missing."
doLast {
project.mkdir("media/music")
project.mkdir("media/longform")
project.mkdir("data/media")
project.mkdir("data/downloads")
listOf("library", "users").forEach { name ->
val target = project.file("config/$name.yaml")
val example = project.file("config/$name.example.yaml")
if (!target.exists()) {
target.parentFile.mkdirs()
example.copyTo(target)
println("Created config/$name.yaml from config/$name.example.yaml")
} else {
println("Keeping existing config/$name.yaml")
}
}
val libraryConfig = project.file("config/library.yaml")
if (libraryConfig.exists()) {
val stale = libraryConfig.readText().lineSequence()
.map { it.trim() }
.filter { it.startsWith("path:") }
.map { it.removePrefix("path:").trim() }
.filter { it.startsWith("/mnt/media/") }
.toList()
if (stale.isNotEmpty()) {
logger.warn(
"config/library.yaml still points at /mnt/media paths that are not available " +
"locally: $stale. Replace it with the local example: " +
"cp config/library.example.yaml config/library.yaml",
)
}
val target = project.file("config/users.yaml")
val example = project.file("config/users.example.yaml")
if (!target.exists()) {
target.parentFile.mkdirs()
example.copyTo(target)
println("Created config/users.yaml from config/users.example.yaml")
} else {
println("Keeping existing config/users.yaml")
}
}
}
+2 -2
View File
@@ -1,6 +1,6 @@
apiVersion: v2
name: audiomanager
description: Self-hosted OpenSubsonic music server
description: Mood-first self-hosted music server
type: application
version: 0.2.0
version: 0.5.0
appVersion: "0.1.0"
@@ -31,3 +31,22 @@ app.kubernetes.io/instance: {{ .Release.Name }}
{{- define "audiomanager.image" -}}
{{- printf "%s:%s" .Values.image.repository (default .Chart.AppVersion .Values.image.tag) -}}
{{- end -}}
{{/*
The namespace every resource is rendered into.
Helm falls back to "default" when --namespace is omitted, which would silently
put the release somewhere the operator did not ask for. Refuse to render in
that case unless the fallback is explicitly allowed, so a forgotten --namespace
(or a kustomization without namespace:) fails instead of deploying.
*/}}
{{- define "audiomanager.namespace" -}}
{{- $namespace := .Release.Namespace -}}
{{- if not $namespace -}}
{{- fail "a namespace is required: pass --namespace <namespace> (or set namespace: in the Kustomize helmCharts entry)" -}}
{{- end -}}
{{- if and (eq $namespace "default") (not .Values.allowDefaultNamespace) -}}
{{- fail "refusing to deploy into the implicit \"default\" namespace: pass --namespace <namespace>, or set allowDefaultNamespace=true to opt in" -}}
{{- end -}}
{{- $namespace -}}
{{- end -}}
@@ -1,17 +0,0 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ include "audiomanager.fullname" . }}-library
namespace: {{ .Release.Namespace }}
labels:
{{- include "audiomanager.labels" . | nindent 4 }}
data:
library.yaml: |
library:
folders:
{{- range .Values.library.folders }}
- id: {{ .id }}
name: {{ .name | quote }}
type: {{ .type }}
path: {{ .path | quote }}
{{- end }}
+43 -26
View File
@@ -1,8 +1,17 @@
{{- if not (has .Values.defaultSource (list "deezer" "youtube")) }}
{{- fail "defaultSource must be deezer or youtube" }}
{{- end }}
{{- if not (has (int .Values.deezer.quality) (list 0 1)) }}
{{- fail "deezer.quality must be 0 (128 kbps) or 1 (320 kbps)" }}
{{- end }}
{{- if and .Values.deezer.arl .Values.deezer.existingSecret }}
{{- fail "set only one of deezer.arl and deezer.existingSecret" }}
{{- end }}
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ include "audiomanager.fullname" . }}
namespace: {{ .Release.Namespace }}
namespace: {{ include "audiomanager.namespace" . }}
labels:
{{- include "audiomanager.labels" . | nindent 4 }}
spec:
@@ -16,8 +25,6 @@ spec:
metadata:
labels:
{{- include "audiomanager.selectorLabels" . | nindent 8 }}
annotations:
checksum/library: {{ include (print $.Template.BasePath "/configmap-library.yaml") . | sha256sum }}
spec:
{{- with .Values.imagePullSecrets }}
imagePullSecrets:
@@ -36,12 +43,43 @@ spec:
value: "8080"
- name: DATA_DIR
value: /data
- name: LIBRARY_CONFIG
value: /etc/audiomanager/library.yaml
- name: MEDIA_DIR
value: /data/media
- name: DOWNLOAD_DIR
value: /data/downloads
- name: USERS_FILE
value: /etc/audiomanager/users.yaml
- name: FFMPEG_PATH
value: ffmpeg
- name: YTDLP_PATH
value: yt-dlp
- name: STREAMRIP_PATH
value: rip
- name: DEFAULT_SOURCE
value: {{ .Values.defaultSource | quote }}
- name: DEEZER_QUALITY
value: {{ .Values.deezer.quality | quote }}
- name: DEEZER_API_BASE
value: {{ .Values.deezer.apiBase | quote }}
- name: LOUDNESS_ENABLED
value: {{ .Values.loudness.enabled | quote }}
- name: LOUDNESS_TARGET_LUFS
value: {{ .Values.loudness.targetLufs | quote }}
- name: CLIENT_LOG_RETENTION_DAYS
value: {{ .Values.clientLogs.retentionDays | quote }}
- name: CLIENT_LOG_MAX_ROWS_PER_USER
value: {{ .Values.clientLogs.maxRowsPerUser | quote }}
{{- if .Values.deezer.existingSecret }}
- name: DEEZER_ARL
valueFrom:
secretKeyRef:
name: {{ .Values.deezer.existingSecret }}
key: {{ .Values.deezer.key }}
optional: true
{{- else if .Values.deezer.arl }}
- name: DEEZER_ARL
value: {{ .Values.deezer.arl | quote }}
{{- end }}
{{- with .Values.env }}
{{- toYaml . | nindent 12 }}
{{- end }}
@@ -63,10 +101,6 @@ spec:
resources:
{{- toYaml .Values.resources | nindent 12 }}
volumeMounts:
- name: library
mountPath: /etc/audiomanager/library.yaml
subPath: library.yaml
readOnly: true
- name: users
mountPath: /etc/audiomanager/users.yaml
subPath: {{ .Values.users.key }}
@@ -75,16 +109,7 @@ spec:
mountPath: /data
- name: tmp
mountPath: /tmp
- name: music
mountPath: {{ .Values.media.music.mountPath }}
readOnly: true
- name: longform
mountPath: {{ .Values.media.longform.mountPath }}
readOnly: true
volumes:
- name: library
configMap:
name: {{ include "audiomanager.fullname" . }}-library
- name: users
secret:
secretName: {{ required "users.existingSecret is required" .Values.users.existingSecret }}
@@ -97,14 +122,6 @@ spec:
{{- end }}
- name: tmp
emptyDir: {}
- name: music
hostPath:
path: {{ .Values.media.music.hostPath }}
type: Directory
- name: longform
hostPath:
path: {{ .Values.media.longform.hostPath }}
type: Directory
{{- with .Values.nodeSelector }}
nodeSelector:
{{- toYaml . | nindent 8 }}
+1 -1
View File
@@ -3,7 +3,7 @@ apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: {{ include "audiomanager.fullname" . }}
namespace: {{ .Release.Namespace }}
namespace: {{ include "audiomanager.namespace" . }}
labels:
{{- include "audiomanager.labels" . | nindent 4 }}
{{- with .Values.ingress.annotations }}
+1 -1
View File
@@ -3,7 +3,7 @@ apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: {{ include "audiomanager.fullname" . }}-data
namespace: {{ .Release.Namespace }}
namespace: {{ include "audiomanager.namespace" . }}
labels:
{{- include "audiomanager.labels" . | nindent 4 }}
spec:
+1 -1
View File
@@ -2,7 +2,7 @@ apiVersion: v1
kind: Service
metadata:
name: {{ include "audiomanager.fullname" . }}
namespace: {{ .Release.Namespace }}
namespace: {{ include "audiomanager.namespace" . }}
labels:
{{- include "audiomanager.labels" . | nindent 4 }}
spec:
+43 -25
View File
@@ -1,5 +1,10 @@
replicaCount: 1
# Every resource is rendered into the namespace passed with --namespace (or
# helmCharts[].namespace in Kustomize). Rendering fails when that namespace is
# missing or falls back to "default"; set this to true to allow "default".
allowDefaultNamespace: false
image:
repository: gitea.tap-tap.win/taptap/audiomanager
# CI publishes :latest and :sha-<commit> for every push to main.
@@ -10,41 +15,54 @@ imagePullSecrets: []
nameOverride: ""
fullnameOverride: ""
# Rendered into a ConfigMap as /etc/audiomanager/library.yaml.
# Keep folder IDs stable: they are part of song IDs.
library:
folders:
- id: 1
name: "Music"
type: music
path: /mnt/media/music
- id: 10
name: "Audiobooks & Podcasts"
type: longform
path: /mnt/media/longform
# Vault-managed users.yaml. The secret must already exist and contain the key below.
users:
existingSecret: audiomanager-users
key: users.yaml
# Default catalog for new moods. Users can switch each mood between Deezer and
# YouTube Music. Deezer needs an ARL cookie: either `deezer.arl` below or a
# Secret referenced by `deezer.existingSecret`. Leave both empty to use
# YouTube Music only.
defaultSource: deezer
deezer:
# Option A (recommended on a shared machine): a Secret you manage separately.
# kubectl -n audiomanager create secret generic audiomanager-deezer \
# --from-literal=arl='<arl>'
existingSecret: ""
key: arl
# Option B: the ARL directly in values. It is stored in plaintext in this
# file, in `helm get values` and in the release Secret, so keep this file out
# of git (or pass it as `--set deezer.arl="$DEEZER_ARL"` instead).
# Set only one of `arl` and `existingSecret`.
arl: ""
# 0 = 128 kbps MP3 (Deezer Free), 1 = 320 kbps MP3 (Deezer Premium).
quality: 0
apiBase: https://api.deezer.com
# Per-track loudness normalization. New downloads are analyzed immediately and
# the existing library is backfilled gradually in the background.
loudness:
enabled: true
# -16 LUFS is the ReplayGain-style target; -14 matches most streaming services.
targetLufs: -16
# Client diagnostics uploaded by the apps. The server redacts credentials again
# and keeps only a short window so bug reports do not grow the database.
clientLogs:
retentionDays: 14
maxRowsPerUser: 20000
# The media root, database, cover cache and download temp dir all live here.
data:
persistence:
enabled: true
size: 5Gi
# Songs are downloaded automatically, so plan for growth.
size: 20Gi
storageClass: ""
accessModes:
- ReadWriteOnce
# hostPath mounts shared by every schedulable node. Read-only in the pod.
media:
music:
hostPath: /mnt/media/music
mountPath: /mnt/media/music
longform:
hostPath: /mnt/media/longform
mountPath: /mnt/media/longform
service:
type: ClusterIP
port: 8080
@@ -66,8 +84,8 @@ resources:
cpu: 100m
memory: 256Mi
limits:
cpu: 1000m
memory: 512Mi
cpu: 2000m
memory: 1024Mi
podSecurityContext:
fsGroup: 10001
-21
View File
@@ -1,21 +0,0 @@
# Example library configuration.
#
# The paths below are relative to the working directory and are meant for local
# development: `./gradlew run` creates a `./media` tree after you run
# `mkdir -p media/music media/longform`.
#
# In Kubernetes the real file is mounted as a ConfigMap and should use absolute
# paths, e.g. /mnt/media/music/alice and /mnt/media/longform.
#
# Folder IDs are stable identifiers used by songs and users. Never reuse an ID
# for a different path.
library:
folders:
- id: 1
name: "Music"
type: music
path: ./media/music
- id: 10
name: "Audiobooks & Podcasts"
type: longform
path: ./media/longform
+3 -3
View File
@@ -1,10 +1,10 @@
# Example users configuration. In the cluster this file comes from a
# Vault-managed Secret; never commit a real one.
#
# Long-form folders are shared automatically, so users only list their music
# folders here.
# Moods are strictly per-user and all music lives in the server-owned media
# store, so there are no folder assignments. An old `folders:` key is accepted
# and ignored.
users:
- name: admin
password: "change-me"
admin: true
folders: [1]
+774
View File
@@ -0,0 +1,774 @@
# Mood-first redesign — decisions and implementation handoff
Status: **approved design, implementation not started** (2026-11-15).
This document supersedes the mounted-library, local-search and longform parts
of the previous design. It is the source of truth for the next development
cycle; update it as implementation details are settled.
Repos:
- server: `/home/taptap/AudioManager` (Kotlin/Ktor, SQLite, Exposed, Flyway)
- client: `/home/taptap/AudioManagerClient` (Compose Multiplatform)
## 1. Product model
- Users have **moods only**. There is no mounted music directory, no file
upload and no longform/podcast content.
- Content enters the server by downloading from **Deezer** (default) or
**YouTube Music** through `streamrip` / `yt-dlp`. Each mood picks one source.
- Search is a remote catalog search against the mood's source. Results are
shown before anything is added.
- A mood contains:
- **Songs** — the user's collection. Added from search, from a recommendation,
or by attaching an already-downloaded song.
- **Recommendations** — candidates derived from the collection and mood
options. They are downloaded automatically so they can take part in radio
playback, but they are not part of the collection until the user adds them.
- A mood needs **at least one song** before it can be played.
- Playback builds each queue slot at random: with probability
`collectionPercent` it comes from the collection, otherwise from the ready
recommendations. An empty pool falls back to the other pool.
- `recommendationCount` (0–30, default 10) is the number of **active**
recommendations per mood: states `downloading`, `ready`, `failed`. `new`
candidates are a hidden pool used to refill slots. `added` and `rejected` are
decisions and do not count.
- Adding a recommendation turns it into a permanent mood song. Rejecting it is
permanent for that mood. Both free a slot and trigger a top-up.
- A failed recommendation download stays visible as `failed` with Retry / Not
interested; it counts as a slot until the user acts. It is never silently
replaced.
- `refreshHours` is the minimum interval for **automatic** discovery passes.
It never replaces a ready recommendation the user has not acted on.
- Removing a song only detaches it from the mood. Files are deleted only when
no mood references them.
- Songs are stored once server-wide, keyed by `(source, externalId)` and
deduplicated. Moods are strictly per-user: a user never sees another user's
moods or collection, but the same file can serve several users.
- No per-song owner, no storage quota.
## 2. Resolved decisions
1. `recommendationCount` counts downloading + ready + failed; add/reject frees
a slot and the server refills it.
2. Adding turns a recommendation into a collection song and removes it from the
recommendation list. Rejecting is permanent per mood and deletes the file
when unreferenced.
3. Failed downloads stay visible with Retry / Not interested.
4. `refreshHours` only throttles automatic discovery/top-up; it never replaces a
ready recommendation.
5. Small pools loop/replay least-recently played instead of going silent.
6. `seedPercent` becomes `collectionPercent`; recommendation plays count in
history and affinity like songs.
7. Mood options: name, description, query, genres, collectionPercent,
recommendationCount (default 10, max 30), refreshHours. `targetSize`,
`ytEnabled` are removed.
8. No accept step. Rows show download state, Add to mood, Not interested; failed
rows add Retry.
9. Search is one track at a time plus paste-a-URL; multi-select later. Results
longer than **10 minutes** are hidden. Already downloaded/in-mood results are
marked, and attaching an existing song is instant. Search exists inside a
mood and globally with a mood picker.
10. Library and Longform tabs are removed; Downloads stays for offline files.
Global Shuffle all is removed; playback stays per mood.
11. **Migration: full wipe.** Songs, moods, recommendations, downloads and
longform data are dropped. No import from the old mount.
12. Existing moods are wiped with the content.
13. Chart uses a PVC for `/data` (media at `/data/media`); music hostPath,
`library.yaml`, user folder lists, `DOWNLOADS_ENABLED` and
`DOWNLOAD_FOLDER_ID` disappear.
14. One song per `(source, externalId)`; the same recording from two catalogs is
two songs. Deezer track ids and YouTube video ids do not overlap.
Manual downloads outrank automatic recommendation downloads. Detach only;
GC deletes unreferenced files. A removed playing track finishes and drops
from future queues.
15. Recommendations are downloaded up to the configured count even if the mood
is never played.
16. Backups are external; no export/import feature and no offline CLI import.
17. Client: remove Library/Longform; keep Downloads for offline; add flow is
search -> download -> attach to mood; existing downloads attach instantly;
downloads run server-side with optional device downloads for offline.
18. Metadata is kept for display and search (title, artist, album, duration,
thumbnail). Separate album/artist/genre browse tables are removed.
## 3. What exists today (server)
Current files and responsibilities, and what happens to them:
| File | Today | New |
|---|---|---|
| `config/AppConfig.kt` | data dir, library/users paths, downloads flags | drop library path and download flags; add `MEDIA_DIR` |
| `config/LibraryConfig.kt` | folders, `FolderType` | delete |
| `config/UsersConfig.kt` | users + folder IDs | users only; ignore/remove `folders` |
| `db/ConfigRepository.kt` | sync folders + users | sync users only |
| `db/Tables.kt` | folders, artists/albums/genres, songs, moods, seeds, recommendations, jobs, sessions, affinity, longform, tombstones | rewrite for the new schema |
| `db/MediaRepository.kt` | scan upsert, album/artist aggregates, orphan delete | simplify to song upsert by video id + GC |
| `db/LibraryRepository.kt` | folder-scoped browse/search/delta/random | delete; replace with `SongRepository` |
| `db/MoodRepository.kt` | moods, seeds, recommendations | rewrite around `mood_songs` + recommendation lifecycle |
| `db/RadioRepository.kt` | sessions, items, affinity | keep, minus folder/library assumptions |
| `db/DownloadRepository.kt` | per-user/mood jobs, longform progress | rewrite job model; move longform repository out |
| `discovery/DiscoveryService.kt` | seeds + genres/query -> candidates, review | same inputs from collection; store `new` candidates; no `matchedSongId` library matching |
| `discovery/YtSearch.kt` | yt-dlp flat search | YouTube Music songs search with resolved metadata; regular `ytsearch` fallback; add URL resolve |
| `download/DownloadService.kt` | accept -> import folder -> scan -> seed | index directly into media store; attach song/recommendation; GC on failure |
| `download/YtDlpDownloader.kt` | MP3 + embedded metadata | keep; target `<source>-<externalId>/<title>.mp3` |
| `download/DeezerDownloader.kt` | - | streamrip with `DEEZER_ARL`; 128/320 MP3 |
| `scanner/MediaScanner.kt` | walks folder config | walks `MEDIA_DIR`; derive video id from path |
| `scanner/Mp3TagExtractor.kt` | tags/cover/lyrics | keep tags+cover; drop album/artist IDs and lyrics use |
| `scanner/CoverArtService.kt` | album-keyed covers | song-keyed covers |
| `radio/RadioService.kt` | library + seeds pool | collection + ready recommendations pool |
| `api/LibraryApi.kt` | library sync/search/stream/covers | keep stream; cover by song; delete sync/search |
| `api/LongformApi.kt` | progress | delete |
| `api/MoodApi.kt` | seeds + accept/reject | songs + add/reject/retry + top-up |
| `api/SessionApi.kt` | sessions, events, add-to-mood | keep sessions/events; adapt add-to-mood |
| `api/ApiModels.kt` | old DTOs | rewrite |
| `api/ApiRoutes.kt` | wiring + scan | wire new services; users only |
| `Application.kt` | library/users/scan/download/discovery loops | media/users/download/top-up/GC loops |
Current tests (server): `ApiRoutesTest`, `DatabaseFactoryTest`,
`DownloadServiceTest`, `MediaScannerTest`, `Mp3TagExtractorTest`,
`CoverArtServiceTest`, `DiscoveryServiceTest`, `RadioServiceTest`,
`AppConfigTest`, etc. They must be rewritten around the new model; keep useful
cases (auth, ownership, streaming, transcription, permissions).
## 4. Target server architecture
### 4.1 Media store
- One server-owned root: `MEDIA_DIR` (default `<data>/media`; chart PVC at
`/data`, `DATA_DIR=/data`, `MEDIA_DIR=/data/media`).
- Download temp/work dir: `<data>/downloads` or `<media>/.tmp`; must be on the
same filesystem as the final path for atomic moves.
- Layout: `<media>/<source>-<externalId>/<sanitized-title>.mp3` plus
`<media>/cover-cache/...` or a song-keyed cover derived from embedded art.
The first path segment carries the source and external id, which makes startup scans
deterministic and independent of tags.
- Song id: `sg-` + SHA-256 prefix of `"yt:<videoId>"` for YouTube (stable legacy
ids) or `"dz:<trackId>"` for Deezer.
- MP3 only (`yt-dlp -x --audio-format mp3 --audio-quality 0`), embedded
metadata and thumbnail.
- Loudness: after indexing, ffmpeg measures `input_i`/`input_tp` with
`loudnorm`. `gain_db = min(target - input_i, truePeak - input_tp)` is stored;
clients apply it per track. New downloads are measured inline, the existing
library is backfilled one file at a time. `LOUDNESS_ENABLED` and
`LOUDNESS_TARGET_LUFS` control it.
- The scanner walks `MEDIA_DIR`, extracts `(source, externalId)` from the top-level folder,
reads tags/cover, and upserts the song. The downloader can call the same
indexer for one file to avoid waiting for a full scan.
- Orphan cleanup keeps the old safety guard: never wipe when the media root is
missing/unreadable. GC is reference-based (see 4.6).
### 4.2 Schema (V8)
Keep `users` and `api_tokens`. Drop and recreate everything else.
```sql
songs(
id TEXT PRIMARY KEY,
source TEXT NOT NULL DEFAULT 'youtube',
external_id TEXT NOT NULL UNIQUE,
title TEXT NOT NULL,
artist TEXT NOT NULL,
album TEXT,
duration INTEGER NOT NULL DEFAULT 0,
bit_rate INTEGER NOT NULL DEFAULT 0,
size INTEGER NOT NULL DEFAULT 0,
suffix TEXT NOT NULL DEFAULT 'mp3',
content_type TEXT NOT NULL DEFAULT 'audio/mpeg',
relative_path TEXT NOT NULL,
file_mtime INTEGER NOT NULL DEFAULT 0,
thumbnail TEXT,
created_at TEXT NOT NULL,
updated_at TEXT NOT NULL
)
moods(
id TEXT PRIMARY KEY,
owner TEXT NOT NULL REFERENCES users(name) ON DELETE CASCADE,
name TEXT NOT NULL,
description TEXT,
query TEXT,
genres TEXT NOT NULL DEFAULT '[]',
collection_percent INTEGER NOT NULL DEFAULT 50,
recommendation_count INTEGER NOT NULL DEFAULT 10
CHECK (recommendation_count BETWEEN 0 AND 30),
refresh_hours INTEGER NOT NULL DEFAULT 24,
last_discovery_at TEXT,
created_at TEXT NOT NULL,
changed_at TEXT NOT NULL,
last_error TEXT
)
mood_songs(
mood_id TEXT NOT NULL REFERENCES moods(id) ON DELETE CASCADE,
song_id TEXT NOT NULL REFERENCES songs(id) ON DELETE CASCADE,
origin TEXT NOT NULL DEFAULT 'search'
CHECK (origin IN ('search', 'recommendation', 'url')),
added_at TEXT NOT NULL,
PRIMARY KEY (mood_id, song_id)
)
mood_recommendations(
mood_id TEXT NOT NULL REFERENCES moods(id) ON DELETE CASCADE,
source TEXT NOT NULL DEFAULT 'youtube',
external_id TEXT NOT NULL,
title TEXT NOT NULL,
artist TEXT NOT NULL,
album TEXT,
duration INTEGER,
thumbnail TEXT,
state TEXT NOT NULL DEFAULT 'new'
CHECK (state IN ('new', 'downloading', 'ready', 'failed', 'added', 'rejected')),
song_id TEXT REFERENCES songs(id) ON DELETE SET NULL,
attempts INTEGER NOT NULL DEFAULT 0,
error TEXT,
first_seen TEXT NOT NULL,
last_seen TEXT NOT NULL,
PRIMARY KEY (mood_id, external_id)
)
download_jobs(
id TEXT PRIMARY KEY,
source TEXT NOT NULL DEFAULT 'youtube',
external_id TEXT NOT NULL,
mood_id TEXT NOT NULL REFERENCES moods(id) ON DELETE CASCADE,
kind TEXT NOT NULL CHECK (kind IN ('song', 'recommendation')),
state TEXT NOT NULL DEFAULT 'queued'
CHECK (state IN ('queued', 'downloading', 'done', 'failed')),
priority INTEGER NOT NULL DEFAULT 0,
attempts INTEGER NOT NULL DEFAULT 0,
error TEXT,
song_id TEXT REFERENCES songs(id) ON DELETE SET NULL,
created_at TEXT NOT NULL,
updated_at TEXT NOT NULL
)
playback_sessions(id, user_name, mood_id, next_position, created_at, last_seen_at, closed_at)
playback_session_items(session_id, position, song_id, state, queued_at, updated_at)
song_affinity(user_name, song_id, completions, skips, last_event_at)
```
Drop: `music_folders`, `artists`, `albums`, `album_artists`, `song_artists`,
`song_genres`, `album_genres`, `artist_genres`, `mood_seeds`,
`library_tombstones`, `longform_progress`, `user_song_state`, `user_stars`,
`bookmarks`, old `download_jobs`, old `mood_recommendations`, old `moods`,
old `songs`.
`DatabaseFactory.normalizeLegacyMoodStates` stays: an old V4 DB must still pass
V5/V6/V7 before V8 can wipe it.
### 4.3 Recommendation lifecycle
States: `new` (hidden candidate pool), `downloading`, `ready`, `failed`,
`added`, `rejected`.
- Discovery reads the **whole collection** (not one seed track): every artist
with its song count, every album, `query`, `genres` and the mood name.
- Discovery has two candidate sources, merged and ranked together:
1. **Text queries**: `query`, then up to 3 collection artists ordered by how
many songs the mood has, then up to 2 genres, capped at 4 queries.
2. **Catalog similarity graph** (`MusicCatalog.discover`): up to 5 collection
artists -> their **radio** (`/artist/{id}/radio`, the platform's own mix)
and top tracks; up to 2 of those artists' related artists -> their top
tracks; up to 2 collection albums -> album deep cuts. YouTube Music runs
the `RDAMVM<videoId>` watch-playlist radio for the 2 newest collection
songs, which is the same queue its player builds (`yt-dlp
--flat-playlist`), so it also returns real recommendations.
- Scoring: exact collection artist `3.0 + 0.5` per extra song (capped `+1.5`),
partial artist `1.5`, album already in the collection `+1.0`, catalog
affinity (platform radio `2.0`, related artist `1.5`, album deep cut `1.0`,
artist top `0.5`), mood words in the title `+1.0`, over 8 minutes `-1.0`.
- A recording is deduplicated by `(artist, title)` as well as by external id, so
two uploads of a song already in the mood do not both enter the pool.
- Stores up to a bounded pool of `new` candidates (60). Excludes external ids
already in `mood_songs`, `added` or `rejected`.
- `topUp(mood, force)`:
1. Count active = `downloading + ready + failed`.
2. Promote the best `new` candidates to `downloading` and enqueue
`download_jobs` until `recommendationCount` is met.
3. Run discovery when slots are missing, or when a song was added after the
last pass (the pool is based on the collection), and
(`force || discoveryDue(mood)`).
4. Promote again after the pass.
- `force = true` on user add/reject/retry, on a finished manual download and on
the explicit refresh action. Background passes use `force = false` and run
hourly.
- `recommendationCount = 0` stops discovery and downloads for the mood.
- On download success:
- `downloading` -> `ready` with `song_id`.
- `added` (user added while downloading) -> insert `mood_songs`, keep
`added`, do not create a duplicate.
- `rejected` -> delete the file if unreferenced.
- On final download failure: `failed` with `error`; keep the slot.
- Add: if `ready`, attach the song; if `downloading`, mark `added` and attach
when the job completes; insert `mood_songs` with origin `recommendation`;
top up.
- Reject: mark `rejected`, cancel the job, GC, top up.
- Retry: `failed` -> `downloading`, reset the job, top up.
- `new` candidates are not shown in the client.
### How this compares to the big services
The commercial engines are trained on behaviour this server does not have: no
account-level listening history, no cross-user co-occurrence, no audio models.
Still, their published architecture explains which parts of the problem can be
replicated from a mood collection alone.
- **Spotify** combines content-based features (audio features such as
danceability, energy and valence, plus metadata and text/visual embeddings)
with collaborative filtering that treats playlist co-occurrence as the
similarity signal ("two tracks are similar if people curate them together").
Retrieval produces candidates, a ranking stage orders them, and each surface
has its own reward (skips, saves, session length).
- **YouTube Music** runs retrieval -> ranking -> filtering. Candidates come from
a deep model over watch history, and a 2024 transformer encodes the *sequence*
of user actions (intent, how much of a track was played, artist, language)
before blending with the ranker. Audio-text embeddings (MuLan) describe the
content itself.
- **Deezer Flow / Flow Moods** (RecSys 2022) mixes collaborative filtering,
audio analysis and professional curator mood annotations to build
mood-specific radios; the 2026 Flow Tuner adds genre toggles, explicit
dislikes, long-term habits and context such as time of day.
What this server reproduces without telemetry:
| Signal | Here | There |
|---|---|---|
| Taste profile | whole mood collection, artists weighted by song count + albums | per-user history and playlists |
| Similarity | the platform's own radio (Deezer artist radio, YouTube `RDAMVM`) + catalog graph + text queries | learned embeddings + co-occurrence |
| Feedback | add / not interested (permanent) | skips, saves, play counts, survey models |
| Context | mood hint, genres, refresh window | session, time of day, device, activity |
| Ranking | weighted score + per-artist diversity penalty | deep rankers with hundreds of features |
Not reproducible here: cross-user collaborative filtering, audio analysis and
sequence models. The add/reject lifecycle and the collection itself are the
only behavioural signals, which is why discovery leans on the platforms' own
radio endpoints. Spotify is not an option: its `/v1/recommendations`, related
artists, audio features and audio analysis were removed for new apps on
2024-11-27, and its terms forbid training on Spotify content.
### 4.4 Download pipeline
- `download_jobs` are global; `kind=song` (manual add) has higher priority than
`kind=recommendation`. Claim by `priority DESC, created_at ASC`.
- Before downloading, check `songs(source, external_id)`:
- if the song exists and the file is present, attach immediately;
- otherwise download.
- A per-video download is effectively deduplicated by the existence check; two
moods may queue the same video, but only the first downloads and the second
completes instantly.
- Worker steps:
1. Claim a job.
2. Resolve target `<media>/<source>-<externalId>/<title>.mp3`.
3. If the song exists, finish.
4. Download to temp with yt-dlp/streamrip; move atomically; index
the file (tags/cover) and upsert the song by video id.
5. Apply the job outcome (song attach or recommendation state).
6. On permanent failure, remove the partial file and set `failed`.
- Keep the 3-attempt retry, stale-job recovery, permission classification and
UTF-8 filename truncation from the current implementation.
- Rejected/cancelled jobs must not later attach the track.
### 4.5 Playback / radio
- Inputs: collection songs (`mood_songs`) and `ready` recommendation songs.
- 0 collection songs -> API rejects session start with a clear error; the client
shows "Add at least one song".
- Candidate scoring: affinity (`completions - 3*skips`), recent-play penalty,
jitter, plus artist/album spacing in `pickMixed`. No genre/artist library
matching.
- `pickMixed(collection, recommendations, limit, collectionPercent)`:
each slot rolls the percentage; an empty pool falls back to the other one.
- Session de-duplication: exclude songs already queued in the session.
- Exhaustion: allow replay of the least-recently queued songs, but never
duplicate within one batch. A one-song mood returns one track per batch and
the client loops it.
- Persist session items and playback events exactly as today.
- Recommendation plays count in history/affinity.
- Remove `pickDiverse` if it becomes dead code.
### 4.6 File GC
A song is referenced by:
- any `mood_songs` row;
- any `mood_recommendations` row with `song_id = song` and state in
`downloading|ready|added|failed` (`added` also has `mood_songs`);
- an open playback session item (avoid deleting a track currently streaming).
When the reference count is zero, delete the song row, the media
directory and the cover. Run GC:
- after removing a song;
- after rejecting a recommendation;
- after deleting a mood;
- after a cancelled/failed job with no references;
- periodically (hourly) to catch leftovers.
### 4.7 Search and URL adds
- `GET /api/v1/search?q=...&limit=...&source=...&moodId=...` -> catalog results
with `source,externalId,title,artist,album,duration,thumbnail,songId?,downloaded,inMood`.
- `source` falls back to the mood's source, then `DEFAULT_SOURCE`.
- Hide results with `duration > 600` seconds and `< 30`.
- `POST /api/v1/moods/{id}/songs` accepts `songId` (existing download),
`source`+`externalId`, or `url`. Deezer URLs use the public API; YouTube URLs
use yt-dlp `--dump-single-json --no-playlist`.
- Search requires auth; Deezer is a fast public API call, while YouTube Music
resolves every hit (use a 60s client timeout). Return 502 on failure.
### 4.8 API surface
```
POST /api/v1/auth/login
POST /api/v1/auth/logout
GET /api/v1/me
GET /api/v1/moods
POST /api/v1/moods
GET /api/v1/moods/{id}
PATCH /api/v1/moods/{id}
DELETE /api/v1/moods/{id}
GET /api/v1/moods/{id}/preview
GET /api/v1/moods/{id}/recommendations
POST /api/v1/moods/{id}/recommendations/refresh
POST /api/v1/moods/{id}/recommendations/{externalId}/add
POST /api/v1/moods/{id}/recommendations/{externalId}/reject
POST /api/v1/moods/{id}/recommendations/{externalId}/retry
POST /api/v1/moods/{id}/songs
DELETE /api/v1/moods/{id}/songs/{songId}
POST /api/v1/moods/{id}/sessions
GET /api/v1/sessions/{id}
POST /api/v1/sessions/{id}/next
POST /api/v1/sessions/{id}/events
DELETE /api/v1/sessions/{id}
GET /api/v1/search
GET /api/v1/songs/{id}/stream (Range/transcoding as today)
GET /api/v1/songs/{id}/cover
GET /api/v1/scan (admin diagnostic; optional)
POST /api/v1/scan (admin; optional)
```
DTO sketch:
```text
UserDto(name, admin)
SongDto(id, source, externalId, title, artist, album, duration, bitRate, size, suffix,
thumbnail)
MoodDto(id, name, description, query, genres, collectionPercent,
recommendationPercent, recommendationCount, refreshHours,
lastDiscoveryAt, createdAt, changedAt, lastError, songCount,
recommendationReady, recommendationDownloading, recommendationFailed)
MoodSongDto(song, origin, addedAt)
MoodDetailDto(mood, songs, recommendations, downloads)
RecommendationDto(source, externalId, title, artist, album, duration, thumbnail, state,
song?, download?, firstSeen, lastSeen)
SearchResultDto(source, externalId, title, artist, album, duration, thumbnail, songId?,
downloaded)
```
Remove: `LibraryPage`, `SearchResponse` of local songs, all longform DTOs,
`SeedDto`, `AddSeedRequest`, `targetSize`, `ytEnabled`, folder fields.
### 4.9 Auth and users
- `users.yaml` keeps `name`, `password`, `admin`. Remove `folders`; accept and
ignore existing `folders` keys for compatibility.
- No folder validation, no `musicFolderIds`/`longformFolderIds`.
- `ConfigRepository.sync(users)` only reconciles users.
- Admin remains for optional scan/maintenance endpoints.
## 5. Client design
### 5.1 Screens
`Screen = Moods | MoodDetail | Search | Downloads | Settings`
- **Moods**: mood grid/list, create, global search icon.
- **MoodDetail**:
- header/play/settings;
- **Songs** collection with play/remove;
- **Add music** -> Search targeted at this mood;
- **Recommendations**: active rows with thumbnail, state/progress, Add to
mood, Not interested, Retry (failed);
- **Next batch** preview;
- mood options: name, description, query, genres, collectionPercent,
recommendationCount (0–30, default 10), refreshHours.
- **Search**: query field, debounced remote search, results (hide >10min done
server-side), Add; if opened globally, choose a mood; badges for downloaded /
already in mood; paste-a-URL action.
- **Downloads**: device-side downloaded songs, play/delete/clear, storage usage.
- **Settings**: server/account/devices/logout; remove scan UI.
- Remove `LibraryScreen` and all longform UI.
### 5.2 State and store
- `AppState` remove: `library`, `longform`, `longformResults`,
`longformProgress`, `musicResults`, `searchResults` (replace with
`searchResults: List<SearchResult>` + `searchTargetMoodId`), longform
progress.
- `CachedState` version 2: `serverUrl`, `userName`, `moods`, `moodDetails`
(each including songs/recommendations). No revision/longform. Invalidate old
caches on version mismatch.
- `AppStore`:
- bootstrap restores session, moods/details, playback snapshot and position;
- `refreshAll` fetches moods only; selected mood detail on open/refresh;
- create/update/delete mood;
- `addSong`, `removeSong`, `addRecommendation`, `rejectRecommendation`,
`retryRecommendation`, `refreshRecommendations`;
- `searchRemote(query)` with debounce and cancellation;
- `playMood` uses sessions; clear error when the mood has no songs; offline
fallback = cached mood songs filtered by device downloads;
- remove `playAll`, `playLongform*`, `resumeLongform`, `savedPosition`,
longform progress reporting;
- keep playback snapshot/history/previous/sleep timer/queue editing;
- keep device downloads (`toggleDownload`, `clearDownloads`) and artwork;
- `addCurrentToMood` works for collection songs via `songId`.
- `ApiClient`: new models/routes; remove library/longform; `coverUrl(songId)`.
- Mood detail polling while recommendations/downloads are active (bounded, with
backoff; no infinite loop).
### 5.3 Playback
Keep `AudioPlayer`, desktop seek, Android media session, notification, sleep
timer, history and queue snapshot. Only the server content model changes.
Offline playback uses device downloads; online mood playback uses sessions.
## 6. Migration
- **V8 drops all content**, keeps users and tokens.
- The old mounted music directory is no longer mounted; the operator must
delete old files manually. The new media root starts empty.
- The chart's old music/longform mounts and `library.yaml` are removed; add a
PVC.
- Client cache version bump clears old library/longform data.
- No CLI import and no app export (external backups only).
## 7. Ops
- `AppConfig`: `port`, `dataDir`, `usersFile`, `ffmpegPath`, `ytDlpPath`,
`mediaDir` (`MEDIA_DIR`, default `<dataDir>/media`), `downloadDir`
(`<dataDir>/downloads`). Remove `libraryConfigPath`, `downloadsEnabled`,
`downloadFolderId`.
- `Application.kt`: users only; scanner over `mediaDir`; download worker;
discovery/top-up loop; GC loop. No library scan-at-start for empty DB (scan
media root at startup is fine).
- Chart:
- one PVC for `/data`, `fsGroup` 10001, no hostPath/library ConfigMap, no
`prepare-imports` init container;
- `DATA_DIR=/data`, `MEDIA_DIR=/data/media`;
- keep users Secret, probes, resources; add persistence values;
- README deployment section rewritten.
- Nix shell: no library config; `DATA_DIR=./data`.
- Docker: `yt-dlp`, `ffmpeg`, Python 3 and `streamrip` (provides `rip`).
- Smoke test: rewrite for mood creation, search stub or seeded songs, add,
recommendations, playback. A fake `yt-dlp` script can return deterministic
search JSON and a tiny MP3 fixture for downloads.
## 8. Implementation plan
### Phase 1 — server schema and repositories
- V8 migration.
- Rewrite `Tables.kt`.
- `SongRepository` (upsert by video id, get by video id/id, list by mood,
attach/detach, refcount, delete).
- `MoodRepository` (CRUD, songs, recommendations lifecycle, top-up helpers).
- `DownloadRepository` (jobs, priority, claim, recovery, cancel, retry).
- GC service/repository.
- Tests: migration wipe, song dedupe, mood songs, recommendation state machine,
refcount/GC.
### Phase 2 — media store and downloader
- `AppConfig`/users cleanup.
- Path layout + indexer shared by scanner and downloader.
- `MediaScanner` over media root, video id from path.
- `CoverArtService` song-keyed.
- `YtDlpDownloader` target path; direct indexing after move.
- `DownloadService` attach-to-song/recommendation; failure cleanup; priority.
- Tests: manual add, existing song attach, shared file across moods/users,
reject during download, retry, GC, UTF-8 names, scan recovery.
### Phase 3 — discovery, top-up, radio
- `DiscoveryService` inputs from collection; store `new`; no library matching;
10-minute filter.
- `RecommendationTopUpService` (or methods in `MoodRepository`/`DiscoveryService`).
- `RadioService` collection + ready recommendations.
- Background loops: top-up hourly, GC hourly.
- Tests: N active, refill on add/reject, failed stays, refresh top-up only,
0/50/100 percent, small-pool repeat, no duplicates per batch, 0 songs error.
### Phase 4 — API
- New DTOs and routes; remove library/longform/seed endpoints.
- Search endpoint + URL resolve.
- Cover by song id.
- Session start guard.
- Tests: ownership, add/reject/retry, search filter, streaming, 400s, auth.
### Phase 5 — server ops
- Chart PVC, env, ConfigMap removal, init container removal.
- Nix/README/smoke.
- Chart check.
### Phase 6 — client data layer
- Models/ApiClient/cache version/store rewrite.
- Remove longform/library code paths.
- Tests: search/add, recommendation actions, play empty mood, offline fallback,
cache migration, playback snapshot.
### Phase 7 — client UI
- Moods/MoodDetail/Search/Downloads/Settings.
- Remove Library/Longform screens; adapt PlayerBar.
- Tests/builds: desktopTest, assembleDebug, Wasm.
### Phase 8 — integration
- Full server tests, smoke, chart check.
- Client tests/builds.
- Commit/push both repos, CI green.
- Optional: deploy, wipe DB/media, validate real downloads.
## 9. Test plan (high signal)
Server:
- V7 -> V8 migration keeps users/tokens, drops content, creates schema.
- Song upsert dedupes by video id; same video different moods/users -> one file.
- Manual add: new download -> attach; existing -> instant attach.
- Recommendation top-up: N active; add/reject refills; failed stays; refresh
does not replace ready; `recommendationCount=0` disables discovery.
- Reject during download cancels and GCs.
- GC keeps files referenced by another mood/session and deletes unreferenced.
- Radio: collection-only, recommendation-only mix, fallback, 1-song replay,
no duplicate in a batch, 0-song start error.
- Search: duration >10min hidden, malformed yt-dlp output, URL resolve.
- Auth/ownership regressions.
Client:
- Search results/add flow, existing song instant, badges.
- Recommendation add/reject/retry UI state.
- Play mood with zero songs shows error; queue and history still work.
- Offline playback from device downloads; cache version reset.
- Device download/delete/clear.
## 10. Current repo and deployment state
- Server latest commit: `4be502b` (plus this uncommitted doc). Image
`sha-4be502b`/`latest`; chart `0.2.4-sha.4be502b`.
- Client latest commit: `d9976a6`; CI green; Play internal release
`v0.1.1`, versionCode 2, package `win.tap_tap.audiomanager.client`.
- Client release automation: Gradle Play Publisher 3.13.0, tag `v*` ->
`publishBundle` to internal, `ResolutionStrategy.AUTO`; Gitea secrets set.
- Server deployment still uses the old mounted-library design; upgrading wipes
content and needs a PVC.
- Play App Signing is active (Google-generated key).
## 11. Risks / implementation notes
- Auto-download of up to 30 recommendations per mood can consume significant
disk/bandwidth even if the mood is never played. This is accepted.
- yt-dlp bot checks can fail downloads/searches; failed states and retry must
be robust, and search must return a clear 502.
- The 10-minute filter must apply to both search and discovery; longform is
gone.
- Cover art becomes song-keyed; the client must use `coverUrl(songId)`.
- Streaming must still support Range and transcoding for the Wasm/desktop
clients.
- Don't delete a file that an open playback session or another mood references.
- `recommendationCount` is clamped 0..30; `collectionPercent` 0..100.
- Search results are transient; no DB rows until add.
- A recommendation that is `added` while downloading must not turn into a
duplicate collection row.
- The migration is destructive by design; document it prominently.
## Implementation status (2026-11-17)
Phases 1-8 are implemented in both repositories.
### Server
- `V8__mood_first.sql` drops the mounted-library schema and creates
`songs`, `moods`, `mood_songs`, `mood_recommendations`, `download_jobs`,
playback tables and `song_affinity`. Users and device tokens survive.
The `mood_recommendations` table has an additional `rank` column so
promotion follows the discovery score order.
- `SongRepository` dedupes by `(source, externalId)` and is the only song table access.
- `MediaStore` owns `<MEDIA_DIR>/<source>-<externalId>/<title>.mp3` (legacy
YouTube folders without a prefix are still read); `SongIndexer` is shared by
the scanner and the downloader.
- `MoodRepository` implements the song collection, hidden `new` candidates,
active slots and the add/reject/retry lifecycle.
- `DiscoveryService` + `TopUpService` implement whole-collection discovery and
slot refills: text queries plus `MusicCatalog.discover` (Deezer related
artists, artist top tracks, album deep cuts; YouTube Music text only).
`TopUpService` coalesces work behind a mutex and can be triggered
fire-and-forget from API handlers and finished downloads. A song added after
the last pass marks discovery due again, so the pool follows the collection.
- `DownloadService` is a single global worker with `song` jobs outranking
`recommendation` jobs. It indexes immediately after moving the file.
- `MediaGcService` implements refcounted deletion (moods, active recommendations,
open sessions).
- The API surface is the one in section 4.8. Differences: search returns
`{"results": [...]}`, recommendation decisions are `add`/`reject`/`retry`, and
covers use `/songs/{id}/cover`. `/moods/{id}/songs` returns `202` with
`downloading: true` when a new video was queued.
- `AppConfig` is `port`, `dataDir`, `usersFile`, `ffmpegPath`, `ytDlpPath`,
`streamripPath`, `defaultSource`, `deezerArl`, `deezerQuality`,
`deezerApiBase`, `mediaDir`, `downloadDir`. `LibraryConfig` and the folder
APIs are deleted.
- `YtDlpSearch` searches `music.youtube.com` with the Songs filter and resolves
each hit, so results carry the canonical artist, album and duration.
`YtDlpDownloader` requests the Music watch URL first and falls back to
`www.youtube.com` for videos that are not on the Music host. If the Music
search itself fails, search falls back to `ytsearch`.
- Background loops: download worker (continuous), recommendation top-up
(hourly), media GC (hourly), startup scan.
- Chart 0.3.0 uses one `/data` PVC, `MEDIA_DIR=/data/media`, no hostPath, no
library ConfigMap and no init container.
- V9 adds `source` to songs, moods, recommendations and download jobs and
renames `video_id` to `external_id`; existing rows are YouTube.
- `MusicCatalogs` dispatches search/resolve by source (`DeezerSearch` +
`YtDlpSearch`). `SourceDownloaders` dispatches downloads (`DeezerDownloader`
via streamrip + `YtDlpDownloader`).
- Deezer is the default for new moods; every mood stores its own source and
can switch it, which clears the active recommendation pool and refills it.
- Chart 0.4.0 adds `DEFAULT_SOURCE`, `DEEZER_*` and an optional ARL secret.
- Tests: 97 server tests, 67-step smoke test with fake `yt-dlp`, `rip` and a
fake Deezer API (including the artist graph and both radio paths).
### Client
- Models, `ApiClient`, `AppCache` (version 3 + separate client settings file) and `AppStore` match the new API.
- Screens are `Moods | MoodDetail | MoodSettings | Search | Downloads | Settings |
NowPlaying`.
`LibraryScreen`, longform UI, seed management and global shuffle are gone.
- Search supports a target-mood picker, a **Deezer / YouTube Music** source
selector, downloaded badges and paste-a-URL.
- Mood detail shows songs, add music, recommendation states (Add / Not
interested / Retry). The preview/next-batch section and the inline settings
form are gone: mood settings (including source and the offline switch) live on
a dedicated screen, with Delete at the bottom.
- **0.4.0 playback**: Android's `MediaSessionService` owns a Media3 timeline,
so notification/headset next/back and gapless playback use real media items.
Previous keeps the 2 s restart rule. The app mirrors the index, appends the
next server batch and still reports skipped/completed events.
- **Audio**: server gain is applied by an Android PCM `AudioProcessor` (boost and
attenuation) and by the desktop mixer; the full-screen player shows source and
bitrate. Android has an optional 0–8 s fade between tracks.
- **Offline moods**: a per-mood switch downloads every song and keeps new
additions in sync on unmetered networks, three at a time.
- **Sleep timer** adds "Finish this track"; the full-screen player shows large
artwork, transport, add-to-mood, queue and Up next.
- **Diagnostics (V11)**: the client keeps 500 local info/warn/error entries and
uploads up to 100 per minute to `POST /logs`, tagged with a random install id,
platform, app version and per-process session id. Warnings and errors flush
immediately so a crash keeps its cause. `Authorization`, `Bearer`, `token=`,
`password=` and `arl=` values are redacted client-side and again server-side.
Rows are per user, admins may read another user with `?user=`, and retention is
`CLIENT_LOG_RETENTION_DAYS` (14) plus `CLIENT_LOG_MAX_ROWS_PER_USER` (20000).
Uncaught exceptions are stored on Android and desktop and uploaded on the next
launch. Every response carries `X-Request-Id`, which the client includes in
server-error diagnostics so a report can be matched to the server log.
- Tests: 26 client tests; desktop, Android debug and Wasm compile.
### Known follow-ups
- Real-device Android background playback and desktop audio checks still need a
machine with an audio device.
- Automatic downloads are bounded by `recommendationCount` but can still consume
noticeable disk/bandwidth for unplayed moods.
- yt-dlp bot checks remain an operational risk; failed rows keep Retry.
- A Deezer Free ARL is limited to 128 kbps and ARL downloads are outside the
public API terms. A Premium ARL enables 320 kbps MP3 (`DEEZER_QUALITY=1`).
+97
View File
@@ -0,0 +1,97 @@
#!/usr/bin/env bash
#
# Render and verify the Helm chart. Every namespaced resource must be rendered
# into the namespace that was passed to Helm, and rendering must fail when no
# namespace is passed instead of silently falling back to "default".
#
# Requires helm on PATH (`nix-shell` provides it).
set -euo pipefail
ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
CHART_DIR="${CHART_DIR:-$ROOT/charts/audiomanager}"
RELEASE="${RELEASE:-audiomanager}"
NAMESPACE="${NAMESPACE:-audiomanager}"
workdir="$(mktemp -d "${TMPDIR:-/tmp}/chart-check.XXXXXX")"
trap 'rm -rf "$workdir"' EXIT
# An empty kubeconfig makes the implicit namespace deterministic ("default"),
# independent of whatever context the developer or runner has configured.
(umask 077 && : > "$workdir/kubeconfig")
echo "helm lint ($NAMESPACE)"
helm lint "$CHART_DIR" --namespace "$NAMESPACE" >/dev/null
echo "render ($NAMESPACE)"
helm template "$RELEASE" "$CHART_DIR" \
--namespace "$NAMESPACE" \
--set users.existingSecret=test \
> "$workdir/rendered.yaml"
# A namespace per document: fail if a namespaced kind has no metadata.namespace
# or carries a different one. Cluster-scoped kinds are listed explicitly so a
# new template cannot quietly skip the namespace.
NAMESPACE="$NAMESPACE" awk '
function cluster_scoped(kind) {
return kind == "Namespace" \
|| kind ~ /^Cluster/ \
|| kind ~ /^(PersistentVolume|StorageClass|IngressClass|Node|PriorityClass)$/ \
|| kind ~ /^(CustomResourceDefinition|APIService|RuntimeClass|CSIDriver|CSINode|VolumeAttachment)$/ \
|| kind ~ /(WebhookConfiguration)$/
}
function check_document() {
if (kind == "") return
if (cluster_scoped(kind)) return
if (namespace == "") {
printf "FAIL: %s has no metadata.namespace\n", kind
failed = 1
} else if (namespace != ENVIRON["NAMESPACE"]) {
printf "FAIL: %s is in namespace %s, want %s\n", kind, namespace, ENVIRON["NAMESPACE"]
failed = 1
}
kind = ""
namespace = ""
}
/^---[[:space:]]*$/ { check_document(); next }
/^kind:[[:space:]]/ { kind = $2 }
/^metadata:[[:space:]]*$/ { in_metadata = 1; next }
/^[^[:space:]]/ { in_metadata = 0 }
in_metadata && /^ namespace:[[:space:]]/ { namespace = $2; in_metadata = 0 }
END { check_document(); exit failed }
' "$workdir/rendered.yaml"
echo "render without a namespace must fail"
if helm template "$RELEASE" "$CHART_DIR" \
--kubeconfig "$workdir/kubeconfig" \
--set users.existingSecret=test \
> "$workdir/without-namespace.yaml" 2>/dev/null; then
echo "FAIL: the chart rendered without --namespace" >&2
exit 1
fi
echo "render with allowDefaultNamespace=true"
helm template "$RELEASE" "$CHART_DIR" \
--kubeconfig "$workdir/kubeconfig" \
--set users.existingSecret=test \
--set allowDefaultNamespace=true >/dev/null
echo "render with deezer.arl"
helm template "$RELEASE" "$CHART_DIR" \
--namespace "$NAMESPACE" \
--set users.existingSecret=test \
--set deezer.arl=test-arl > "$workdir/arl.yaml"
grep -A1 'name: DEEZER_ARL' "$workdir/arl.yaml" | grep -q 'value: "test-arl"'
echo "deezer.arl and deezer.existingSecret are mutually exclusive"
if helm template "$RELEASE" "$CHART_DIR" \
--namespace "$NAMESPACE" \
--set users.existingSecret=test \
--set deezer.arl=test-arl \
--set deezer.existingSecret=test-secret >/dev/null 2>&1; then
echo "FAIL: the chart accepted both deezer.arl and deezer.existingSecret" >&2
exit 1
fi
echo "chart check passed"
+177
View File
@@ -0,0 +1,177 @@
#!/usr/bin/env bash
#
# Fetch and print client diagnostics stored by the server.
#
# The password is read from LOGS_PASSWORD or prompted, and is only used for the
# login request. The token is revoked before the script exits.
#
# Usage:
# scripts/logs.sh --server https://music.example.com --user alice
# scripts/logs.sh --server ... --user admin --target bob --level error
# scripts/logs.sh --server ... --user alice --since 7d --limit 500
# scripts/logs.sh --server ... --user alice --clear
set -euo pipefail
SERVER=""
USER_NAME=""
PASSWORD="${LOGS_PASSWORD:-}"
TARGET=""
LEVEL=""
DEVICE=""
LIMIT="200"
SINCE=""
CLEAR=0
RAW=0
usage() {
cat <<'EOF'
Usage: scripts/logs.sh --server URL --user NAME [options]
Options:
--password PW Password (otherwise LOGS_PASSWORD or a prompt)
--target NAME Read another user's logs (administrators only)
--level LEVEL debug, info, warn or error
--device ID Only this install id
--since VALUE Epoch milliseconds or a window such as 7d / 12h
--limit N Rows to fetch (1..1000, default 200)
--clear Delete the selected user's logs instead of printing them
--raw Print the JSON response unchanged
-h, --help Show this help
EOF
}
while [[ $# -gt 0 ]]; do
case "$1" in
--server) SERVER="${2:?--server needs a value}"; shift 2 ;;
--user) USER_NAME="${2:?--user needs a value}"; shift 2 ;;
--password) PASSWORD="${2:?--password needs a value}"; shift 2 ;;
--target) TARGET="${2:?--target needs a value}"; shift 2 ;;
--level) LEVEL="${2:?--level needs a value}"; shift 2 ;;
--device) DEVICE="${2:?--device needs a value}"; shift 2 ;;
--since) SINCE="${2:?--since needs a value}"; shift 2 ;;
--limit) LIMIT="${2:?--limit needs a value}"; shift 2 ;;
--clear) CLEAR=1; shift ;;
--raw) RAW=1; shift ;;
-h|--help) usage; exit 0 ;;
*) echo "Unknown argument: $1" >&2; usage >&2; exit 2 ;;
esac
done
if [[ -z "$SERVER" || -z "$USER_NAME" ]]; then
usage >&2
exit 2
fi
SERVER="${SERVER%/}"
if [[ -z "$PASSWORD" ]]; then
read -rsp "Password for $USER_NAME: " PASSWORD
echo
fi
since_millis() {
local value="$1"
if [[ "$value" =~ ^([0-9]+)([dh])$ ]]; then
python3 - "$value" <<'PY'
import re, sys, time
value = sys.argv[1]
amount = int(re.match(r"([0-9]+)", value).group(1))
unit = value[-1]
seconds = amount * (86400 if unit == "d" else 3600)
print(int((time.time() - seconds) * 1000))
PY
else
printf '%s' "$value"
fi
}
# Keep the password out of argv: it is piped to the JSON encoder as stdin.
login_body="$(printf '%s\n%s\n' "$USER_NAME" "$PASSWORD" | python3 -c '
import json, sys
user = sys.stdin.readline().rstrip("\n")
password = sys.stdin.readline().rstrip("\n")
print(json.dumps({
"username": user,
"password": password,
"deviceName": "diagnostics-cli",
"platform": "cli",
}))
')"
login_response="$(printf '%s' "$login_body" | curl -sS -w '\n%{http_code}' -X POST \
"$SERVER/api/v1/auth/login" \
-H 'Content-Type: application/json' \
--data-binary @-)"
login_status="$(printf '%s' "$login_response" | tail -1)"
login_body="$(printf '%s' "$login_response" | sed '$d')"
if [[ "$login_status" != "200" ]]; then
echo "Login failed (HTTP $login_status): $login_body" >&2
exit 1
fi
TOKEN="$(printf '%s' "$login_body" | python3 -c 'import json,sys; print(json.load(sys.stdin)["token"])')"
# The token stays in a 0600 curl config instead of the process list.
AUTH_CONFIG="$(mktemp "${TMPDIR:-/tmp}/audiomanager-logs.XXXXXX")"
PAYLOAD_FILE=""
chmod 600 "$AUTH_CONFIG"
printf 'header = "Authorization: Bearer %s"\n' "$TOKEN" > "$AUTH_CONFIG"
cleanup() {
curl -sS -o /dev/null -X POST "$SERVER/api/v1/auth/logout" \
-K "$AUTH_CONFIG" 2>/dev/null || true
rm -f "$AUTH_CONFIG" "$PAYLOAD_FILE"
}
trap cleanup EXIT
query=()
[[ -n "$LEVEL" ]] && query+=(--data-urlencode "level=$LEVEL")
[[ -n "$DEVICE" ]] && query+=(--data-urlencode "device=$DEVICE")
[[ -n "$TARGET" ]] && query+=(--data-urlencode "user=$TARGET")
[[ -n "$SINCE" ]] && query+=(--data-urlencode "since=$(since_millis "$SINCE")")
if [[ "$CLEAR" -eq 1 ]]; then
response="$(curl -sS -G -X DELETE "$SERVER/api/v1/logs" \
-K "$AUTH_CONFIG" "${query[@]}")"
printf '%s' "$response" | python3 -c 'import json,sys; print("deleted", json.load(sys.stdin)["deleted"], "entries")'
exit 0
fi
if [[ "$LIMIT" != "200" ]]; then
query+=(--data-urlencode "limit=$LIMIT")
fi
response="$(curl -sS -G "$SERVER/api/v1/logs" \
-K "$AUTH_CONFIG" "${query[@]}")"
PAYLOAD_FILE="$(mktemp "${TMPDIR:-/tmp}/audiomanager-logs-payload.XXXXXX")"
printf '%s' "$response" > "$PAYLOAD_FILE"
python3 - "$PAYLOAD_FILE" "$RAW" <<'PY'
import json
import sys
with open(sys.argv[1], encoding="utf-8") as handle:
data = json.load(handle)
if sys.argv[2] == "1":
print(json.dumps(data, indent=2))
raise SystemExit(0)
logs = data.get("logs", [])
if not logs:
print("No diagnostics stored for this user.")
raise SystemExit(0)
print(f"{len(logs)} entr{'y' if len(logs) == 1 else 'ies'} (newest first)")
print()
for row in logs:
when = row.get("createdAt") or row.get("receivedAt") or "?"
device = f"{row.get('platform', '?')} {row.get('appVersion', '?')} {row.get('deviceId', '?')}"
session = row.get("sessionId")
suffix = f" session={session}" if session else ""
print(f"{when} {row.get('level', '?').upper():5} {row.get('tag', '?')} [{device}]{suffix}")
print(f" {row.get('message', '')}")
detail = row.get("detail")
if detail:
for line in detail.splitlines():
print(f" | {line}")
print()
PY
+87
View File
@@ -99,6 +99,93 @@ fi
helm package "$CHART_DIR" --version "$VERSION" --destination "$workdir/repo"
helm repo index "$workdir/repo" --url "$CHARTS_RAW_URL"
# Keep the repository README pointing at the newest version of every chart so
# the published version is visible without reading index.yaml.
app_version="$(awk '/^appVersion:/ { print $2; exit }' "$CHART_DIR/Chart.yaml" | tr -d '"')"
source_sha="${GITHUB_SHA:-$(git -C "$ROOT" rev-parse HEAD)}"
source_repo="${GITHUB_REPOSITORY:-TapTap/AudioManager}"
# GITHUB_SERVER_URL is the internal Actions service URL, so derive the public
# host from the published raw URL instead.
source_server="${SOURCE_SERVER_URL:-$(printf '%s' "$CHARTS_RAW_URL" | sed -E 's#^(https?://[^/]+).*#\1#')}"
published_at="$(date -u '+%Y-%m-%d %H:%M UTC')"
newest_charts="$(
for archive in "$workdir/repo"/*.tgz; do
[[ -f "$archive" ]] || continue
helm show chart "$archive" 2>/dev/null \
| awk '
/^name:/ { name = $2 }
/^version:/ { version = $2 }
/^appVersion:/ { app = $2; gsub(/"/, "", app) }
END { if (name != "") printf "%s|%s|%s\n", name, version, app }
'
done | sort -t '|' -k1,1 -k2,2V \
| awk -F '|' '
{ if ($1 == last) { row = $0 } else { if (last != "") print row; last = $1; row = $0 } }
END { if (last != "") print row }
'
)"
section="$workdir/newest-chart.md"
{
echo '<!-- newest-chart:begin -->'
echo '## Newest chart'
echo
echo "\`$chart_name\` **$VERSION** (app \`$app_version\`), published $published_at"
echo "from commit [\`${source_sha:0:7}\`]($source_server/$source_repo/commit/$source_sha)."
echo
echo '| Chart | Newest version | App version |'
echo '| --- | --- | --- |'
while IFS='|' read -r name version app; do
[[ -n "$name" ]] || continue
echo "| \`$name\` | \`$version\` | ${app:--} |"
done <<< "$newest_charts"
echo '<!-- newest-chart:end -->'
} > "$section"
readme="$workdir/repo/README.md"
updated_readme="$workdir/repo/README.md.new"
if [[ -f "$readme" ]] && grep -q '<!-- newest-chart:begin -->' "$readme"; then
awk -v section="$section" '
/<!-- newest-chart:begin -->/ {
while ((getline line < section) > 0) print line
skipping = 1
next
}
/<!-- newest-chart:end -->/ { skipping = 0; next }
!skipping { print }
' "$readme" > "$updated_readme"
else
# No generated section yet: put it before the first "## " heading.
{
if [[ -f "$readme" ]]; then
awk -v section="$section" '
!inserted && /^## / {
while ((getline line < section) > 0) print line
print ""
inserted = 1
}
{ print }
END {
if (!inserted) {
print ""
while ((getline line < section) > 0) print line
}
}
' "$readme"
else
{
echo '# Charts'
echo
echo 'Published Helm chart repository for TapTap projects.'
echo
}
cat "$section"
fi
} > "$updated_readme"
fi
mv "$updated_readme" "$readme"
git -C "$workdir/repo" add .
if git -C "$workdir/repo" diff --cached --quiet; then
echo "Chart $(basename "$CHART_DIR") $VERSION is already published."
+347 -83
View File
@@ -1,19 +1,21 @@
#!/usr/bin/env bash
#
# End-to-end smoke test for the native mood-first API.
# End-to-end smoke test for the mood-first API.
#
# Generates a tiny tagged MP3 library, starts the server against it, and
# exercises auth, sync, moods, radio sessions, add-to-mood and streaming.
# Seeds a tiny media store, starts the server with fake `yt-dlp`/`rip` binaries
# and a fake Deezer API, and exercises both sources: auth, search, moods,
# downloads, recommendations, radio sessions, streaming and covers.
#
# Requirements: ffmpeg, sqlite3, curl, python3, java.
# Requirements: ffmpeg, sqlite3, curl, python3, java, bash.
# Usage: scripts/smoke.sh [work-dir]
#
set -euo pipefail
ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
GRADLE="${GRADLE:-./gradlew}"
WORK="${1:-$(mktemp -d "${TMPDIR:-/tmp}/musicmanager-smoke.XXXXXX")}"
WORK="${1:-$(mktemp -d "${TMPDIR:-/tmp}/audiomanager-smoke.XXXXXX")}"
PORT="${SMOKE_PORT:-18099}"
DZ_API_PORT="${SMOKE_DEEZER_PORT:-18098}"
USER="smoke"
PASS="smoke-pass"
BASE="http://127.0.0.1:${PORT}"
@@ -30,51 +32,211 @@ fail() { FAILED=$((FAILED + 1)); log "FAIL $1: $2"; }
json_field() { python3 -c 'import json,sys;d=json.load(sys.stdin);print(eval("d"+sys.argv[1]))' "$1"; }
mkdir -p "$WORK/music/Smoke Artist/Smoke Album" "$WORK/music/Second Artist/Second Album" \
"$WORK/longform/Long Narrator/Long Book" "$WORK/data"
MEDIA="$WORK/data/media"
mkdir -p "$MEDIA/vid-seed-1" "$MEDIA/vid-seed-2" "$WORK/bin" "$WORK/data"
log "Generated test library in $WORK"
log "Seeding the media store in $MEDIA"
ffmpeg -hide_banner -loglevel error -y -f lavfi -i "color=c=red:s=64x64" -frames:v 1 "$WORK/cover.jpg"
ffmpeg -hide_banner -loglevel error -y \
-f lavfi -i "sine=frequency=440:duration=2" \
-metadata title="Smoke Song" -metadata artist="Smoke Artist" \
-metadata album="Smoke Album" -metadata album_artist="Smoke Artist" \
-metadata genre="Rock" -metadata track="1" -metadata date="2026" \
-b:a 192k "$WORK/music/Smoke Artist/Smoke Album/01 - Smoke Song.mp3"
-f lavfi -i "sine=frequency=440:duration=2" -i "$WORK/cover.jpg" \
-map 0:a -map 1:v -c:v copy -c:a libmp3lame -b:a 192k -id3v2_version 3 \
-metadata title="Seed One" -metadata artist="Seed Artist" -metadata album="Seed Album" \
-metadata:s:v title="Album cover" -metadata:s:v comment="Cover (front)" \
"$MEDIA/vid-seed-1/Seed One.mp3"
ffmpeg -hide_banner -loglevel error -y \
-f lavfi -i "sine=frequency=550:duration=2" \
-metadata title="Second Song" -metadata artist="Second Artist" \
-metadata album="Second Album" -metadata album_artist="Second Artist" \
-metadata genre="Ambient" -metadata track="1" -metadata date="2026" \
-b:a 192k "$WORK/music/Second Artist/Second Album/01 - Second Song.mp3"
ffmpeg -hide_banner -loglevel error -y \
-f lavfi -i "sine=frequency=660:duration=2" \
-metadata title="Chapter One" -metadata artist="Long Narrator" \
-metadata album="Long Book" -metadata album_artist="Long Narrator" \
-metadata genre="Audiobook" -metadata track="1" -metadata date="2026" \
-b:a 128k "$WORK/longform/Long Narrator/Long Book/01 - Chapter One.mp3"
ffmpeg -hide_banner -loglevel error -y \
-f lavfi -i "color=c=red:s=64x64" -frames:v 1 \
"$WORK/music/Smoke Artist/Smoke Album/cover.jpg"
-c:a libmp3lame -b:a 192k -id3v2_version 3 \
-metadata title="Seed Two" -metadata artist="Second Artist" -metadata album="Second Album" \
"$MEDIA/vid-seed-2/Seed Two.mp3"
cat > "$WORK/library.yaml" <<EOF
library:
folders:
- id: 1
name: "Music"
type: music
path: $WORK/music
- id: 10
name: "Long-form"
type: longform
path: $WORK/longform
EOF
# Deterministic stand-in for yt-dlp: search, URL resolve and download.
cat > "$WORK/bin/yt-dlp" <<'FAKE'
#!/usr/bin/env bash
set -euo pipefail
out=""
prev=""
for arg in "$@"; do
if [[ "$prev" == "-o" ]]; then out="$arg"; fi
prev="$arg"
done
if [[ " $* " == *"RDAMVM"* ]]; then
# YouTube Music's watch-playlist radio: record that it was requested.
printf '%s\n' "$*" >> "$(dirname "$0")/../yt-radio.log"
cat <<'JSON'
{"entries":[
{"id":"vid-radio","title":"Radio Artist - Radio Track","duration":210,"channel":"Radio Artist"},
{"id":"vid-rec-a","title":"Downloaded Track","artist":"New Artist","album":"New Album","duration":180,"channel":"New Artist"}
]}
JSON
exit 0
fi
if [[ " $* " == *" music.youtube.com/search "* || " $* " == *" --flat-playlist "* ]]; then
cat <<'JSON'
{"entries":[
{"id":"vid-seed-1","title":"Seed One","artist":"Seed Artist","album":"Seed Album","duration":180,"channel":"Seed Artist"},
{"id":"vid-rec-a","title":"Downloaded Track","artist":"New Artist","album":"New Album","duration":180,"channel":"New Artist","thumbnail":"https://example/a.jpg"},
{"id":"vid-rec-b","title":"Another Track","artist":"Other Artist","album":"Other Album","duration":200,"channel":"Other Artist"},
{"id":"vid-long","title":"Long Set","artist":"DJ","duration":900,"channel":"DJ"},
{"id":"vid-short","title":"Interlude","artist":"DJ","duration":10,"channel":"DJ"}
]}
JSON
exit 0
fi
if [[ " $* " == *" --dump-single-json "* ]]; then
url="${*: -1}"
id="${url##*=}"; id="${id##*/}"
printf '{"id":"%s","title":"URL Artist - URL Track","channel":"URL Artist","duration":200}' "$id"
exit 0
fi
url="${*: -1}"
id="${url##*=}"
dir="$(dirname "$out")"
mkdir -p "$dir"
ffmpeg -hide_banner -loglevel error -y \
-f lavfi -i "sine=frequency=700:duration=1" \
-c:a libmp3lame -b:a 128k -id3v2_version 3 \
-metadata title="Downloaded Track" -metadata artist="New Artist" \
"$dir/$id.mp3"
printf '%s\n' "$dir/$id.mp3"
FAKE
chmod +x "$WORK/bin/yt-dlp"
# Deterministic stand-in for the Deezer public API.
cat > "$WORK/deezer_api.py" <<'PY'
import json
import sys
import urllib.parse
from http.server import BaseHTTPRequestHandler, HTTPServer
TRACKS = {
"67238732": {"id": 67238732, "title": "Instant Crush", "duration": 337,
"readable": True, "artist": {"name": "Daft Punk"},
"album": {"title": "Random Access Memories"}},
"66609426": {"id": 66609426, "title": "Get Lucky", "duration": 248,
"readable": True, "artist": {"name": "Daft Punk"},
"album": {"title": "Get Lucky"}},
"77777777": {"id": 77777777, "title": "Around the World", "duration": 430,
"readable": True, "artist": {"name": "Daft Punk"},
"album": {"title": "Homework"}},
"1": {"id": 1, "title": "Too Long", "duration": 900, "readable": True,
"artist": {"name": "DJ"}, "album": {"title": "Long"}},
}
# Only resolvable by URL, so adding it cannot race a recommendation download.
EXTRA_TRACKS = {
"55555555": {"id": 55555555, "title": "One More Time", "duration": 320,
"readable": True, "artist": {"name": "Daft Punk"},
"album": {"title": "Discovery"}},
}
# Artist graph: the mood holds Daft Punk, so discovery walks to a related act
# and to a deep cut of an album already in the collection.
ARTISTS = {
"27": {"id": 27, "name": "Daft Punk"},
"99": {"id": 99, "name": "Daft Punk Tribute"},
}
TOP_TRACKS = {
"27": [TRACKS["77777777"]],
"99": [{"id": 88888888, "title": "Robot Rock", "duration": 305,
"readable": True, "artist": {"name": "Daft Punk Tribute"},
"album": {"title": "Tribute"}}],
}
RELATED = {"27": [ARTISTS["99"]]}
ALBUMS = {"5": {"id": 5, "title": "Random Access Memories",
"artist": {"name": "Daft Punk"}}}
ALBUM_TRACKS = {
"5": [{"id": 99999999, "title": "Contact", "duration": 382,
"readable": True, "artist": {"name": "Daft Punk"},
"album": {"title": "Random Access Memories"}}],
}
STATS = {"artist_search": 0, "artist_top": 0, "artist_related": 0,
"album_search": 0, "album_tracks": 0}
class Handler(BaseHTTPRequestHandler):
def log_message(self, *args):
pass
def reply(self, body, status=200):
data = json.dumps(body).encode()
self.send_response(status)
self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(data)))
self.end_headers()
self.wfile.write(data)
def do_GET(self):
parsed = urllib.parse.urlparse(self.path)
parts = [p for p in parsed.path.split("/") if p]
if parsed.path == "/search/track":
self.reply({"data": list(TRACKS.values())})
elif parsed.path == "/search/artist":
STATS["artist_search"] += 1
self.reply({"data": list(ARTISTS.values())})
elif parsed.path == "/search/album":
STATS["album_search"] += 1
self.reply({"data": list(ALBUMS.values())})
elif parsed.path == "/stats":
self.reply(STATS)
elif len(parts) == 3 and parts[0] == "artist" and parts[2] == "top":
STATS["artist_top"] += 1
self.reply({"data": TOP_TRACKS.get(parts[1], [])})
elif len(parts) == 3 and parts[0] == "artist" and parts[2] == "related":
STATS["artist_related"] += 1
self.reply({"data": RELATED.get(parts[1], [])})
elif len(parts) == 3 and parts[0] == "album" and parts[2] == "tracks":
STATS["album_tracks"] += 1
self.reply({"data": ALBUM_TRACKS.get(parts[1], [])})
elif parsed.path.startswith("/track/"):
track = TRACKS.get(parts[-1]) or EXTRA_TRACKS.get(parts[-1])
if track is None:
self.reply({"error": {"type": "DataException", "message": "no track"}}, 404)
else:
self.reply(track)
else:
self.reply({"error": {"type": "DataException", "message": "not found"}}, 404)
HTTPServer(("127.0.0.1", int(sys.argv[1])), Handler).serve_forever()
PY
# Deterministic stand-in for streamrip. It ignores the ARL and writes one MP3
# per track, exactly like the real downloader from the server's perspective.
cat > "$WORK/bin/rip" <<'FAKE'
#!/usr/bin/env bash
set -euo pipefail
outdir=""
prev=""
for arg in "$@"; do
if [[ "$prev" == "-f" ]]; then outdir="$arg"; fi
prev="$arg"
done
url="${*: -1}"
id="${url##*/}"
mkdir -p "$outdir"
case "$id" in
67238732) title="Instant Crush"; artist="Daft Punk"; album="Random Access Memories" ;;
66609426) title="Get Lucky"; artist="Daft Punk"; album="Get Lucky" ;;
77777777) title="Around the World"; artist="Daft Punk"; album="Homework" ;;
55555555) title="One More Time"; artist="Daft Punk"; album="Discovery" ;;
*) title="Deezer Track $id"; artist="Deezer Artist"; album="Deezer Album" ;;
esac
ffmpeg -hide_banner -loglevel error -y \
-f lavfi -i "sine=frequency=800:duration=1" \
-c:a libmp3lame -b:a 128k -id3v2_version 3 \
-metadata title="$title" -metadata artist="$artist" -metadata album="$album" \
"$outdir/$title.mp3"
FAKE
chmod +x "$WORK/bin/rip"
cat > "$WORK/users.yaml" <<EOF
users:
- name: $USER
password: "$PASS"
admin: true
folders: [1]
EOF
log "Building fat JAR..."
@@ -86,32 +248,39 @@ if [[ -z "$JAR" ]]; then
exit 1
fi
DATA_DIR="$WORK/data" LIBRARY_CONFIG="$WORK/library.yaml" USERS_FILE="$WORK/users.yaml" PORT="$PORT" \
python3 "$WORK/deezer_api.py" "$DZ_API_PORT" > "$WORK/deezer_api.log" 2>&1 &
DZ_API_PID=$!
for _ in $(seq 1 30); do
curl -sf "http://127.0.0.1:${DZ_API_PORT}/track/67238732" >/dev/null 2>&1 && break
sleep 1
done
PATH="$WORK/bin:$PATH" DATA_DIR="$WORK/data" USERS_FILE="$WORK/users.yaml" PORT="$PORT" \
DEEZER_API_BASE="http://127.0.0.1:${DZ_API_PORT}" DEEZER_ARL="fake-arl" \
STREAMRIP_PATH="$WORK/bin/rip" DEFAULT_SOURCE="deezer" \
java -jar "$JAR" > "$WORK/server.log" 2>&1 &
SERVER_PID=$!
trap 'kill $SERVER_PID 2>/dev/null || true' EXIT
trap 'kill $SERVER_PID $DZ_API_PID 2>/dev/null || true' EXIT
log "Waiting for the server and initial scan..."
log "Waiting for the server and the startup scan..."
SONGS=0
for _ in $(seq 1 120); do
if curl -sf "$BASE/health" >/dev/null 2>&1; then
SONGS="$(sqlite3 "$WORK/data/musicmanager.db" 'select count(*) from songs' 2>/dev/null || echo 0)"
[[ "$SONGS" -ge 3 ]] && break
[[ "$SONGS" -ge 2 ]] && break
fi
sleep 1
done
if [[ "${SONGS:-0}" -lt 3 ]]; then
log "Server did not scan the test library; see $WORK/server.log"
if [[ "${SONGS:-0}" -lt 2 ]]; then
log "Server did not index the media store; see $WORK/server.log"
tail -20 "$WORK/server.log" || true
exit 1
fi
SONG_ID="$(sqlite3 "$WORK/data/musicmanager.db" "select id from songs where title='Smoke Song'")"
SONG2_ID="$(sqlite3 "$WORK/data/musicmanager.db" "select id from songs where title='Second Song'")"
LONG_ID="$(sqlite3 "$WORK/data/musicmanager.db" "select id from songs where title='Chapter One'")"
ALBUM_ID="$(sqlite3 "$WORK/data/musicmanager.db" "select album_id from songs where id='$SONG_ID'")"
log "Scanned: song=$SONG_ID second=$SONG2_ID longform=$LONG_ID"
SEED_ONE="$(sqlite3 "$WORK/data/musicmanager.db" "select id from songs where source='youtube' and external_id='vid-seed-1'")"
SEED_TWO="$(sqlite3 "$WORK/data/musicmanager.db" "select id from songs where source='youtube' and external_id='vid-seed-2'")"
log "Indexed: seed1=$SEED_ONE seed2=$SEED_TWO"
# api <method> <path> [json-body] [token-override]
api() {
@@ -146,6 +315,21 @@ expect_eq() {
fi
}
# wait_for <seconds> <name> <python-condition> <path>
wait_for() {
local seconds="$1" name="$2" condition="$3" path="$4" body=""
for _ in $(seq 1 "$seconds"); do
api GET "$path"
if [[ "$HTTP_STATUS" == "200" ]] \
&& printf '%s' "$HTTP_BODY" | python3 -c "import json,sys;d=json.load(sys.stdin);sys.exit(0 if ($condition) else 1)" 2>/dev/null; then
pass "$name"
return 0
fi
sleep 1
done
fail "$name" "timed out; last body: ${HTTP_BODY:0:240}"
}
log ""
log "Authentication"
LOGIN_STATUS="$(curl -s -o "$WORK/login.json" -w '%{http_code}' -X POST \
@@ -163,62 +347,119 @@ BAD_STATUS="$(curl -s -o /dev/null -w '%{http_code}' -X POST \
expect_eq "login rejects bad password" "401" "$BAD_STATUS"
UNAUTH_STATUS="$(curl -s -o /dev/null -w '%{http_code}' "$BASE/api/v1/moods")"
expect_eq "unauthenticated request rejected" "401" "$UNAUTH_STATUS"
expect "me" "200" "GET" "/me"
expect_eq "me name" "$USER" "$(printf '%s' "$HTTP_BODY" | json_field '["name"]')"
log ""
log "Library sync"
expect "library snapshot (music)" "200" "GET" "/library/songs?scope=music"
MUSIC_COUNT="$(printf '%s' "$HTTP_BODY" | json_field '["songs"].__len__()' 2>/dev/null || echo 0)"
expect_eq "music snapshot has 2 songs" "2" "$MUSIC_COUNT"
expect_eq "snapshot contains smoke song" "1" "$(printf '%s' "$HTTP_BODY" | python3 -c 'import json,sys;print(sum(1 for s in json.load(sys.stdin)["songs"] if s["id"]=="'"$SONG_ID"'"))')"
expect "library snapshot (longform)" "200" "GET" "/library/songs?scope=longform"
expect_eq "longform snapshot has 1 song" "1" "$(printf '%s' "$HTTP_BODY" | json_field '["songs"].__len__()')"
expect "search" "200" "GET" "/library/search?q=smoke&scope=music"
expect_eq "search finds one song" "1" "$(printf '%s' "$HTTP_BODY" | json_field '["songs"].__len__()')"
log "Search"
expect "youtube search" "200" "GET" "/search?q=anything&source=youtube"
expect_eq "search hides long and short tracks" "3" "$(printf '%s' "$HTTP_BODY" | json_field '["results"].__len__()')"
expect_eq "search marks the seeded video" "true" "$(printf '%s' "$HTTP_BODY" | python3 -c 'import json,sys;print(str(any(r["externalId"]=="vid-seed-1" and r["downloaded"] for r in json.load(sys.stdin)["results"])).lower())')"
expect "deezer search" "200" "GET" "/search?q=daft&source=deezer"
expect_eq "deezer search hides long tracks" "3" "$(printf '%s' "$HTTP_BODY" | json_field '["results"].__len__()')"
expect_eq "deezer search source" "deezer" "$(printf '%s' "$HTTP_BODY" | json_field '["results"][0]["source"]')"
expect "default search uses deezer" "200" "GET" "/search?q=daft"
expect_eq "default search source" "deezer" "$(printf '%s' "$HTTP_BODY" | json_field '["results"][0]["source"]')"
log ""
log "Moods"
expect "create mood" "201" "POST" "/moods" \
"{\"name\":\"Smoke Mood\",\"genres\":[\"Rock\"],\"targetSize\":10,\"seedSongIds\":[\"$SONG_ID\"]}"
'{"name":"Smoke Mood","source":"youtube","genres":["Rock"],"collectionPercent":50,"recommendationCount":2}'
MOOD_ID="$(printf '%s' "$HTTP_BODY" | json_field '["mood"]["id"]' 2>/dev/null || true)"
if [[ -n "$MOOD_ID" ]]; then pass "mood id"; else fail "mood id" "${HTTP_BODY:0:200}"; fi
expect_eq "mood seed count" "1" "$(printf '%s' "$HTTP_BODY" | json_field '["mood"]["seedCount"]')"
expect "empty mood cannot start a session" "400" "POST" "/moods/$MOOD_ID/sessions" '{"limit":5}'
expect_eq "empty mood error code" "empty_mood" "$(printf '%s' "$HTTP_BODY" | json_field '["error"]["code"]')"
expect "add an existing song" "200" "POST" "/moods/$MOOD_ID/songs" "{\"songId\":\"$SEED_ONE\"}"
expect_eq "song count" "1" "$(printf '%s' "$HTTP_BODY" | json_field '["detail"]["mood"]["songCount"]')"
expect "add a second song" "200" "POST" "/moods/$MOOD_ID/songs" "{\"songId\":\"$SEED_TWO\"}"
expect_eq "song count after second add" "2" "$(printf '%s' "$HTTP_BODY" | json_field '["detail"]["mood"]["songCount"]')"
expect "add by URL" "202" "POST" "/moods/$MOOD_ID/songs" '{"url":"https://youtu.be/vid-url"}'
wait_for 60 "URL download attaches to the mood" 'd["mood"]["songCount"] >= 3' "/moods/$MOOD_ID"
expect "list moods" "200" "GET" "/moods"
expect_eq "list contains mood" "1" "$(printf '%s' "$HTTP_BODY" | python3 -c 'import json,sys;print(sum(1 for m in json.load(sys.stdin)["moods"] if m["id"]=="'"$MOOD_ID"'"))')"
expect "get mood" "200" "GET" "/moods/$MOOD_ID"
expect "mood preview" "200" "GET" "/moods/$MOOD_ID/preview?limit=10"
expect_eq "preview is not empty" "yes" "$(printf '%s' "$HTTP_BODY" | python3 -c 'import json,sys;print("yes" if json.load(sys.stdin)["songs"] else "no")')"
expect "add seed" "200" "POST" "/moods/$MOOD_ID/seeds" "{\"songId\":\"$SONG2_ID\"}"
expect_eq "seed count after add" "2" "$(printf '%s' "$HTTP_BODY" | json_field '["mood"]["seedCount"]')"
expect "remove seed" "204" "DELETE" "/moods/$MOOD_ID/seeds/$SONG2_ID"
expect "get mood after seed removal" "200" "GET" "/moods/$MOOD_ID"
expect_eq "seed count after remove" "1" "$(printf '%s' "$HTTP_BODY" | json_field '["mood"]["seedCount"]')"
expect "remove a song" "204" "DELETE" "/moods/$MOOD_ID/songs/$SEED_TWO"
log ""
log "Deezer source"
expect "create Deezer mood" "201" "POST" "/moods" \
'{"name":"Deezer Mood","source":"deezer","query":"daft punk","recommendationCount":2}'
DZ_MOOD="$(printf '%s' "$HTTP_BODY" | json_field '["mood"]["id"]' 2>/dev/null || true)"
if [[ -n "$DZ_MOOD" ]]; then pass "Deezer mood id"; else fail "Deezer mood id" "${HTTP_BODY:0:200}"; fi
expect_eq "Deezer mood source" "deezer" "$(printf '%s' "$HTTP_BODY" | json_field '["mood"]["source"]')"
expect "add a Deezer track" "202" "POST" "/moods/$DZ_MOOD/songs" \
'{"source":"deezer","externalId":"67238732"}'
wait_for 90 "Deezer download attaches to the mood" 'd["mood"]["songCount"] >= 1' "/moods/$DZ_MOOD"
wait_for 90 "Deezer recommendations download" 'd["mood"]["recommendationReady"] >= 1' "/moods/$DZ_MOOD"
expect "add a Deezer URL" "202" "POST" "/moods/$DZ_MOOD/songs" \
'{"url":"https://www.deezer.com/track/55555555"}'
wait_for 90 "Deezer URL attaches to the mood" 'd["mood"]["songCount"] >= 2' "/moods/$DZ_MOOD"
# The collection changed, so the next top-up re-runs discovery against the
# artist graph instead of waiting for the refresh window.
DZ_STATS=""
for _ in $(seq 1 90); do
DZ_STATS="$(curl -sf "http://127.0.0.1:${DZ_API_PORT}/stats" || true)"
if [[ -n "$DZ_STATS" ]] && printf '%s' "$DZ_STATS" | python3 -c '
import json, sys
d = json.load(sys.stdin)
sys.exit(0 if d["artist_related"] > 0 and d["album_tracks"] > 0 else 1)
'; then
break
fi
sleep 1
done
expect_eq "Deezer discovery walks the artist graph" "true" \
"$(printf '%s' "$DZ_STATS" | python3 -c 'import json,sys;d=json.load(sys.stdin);print(str(d["artist_related"] > 0 and d["artist_top"] > 0).lower())')"
expect_eq "Deezer discovery reads album deep cuts" "true" \
"$(printf '%s' "$DZ_STATS" | python3 -c 'import json,sys;d=json.load(sys.stdin);print(str(d["album_tracks"] > 0).lower())')"
if [[ -f "$MEDIA/deezer-67238732/Instant Crush.mp3" ]]; then
pass "Deezer media path"
else
fail "Deezer media path" "missing $MEDIA/deezer-67238732/Instant Crush.mp3"
fi
expect "Deezer session" "201" "POST" "/moods/$DZ_MOOD/sessions" '{"limit":5}'
DZ_SESSION="$(printf '%s' "$HTTP_BODY" | json_field '["id"]' 2>/dev/null || true)"
if [[ -n "$DZ_SESSION" ]]; then pass "Deezer session id"; else fail "Deezer session id" "${HTTP_BODY:0:200}"; fi
log ""
log "Recommendations"
expect "refresh recommendations" "200" "POST" "/moods/$MOOD_ID/recommendations/refresh"
REC_VIDEO="$(printf '%s' "$HTTP_BODY" | python3 -c 'import json,sys;r=json.load(sys.stdin)["recommendations"];print(r[0]["externalId"] if r else "")')"
wait_for 90 "recommendations download" 'd["mood"]["recommendationReady"] >= 1' "/moods/$MOOD_ID"
if [[ -f "$WORK/yt-radio.log" ]]; then
pass "YouTube Music radio discovery"
else
fail "YouTube Music radio discovery" "no RDAMVM playlist was requested"
fi
expect "add a recommendation" "200" "POST" "/moods/$MOOD_ID/recommendations/$REC_VIDEO/add"
SECOND_REC="$(printf '%s' "$HTTP_BODY" | python3 -c 'import json,sys;r=json.load(sys.stdin)["detail"]["recommendations"];print(r[0]["externalId"] if r else "")')"
if [[ -n "$SECOND_REC" ]]; then
expect "reject a recommendation" "200" "POST" "/moods/$MOOD_ID/recommendations/$SECOND_REC/reject"
expect_eq "rejected state" "rejected" "$(printf '%s' "$HTTP_BODY" | json_field '["recommendation"]["state"]')"
else
fail "reject a recommendation" "no second recommendation was available"
fi
log ""
log "Radio sessions"
expect "start session" "201" "POST" "/moods/$MOOD_ID/sessions" '{"limit":10}'
SESSION_ID="$(printf '%s' "$HTTP_BODY" | json_field '["id"]' 2>/dev/null || true)"
if [[ -n "$SESSION_ID" ]]; then pass "session id"; else fail "session id" "${HTTP_BODY:0:200}"; fi
expect_eq "queue contains seed" "yes" "$(printf '%s' "$HTTP_BODY" | python3 -c 'import json,sys;print("yes" if any(s["id"]=="'"$SONG_ID"'" for s in json.load(sys.stdin)["queue"]) else "no")')"
expect_eq "queue excludes longform" "yes" "$(printf '%s' "$HTTP_BODY" | python3 -c 'import json,sys;print("yes" if all(s["id"]!="'"$LONG_ID"'" for s in json.load(sys.stdin)["queue"]) else "no")')"
expect_eq "queue contains a collection song" "yes" "$(printf '%s' "$HTTP_BODY" | python3 -c 'import json,sys;print("yes" if json.load(sys.stdin)["queue"] else "no")')"
expect "next tracks" "200" "POST" "/sessions/$SESSION_ID/next" '{"limit":10}'
expect "playback event" "200" "POST" "/sessions/$SESSION_ID/events" \
"{\"events\":[{\"songId\":\"$SONG_ID\",\"type\":\"completed\",\"positionSeconds\":2}]}"
"{\"events\":[{\"songId\":\"$SEED_ONE\",\"type\":\"completed\",\"positionSeconds\":2}]}"
expect_eq "accepted events" "1" "$(printf '%s' "$HTTP_BODY" | json_field '["accepted"]')"
expect "add playing track to current mood" "200" "POST" "/sessions/$SESSION_ID/add-to-mood" \
"{\"songId\":\"$SONG2_ID\"}"
expect_eq "seed count after add-to-mood" "2" "$(printf '%s' "$HTTP_BODY" | json_field '["mood"]["seedCount"]')"
expect "resume session" "200" "GET" "/sessions/$SESSION_ID"
expect_eq "session mood" "$MOOD_ID" "$(printf '%s' "$HTTP_BODY" | json_field '["moodId"]')"
expect "close session" "204" "DELETE" "/sessions/$SESSION_ID"
log ""
log "Media"
STREAM_STATUS="$(curl -s -o "$WORK/stream.mp3" -w '%{http_code}' \
"$BASE/api/v1/songs/$SONG_ID/stream?token=$TOKEN")"
"$BASE/api/v1/songs/$SEED_ONE/stream?token=$TOKEN")"
if [[ "$STREAM_STATUS" == "200" && -s "$WORK/stream.mp3" ]]; then
pass "stream with query token"
else
@@ -226,11 +467,11 @@ else
fi
RANGE_STATUS="$(curl -s -o /dev/null -w '%{http_code}' -H 'Range: bytes=0-9' \
"$BASE/api/v1/songs/$SONG_ID/stream?token=$TOKEN")"
"$BASE/api/v1/songs/$SEED_ONE/stream?token=$TOKEN")"
expect_eq "stream range" "206" "$RANGE_STATUS"
TRANSCODE_STATUS="$(curl -s -D "$WORK/transcode.headers" -o "$WORK/transcoded.mp3" -w '%{http_code}' \
"$BASE/api/v1/songs/$SONG_ID/stream?token=$TOKEN&maxBitRate=64")"
"$BASE/api/v1/songs/$SEED_ONE/stream?token=$TOKEN&maxBitRate=64")"
if [[ "$TRANSCODE_STATUS" == "200" && -s "$WORK/transcoded.mp3" ]] \
&& grep -qi '^content-type: audio/mpeg' "$WORK/transcode.headers"; then
pass "stream transcode"
@@ -239,16 +480,16 @@ else
fi
DOWNLOAD_STATUS="$(curl -s -o "$WORK/download.mp3" -w '%{http_code}' \
"$BASE/api/v1/songs/$SONG_ID/stream?token=$TOKEN&download=true")"
"$BASE/api/v1/songs/$SEED_ONE/stream?token=$TOKEN&download=true")"
if [[ "$DOWNLOAD_STATUS" == "200" && -s "$WORK/download.mp3" ]]; then
pass "download original"
else
fail "download original" "HTTP $DOWNLOAD_STATUS"
fi
COVER_STATUS="$(curl -s -o "$WORK/cover.jpg" -w '%{http_code}' \
"$BASE/api/v1/covers/$ALBUM_ID?size=32&token=$TOKEN")"
if [[ "$COVER_STATUS" == "200" && -s "$WORK/cover.jpg" ]]; then
COVER_STATUS="$(curl -s -o "$WORK/cover-out.jpg" -w '%{http_code}' \
"$BASE/api/v1/songs/$SEED_ONE/cover?size=32&token=$TOKEN")"
if [[ "$COVER_STATUS" == "200" && -s "$WORK/cover-out.jpg" ]]; then
pass "cover art"
else
fail "cover art" "HTTP $COVER_STATUS"
@@ -257,6 +498,29 @@ fi
expect "scan status" "200" "GET" "/scan"
expect "start scan" "202" "POST" "/scan"
log ""
log "Diagnostics"
expect "upload client logs" "200" "POST" "/logs" \
'{"deviceId":"smoke-dev","platform":"desktop","appVersion":"0.4.3","entries":[{"timestamp":0,"level":"error","tag":"smoke","message":"failed with Bearer smoke-secret","detail":"https://music/stream?token=abc123"}]}'
expect_eq "client log accepted" "1" "$(printf '%s' "$HTTP_BODY" | json_field '["accepted"]')"
expect "list client logs" "200" "GET" "/logs"
LOGS_RESULT="$(printf '%s' "$HTTP_BODY" | python3 -c '
import json, sys
logs = json.load(sys.stdin)["logs"]
text = json.dumps(logs)
ok = len(logs) == 1 and logs[0]["level"] == "error" and "smoke-secret" not in text and "abc123" not in text
print("ok" if ok else "bad")
')"
expect_eq "client logs are stored and redacted" "ok" "$LOGS_RESULT"
LOGS_OUTPUT="$(LOGS_PASSWORD="$PASS" "$ROOT/scripts/logs.sh" --server "$BASE" --user "$USER" --limit 5 2>&1)" || true
if printf '%s' "$LOGS_OUTPUT" | grep -q 'failed with Bearer \*\*\*'; then
pass "logs.sh prints redacted diagnostics"
else
fail "logs.sh prints redacted diagnostics" "${LOGS_OUTPUT:0:240}"
fi
expect "clear client logs" "200" "DELETE" "/logs"
expect_eq "client logs deleted" "1" "$(printf '%s' "$HTTP_BODY" | json_field '["deleted"]')"
log ""
log "Cleanup"
expect "delete mood" "204" "DELETE" "/moods/$MOOD_ID"
+2 -1
View File
@@ -5,6 +5,7 @@ pkgs.mkShell {
jdk21
gradle
ffmpeg
yt-dlp
sqlite
git
curl
@@ -17,7 +18,7 @@ pkgs.mkShell {
JAVA_HOME = pkgs.jdk21.home;
shellHook = ''
echo "MusicManager dev shell (JDK 21, Gradle, ffmpeg, sqlite, helm, kustomize)"
echo "AudioManager dev shell (JDK 21, Gradle, ffmpeg, yt-dlp, sqlite, helm, kustomize)"
java -version
echo ""
echo "Test: gradle test"
@@ -9,6 +9,8 @@ import io.ktor.server.application.install
import io.ktor.server.engine.embeddedServer
import io.ktor.server.netty.Netty
import io.ktor.server.plugins.autohead.AutoHeadResponse
import io.ktor.server.plugins.callid.CallId
import io.ktor.server.plugins.callid.callIdMdc
import io.ktor.server.plugins.calllogging.CallLogging
import io.ktor.server.plugins.contentnegotiation.ContentNegotiation
import io.ktor.server.plugins.cors.routing.CORS
@@ -19,52 +21,200 @@ import io.ktor.server.response.respond
import io.ktor.server.response.respondText
import io.ktor.server.routing.get
import io.ktor.server.routing.routing
import java.nio.file.Files
import java.util.UUID
import java.util.concurrent.TimeUnit
import kotlinx.coroutines.CoroutineScope
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.SupervisorJob
import kotlinx.coroutines.cancelAndJoin
import kotlinx.coroutines.delay
import kotlinx.coroutines.isActive
import kotlinx.coroutines.job
import kotlinx.coroutines.launch
import kotlinx.coroutines.runBlocking
import kotlinx.serialization.json.Json
import org.slf4j.LoggerFactory
import win.taptap.musicmanager.config.AppConfig
import win.taptap.musicmanager.config.LibraryConfig
import win.taptap.musicmanager.config.UsersConfig
import win.taptap.musicmanager.api.ApiErrorBody
import win.taptap.musicmanager.api.ApiError
import win.taptap.musicmanager.api.ApiErrorBody
import win.taptap.musicmanager.api.ApiException
import win.taptap.musicmanager.api.apiRoutes
import win.taptap.musicmanager.config.AppConfig
import win.taptap.musicmanager.config.UsersConfig
import win.taptap.musicmanager.db.ClientLogRepository
import win.taptap.musicmanager.db.ConfigRepository
import win.taptap.musicmanager.db.DatabaseFactory
import win.taptap.musicmanager.db.MediaRepository
import win.taptap.musicmanager.db.MoodRepository
import win.taptap.musicmanager.db.SongRepository
import win.taptap.musicmanager.discovery.DeezerSearch
import win.taptap.musicmanager.discovery.DiscoveryService
import win.taptap.musicmanager.discovery.MusicCatalogs
import win.taptap.musicmanager.discovery.TopUpService
import win.taptap.musicmanager.discovery.YtDlpSearch
import win.taptap.musicmanager.download.DeezerDownloader
import win.taptap.musicmanager.download.DownloadService
import win.taptap.musicmanager.download.SourceDownloaders
import win.taptap.musicmanager.download.YtDlpDownloader
import win.taptap.musicmanager.source.MusicSource
import win.taptap.musicmanager.media.LoudnessAnalyzer
import win.taptap.musicmanager.media.MediaGcService
import win.taptap.musicmanager.media.MediaStore
import win.taptap.musicmanager.radio.RadioService
import win.taptap.musicmanager.scanner.CoverArtService
import win.taptap.musicmanager.scanner.MediaScanner
import win.taptap.musicmanager.scanner.SongIndexer
import win.taptap.musicmanager.transcoding.FfmpegTranscoder
private val log = LoggerFactory.getLogger("win.taptap.musicmanager.Application")
fun main() {
val config = AppConfig.fromEnv()
val users = UsersConfig.load(config.usersFile)
val library = LibraryConfig.load(config.libraryConfigPath).also { it.validatePaths() }
val users = UsersConfig.load(config.usersFile).also { it.validateAgainst(library) }
Files.createDirectories(config.dataDir)
Files.createDirectories(config.mediaDir)
Files.createDirectories(config.downloadDir)
val db = DatabaseFactory.connect(config)
try {
ConfigRepository.sync(library, users)
ConfigRepository.sync(users)
val applicationScope = CoroutineScope(SupervisorJob() + Dispatchers.IO)
val coverArtService = CoverArtService(config.dataDir)
val mediaStore = MediaStore(config.mediaDir)
val indexer = SongIndexer(mediaStore, coverArtService)
val gc = MediaGcService(mediaStore, coverArtService)
val loudnessAnalyzer = if (config.loudnessEnabled) {
LoudnessAnalyzer(
ffmpegPath = config.ffmpegPath,
targetLufs = config.loudnessTargetLufs,
).takeIf { it.isAvailable() }
} else {
null
}
val transcoder = FfmpegTranscoder(config.ffmpegPath)
val radioService = RadioService()
val scanner = MediaScanner(
library = library,
coverArtService = coverArtService,
val catalogs = MusicCatalogs(
mapOf(
MusicSource.YOUTUBE to YtDlpSearch(config.ytDlpPath),
MusicSource.DEEZER to DeezerSearch(apiBase = config.deezerApiBase),
),
)
if (MediaRepository.countSongs() == 0L) {
log.info("Song table is empty, starting initial library scan")
scanner.start(applicationScope)
val discoveryService = DiscoveryService(catalogs)
val topUpService = TopUpService(discoveryService, applicationScope)
val scanner = MediaScanner(mediaStore, indexer)
val downloaders = SourceDownloaders(
mapOf(
MusicSource.YOUTUBE to YtDlpDownloader(config.ytDlpPath),
MusicSource.DEEZER to DeezerDownloader(
executable = config.streamripPath,
arl = config.deezerArl,
quality = config.deezerQuality,
),
),
)
val downloadService = DownloadService(
downloaders = downloaders,
mediaStore = mediaStore,
indexer = indexer,
gc = gc,
loudnessAnalyzer = loudnessAnalyzer,
tempDir = config.downloadDir,
onSongAttached = topUpService::trigger,
)
if (config.defaultSource == MusicSource.DEEZER && config.deezerArl == null) {
log.warn("DEFAULT_SOURCE is deezer but DEEZER_ARL is empty: downloads will fail")
}
// The media root is the only source of songs, so index it at startup.
log.info("Scanning media root {}", config.mediaDir)
scanner.start(applicationScope)
applicationScope.launch(Dispatchers.IO) {
delay(TimeUnit.SECONDS.toMillis(5))
while (isActive) {
val worked = runCatching { downloadService.processOne() }
.onFailure { log.warn("Download worker error", it) }
.getOrDefault(false)
delay(if (worked) 1_000 else 15_000)
}
}
// Refill recommendation slots and run due discovery passes.
applicationScope.launch(Dispatchers.IO) {
delay(TimeUnit.MINUTES.toMillis(1))
while (isActive) {
runCatching {
MoodRepository.all()
.filter { it.recommendationCount > 0 }
.forEach { mood ->
topUpService.topUp(mood, force = false)
}
}.onFailure { log.warn("Background recommendation top-up failed", it) }
delay(TimeUnit.HOURS.toMillis(1))
}
}
// Normalize the existing library gradually; one file at a time keeps
// download and streaming work responsive.
if (loudnessAnalyzer != null) {
applicationScope.launch(Dispatchers.IO) {
delay(TimeUnit.MINUTES.toMillis(2))
while (isActive) {
val pending = runCatching { SongRepository.pendingLoudness(1) }
.onFailure { log.warn("Loudness backfill query failed", it) }
.getOrDefault(emptyList())
val song = pending.firstOrNull()
if (song == null) {
delay(TimeUnit.MINUTES.toMillis(30))
continue
}
val file = mediaStore.file(song)
if (file == null || !Files.isRegularFile(file)) {
// Keep the row out of the retry loop; a later upsert
// from the scanner will reset it if the file returns.
SongRepository.updateLoudness(song.id, -70.0, 0.0)
} else {
val updated = loudnessAnalyzer.analyzeAndStore(song, file)
if (updated.gainDb == null) {
// Unreadable/corrupt file: record a neutral gain so
// the backfill does not retry it every second.
log.warn("Could not analyze {}; leaving it unnormalized", file)
SongRepository.updateLoudness(song.id, -70.0, 0.0)
}
}
delay(1_000)
}
}
}
// Client diagnostics are kept for a short window so a bug can be
// investigated without letting the table grow without bound.
applicationScope.launch(Dispatchers.IO) {
delay(TimeUnit.MINUTES.toMillis(1))
while (isActive) {
runCatching {
ClientLogRepository.prune(
config.clientLogRetentionDays,
config.clientLogMaxRowsPerUser,
)
}
.onSuccess { count ->
if (count > 0) log.info("Pruned {} client diagnostics rows", count)
}
.onFailure { log.warn("Client diagnostics prune failed", it) }
delay(TimeUnit.HOURS.toMillis(6))
}
}
// Delete files that no mood, recommendation or open session references.
applicationScope.launch(Dispatchers.IO) {
delay(TimeUnit.MINUTES.toMillis(10))
while (isActive) {
runCatching { gc.collectGarbage() }
.onFailure { log.warn("Media GC failed", it) }
delay(TimeUnit.HOURS.toMillis(1))
}
}
val server = embeddedServer(Netty, port = config.port, host = "0.0.0.0") {
@@ -94,7 +244,15 @@ fun main() {
}
install(PartialContent)
install(AutoHeadResponse)
install(CallLogging)
// A short id on every response makes server log lines and client
// diagnostics meet in the middle.
install(CallId) {
generate { UUID.randomUUID().toString().take(8) }
replyToHeader("X-Request-Id")
}
install(CallLogging) {
callIdMdc("requestId")
}
install(StatusPages) {
exception<ApiException> { call, cause ->
call.respond(
@@ -132,10 +290,19 @@ fun main() {
}
apiRoutes(
users = users,
library = library,
coverArtService = coverArtService,
transcoder = transcoder,
radioService = radioService,
discoveryService = discoveryService,
topUpService = topUpService,
catalogs = catalogs,
mediaStore = mediaStore,
gc = gc,
defaultSource = config.defaultSource,
deezerQuality = config.deezerQuality,
loudnessEnabled = config.loudnessEnabled && loudnessAnalyzer != null,
loudnessTargetLufs = config.loudnessTargetLufs,
clientLogRetentionDays = config.clientLogRetentionDays,
startScan = { scanner.start(applicationScope) },
scanStatus = { scanner.status() },
)
@@ -152,10 +319,10 @@ fun main() {
)
log.info(
"Starting MusicManager on port {} with {} library folders and {} users",
"Starting AudioManager on port {} with {} users and media root {}",
config.port,
library.folders.size,
users.users.size,
config.mediaDir,
)
server.start(wait = true)
} catch (t: Throwable) {
@@ -2,7 +2,8 @@ package win.taptap.musicmanager.api
import kotlinx.serialization.Serializable
import win.taptap.musicmanager.db.MoodRecord
import win.taptap.musicmanager.db.MoodSeedRecord
import win.taptap.musicmanager.db.MoodSongRecord
import win.taptap.musicmanager.db.RecommendationRecord
import win.taptap.musicmanager.db.SongRecord
@Serializable
@@ -23,8 +24,6 @@ data class LoginRequest(
data class UserDto(
val name: String,
val admin: Boolean,
val musicFolders: List<Int>,
val longformFolders: List<Int>,
)
@Serializable
@@ -45,52 +44,105 @@ data class DeviceDto(
val current: Boolean,
)
@Serializable
data class ClientLogEntryDto(
/** Device clock in epoch milliseconds; informational, the server clamps it. */
val timestamp: Long = 0,
val level: String,
val tag: String,
val message: String,
val detail: String? = null,
val sessionId: String? = null,
)
@Serializable
data class ClientLogBatchDto(
val deviceId: String = "unknown",
val platform: String = "unknown",
val appVersion: String = "unknown",
val entries: List<ClientLogEntryDto> = emptyList(),
)
@Serializable
data class ClientLogDto(
val id: Long,
val deviceId: String,
val platform: String,
val appVersion: String,
val sessionId: String? = null,
val level: String,
val tag: String,
val message: String,
val detail: String? = null,
val createdAt: String,
val receivedAt: String,
)
@Serializable
data class ClientLogsDto(val logs: List<ClientLogDto>)
@Serializable
data class ClientLogsAcceptedDto(val accepted: Int)
@Serializable
data class ClientLogsDeletedDto(val deleted: Int)
@Serializable
data class ServerInfoDto(
val defaultSource: String,
/** 0 = 128 kbps MP3, 1 = 320 kbps MP3 for Deezer downloads. */
val deezerQuality: Int,
val loudnessEnabled: Boolean,
val loudnessTargetLufs: Double,
/** How long the server keeps uploaded client diagnostics. */
val clientLogRetentionDays: Int,
)
@Serializable
data class SongDto(
val id: String,
val source: String,
val externalId: String,
val title: String,
val artist: String,
val album: String,
val albumId: String,
val trackNumber: Int? = null,
val discNumber: Int = 1,
val year: Int? = null,
val album: String? = null,
val duration: Int,
val bitRate: Int,
val size: Long,
val suffix: String,
val genres: List<String> = emptyList(),
val folderId: Int,
val relativePath: String,
val updatedAt: String,
val thumbnail: String? = null,
/** Measured integrated loudness; null when the server has not analyzed it yet. */
val loudnessLufs: Double? = null,
/** Gain in dB the client applies to reach the server's target loudness. */
val gainDb: Double? = null,
)
fun SongRecord.toDto(): SongDto = SongDto(
id = id,
source = source,
externalId = externalId,
title = title,
artist = displayArtist,
album = albumName,
albumId = albumId,
trackNumber = trackNumber,
discNumber = discNumber,
year = year,
artist = artist,
album = album,
duration = duration,
bitRate = bitRate,
size = size,
suffix = suffix,
genres = genres,
folderId = folderId,
relativePath = relativePath,
updatedAt = updatedAt,
thumbnail = thumbnail,
loudnessLufs = loudnessLufs,
gainDb = gainDb,
)
@Serializable
data class SeedDto(
data class MoodSongDto(
val song: SongDto,
val origin: String,
val addedAt: String,
)
fun MoodSongRecord.toDto(song: SongRecord): MoodSongDto =
MoodSongDto(song = song.toDto(), origin = origin, addedAt = addedAt)
@Serializable
data class MoodDto(
val id: String,
@@ -98,15 +150,19 @@ data class MoodDto(
val description: String? = null,
val query: String? = null,
val genres: List<String> = emptyList(),
val targetSize: Int,
val source: String,
val collectionPercent: Int,
val recommendationPercent: Int,
val recommendationCount: Int,
val refreshHours: Int,
val ytEnabled: Boolean,
val lastDiscoveryAt: String? = null,
val createdAt: String,
val changedAt: String,
val lastError: String? = null,
val seedCount: Int,
val newRecommendations: Int,
val songCount: Int,
val recommendationReady: Int,
val recommendationDownloading: Int,
val recommendationFailed: Int,
)
fun MoodRecord.toDto(): MoodDto = MoodDto(
@@ -115,34 +171,111 @@ fun MoodRecord.toDto(): MoodDto = MoodDto(
description = description,
query = query,
genres = genres,
targetSize = targetSize,
source = source,
collectionPercent = collectionPercent,
recommendationPercent = 100 - collectionPercent,
recommendationCount = recommendationCount,
refreshHours = refreshHours,
ytEnabled = ytEnabled,
lastDiscoveryAt = lastDiscoveryAt,
createdAt = createdAt,
changedAt = changedAt,
lastError = lastError,
seedCount = seedCount,
newRecommendations = newRecommendations,
songCount = songCount,
recommendationReady = recommendationReady,
recommendationDownloading = recommendationDownloading,
recommendationFailed = recommendationFailed,
)
@Serializable
data class MoodDetailDto(
val mood: MoodDto,
val seeds: List<SeedDto>,
val songs: List<MoodSongDto> = emptyList(),
val recommendations: List<RecommendationDto> = emptyList(),
/** In-flight and recent downloads, so the client can show a per-track status. */
val downloads: List<DownloadStatusDto> = emptyList(),
)
@Serializable
data class DownloadStatusDto(
val source: String,
val externalId: String,
val state: String,
val attempts: Int = 0,
val error: String? = null,
val songId: String? = null,
)
@Serializable
data class RecommendationDto(
val source: String,
val externalId: String,
val title: String,
val artist: String,
val album: String? = null,
val duration: Int? = null,
val thumbnail: String? = null,
val state: String,
val song: SongDto? = null,
val download: DownloadDto? = null,
val error: String? = null,
val firstSeen: String,
val lastSeen: String,
)
fun RecommendationRecord.toDto(
song: SongRecord? = null,
download: DownloadDto? = null,
): RecommendationDto = RecommendationDto(
source = source,
externalId = externalId,
title = title,
artist = artist,
album = album,
duration = duration,
thumbnail = thumbnail,
state = state,
song = song?.toDto(),
download = download,
error = error,
firstSeen = firstSeen,
lastSeen = lastSeen,
)
@Serializable
data class DownloadDto(
val state: String,
val attempts: Int,
val error: String? = null,
val updatedAt: String,
)
@Serializable
data class SearchResultDto(
val source: String,
val externalId: String,
val title: String,
val artist: String,
val album: String? = null,
val duration: Int? = null,
val thumbnail: String? = null,
val songId: String? = null,
val downloaded: Boolean = false,
val inMood: Boolean = false,
)
@Serializable
data class SearchResponse(val results: List<SearchResultDto> = emptyList())
@Serializable
data class CreateMoodRequest(
val name: String,
val description: String? = null,
val query: String? = null,
val genres: List<String> = emptyList(),
val targetSize: Int = 30,
val source: String? = null,
val collectionPercent: Int = 50,
val recommendationCount: Int = 10,
val refreshHours: Int = 24,
val ytEnabled: Boolean = true,
val seedSongIds: List<String> = emptyList(),
)
@Serializable
@@ -153,13 +286,38 @@ data class UpdateMoodRequest(
val query: String? = null,
val clearQuery: Boolean = false,
val genres: List<String>? = null,
val targetSize: Int? = null,
val source: String? = null,
val collectionPercent: Int? = null,
val recommendationCount: Int? = null,
val refreshHours: Int? = null,
val ytEnabled: Boolean? = null,
)
/**
* Adding music accepts one of:
* - `songId`: an already downloaded song, attached instantly;
* - `videoId`: a YouTube id from search, downloaded if necessary;
* - `url`: a pasted watch URL resolved by yt-dlp.
*/
@Serializable
data class AddSongRequest(
val songId: String? = null,
val source: String? = null,
val externalId: String? = null,
val url: String? = null,
)
@Serializable
data class AddSeedRequest(val songId: String, val origin: String = "manual")
data class AddSongResponse(
val detail: MoodDetailDto,
val added: Boolean,
val downloading: Boolean = false,
)
@Serializable
data class RecommendationDecisionDto(
val detail: MoodDetailDto,
val recommendation: RecommendationDto,
)
@Serializable
data class AddToMoodRequest(val songId: String, val moodId: String? = null)
@@ -171,16 +329,7 @@ data class AddToMoodResponse(
)
@Serializable
data class LibraryPage(
val songs: List<SongDto>,
val deletedSongIds: List<String> = emptyList(),
val revision: String,
val nextCursor: String? = null,
val nextSince: String? = null,
)
@Serializable
data class SearchResponse(val songs: List<SongDto>)
data class TracksResponse(val songs: List<SongDto>)
@Serializable
data class StartSessionRequest(val limit: Int? = null)
@@ -220,21 +369,3 @@ data class ScanStatusDto(
val count: Long,
val error: String? = null,
)
@Serializable
data class RecommendationDto(
val videoId: String,
val title: String,
val artist: String,
val album: String? = null,
val duration: Int? = null,
val thumbnail: String? = null,
val state: String,
val matchedSong: SongDto? = null,
val timesAdded: Int,
val firstSeen: String,
val lastSeen: String,
)
fun seedDto(seed: MoodSeedRecord, song: SongRecord): SeedDto =
SeedDto(song = song.toDto(), origin = seed.origin, addedAt = seed.addedAt)
@@ -8,14 +8,13 @@ import io.ktor.server.routing.get
import io.ktor.server.routing.post
import io.ktor.server.routing.route
import java.security.MessageDigest
import win.taptap.musicmanager.api.DeviceDto
import win.taptap.musicmanager.api.LoginRequest
import win.taptap.musicmanager.api.LoginResponse
import win.taptap.musicmanager.api.ScanStatusDto
import win.taptap.musicmanager.api.UserDto
import win.taptap.musicmanager.config.LibraryConfig
import win.taptap.musicmanager.config.UsersConfig
import win.taptap.musicmanager.db.ApiTokenRepository
import win.taptap.musicmanager.discovery.DiscoveryService
import win.taptap.musicmanager.discovery.MusicCatalogs
import win.taptap.musicmanager.discovery.TopUpService
import win.taptap.musicmanager.media.MediaGcService
import win.taptap.musicmanager.media.MediaStore
import win.taptap.musicmanager.radio.RadioService
import win.taptap.musicmanager.scanner.CoverArtService
import win.taptap.musicmanager.scanner.ScanStatus
@@ -29,10 +28,19 @@ import win.taptap.musicmanager.transcoding.FfmpegTranscoder
*/
fun Route.apiRoutes(
users: UsersConfig,
library: LibraryConfig,
coverArtService: CoverArtService,
transcoder: FfmpegTranscoder,
radioService: RadioService,
discoveryService: DiscoveryService,
topUpService: TopUpService,
catalogs: MusicCatalogs,
mediaStore: MediaStore,
gc: MediaGcService,
defaultSource: String,
deezerQuality: Int,
loudnessEnabled: Boolean,
loudnessTargetLufs: Double,
clientLogRetentionDays: Int,
startScan: () -> Boolean,
scanStatus: () -> ScanStatus,
) {
@@ -63,23 +71,32 @@ fun Route.apiRoutes(
LoginResponse(
token = issued.raw,
expiresAt = issued.record.expiresAt,
user = user.toDto(library),
user = user.toDto(),
),
)
}
get("/me") {
val user = call.requireApiUser(users)
call.respond(UserDto(
name = user.name,
admin = user.admin,
musicFolders = user.musicFolderIds(library).sorted(),
longformFolders = user.longformFolderIds(library).sorted(),
))
call.respond(user.toDto())
}
/** Source and audio settings the client cannot infer from a song. */
get("/info") {
call.requireApiUser(users)
call.respond(
ServerInfoDto(
defaultSource = defaultSource,
deezerQuality = deezerQuality,
loudnessEnabled = loudnessEnabled,
loudnessTargetLufs = loudnessTargetLufs,
clientLogRetentionDays = clientLogRetentionDays,
),
)
}
post("/auth/logout") {
val user = call.requireApiUser(users)
call.requireApiUser(users)
ApiTokenRepository.revokeRaw(call.bearerToken().orEmpty())
call.respond(HttpStatusCode.NoContent)
}
@@ -124,9 +141,19 @@ fun Route.apiRoutes(
)
}
libraryRoutes(users, library, coverArtService, transcoder)
moodRoutes(users, library, radioService)
sessionRoutes(users, library, radioService)
songRoutes(users, catalogs, mediaStore, coverArtService, transcoder, defaultSource)
moodRoutes(
users = users,
radioService = radioService,
discoveryService = discoveryService,
topUpService = topUpService,
catalogs = catalogs,
mediaStore = mediaStore,
gc = gc,
defaultSource = defaultSource,
)
sessionRoutes(users, radioService, gc)
logRoutes(users)
}
}
@@ -3,15 +3,11 @@ package win.taptap.musicmanager.api
import io.ktor.http.HttpHeaders
import io.ktor.http.HttpStatusCode
import io.ktor.server.application.ApplicationCall
import io.ktor.server.plugins.BadRequestException
import io.ktor.server.plugins.ContentTransformationException
import io.ktor.server.request.receive
import io.ktor.server.response.respond
import io.ktor.util.AttributeKey
import kotlinx.serialization.SerializationException
import win.taptap.musicmanager.api.ApiError
import win.taptap.musicmanager.api.ApiErrorBody
import win.taptap.musicmanager.api.UserDto
import win.taptap.musicmanager.config.FolderType
import win.taptap.musicmanager.config.LibraryConfig
import win.taptap.musicmanager.config.UserAccount
import win.taptap.musicmanager.config.UsersConfig
import win.taptap.musicmanager.db.ApiTokenRecord
@@ -27,26 +23,7 @@ class ApiException(
val ApiUserKey = AttributeKey<UserAccount>("musicmanager.api.user")
val ApiTokenKey = AttributeKey<ApiTokenRecord>("musicmanager.api.token")
fun UserAccount.musicFolderIds(library: LibraryConfig): Set<Int> =
library.folders
.asSequence()
.filter { it.type == FolderType.MUSIC && it.id in folders }
.map { it.id }
.toSet()
fun UserAccount.longformFolderIds(library: LibraryConfig): Set<Int> =
library.folders
.asSequence()
.filter { it.type == FolderType.LONGFORM }
.map { it.id }
.toSet()
fun UserAccount.toDto(library: LibraryConfig): UserDto = UserDto(
name = name,
admin = admin,
musicFolders = musicFolderIds(library).sorted(),
longformFolders = longformFolderIds(library).sorted(),
)
fun UserAccount.toDto(): UserDto = UserDto(name = name, admin = admin)
fun ApplicationCall.apiUser(): UserAccount =
attributes.getOrNull(ApiUserKey) ?: throw ApiException(
@@ -74,8 +51,7 @@ fun ApplicationCall.bearerToken(): String? {
}
/**
* Resolves the token and stores the user on the call. Returns null after sending
* a 401, so handlers can simply `?: return@get`.
* Resolves the token and stores the user on the call.
*/
suspend fun ApplicationCall.requireApiUser(users: UsersConfig): UserAccount {
val raw = bearerToken()
@@ -94,6 +70,10 @@ suspend inline fun <reified T : Any> ApplicationCall.receiveJson(): T =
receive<T>()
} catch (e: SerializationException) {
throw ApiException(HttpStatusCode.BadRequest, "invalid_body", "Invalid JSON body")
} catch (e: ContentTransformationException) {
throw ApiException(HttpStatusCode.BadRequest, "invalid_body", "Invalid JSON body")
} catch (e: BadRequestException) {
throw ApiException(HttpStatusCode.BadRequest, "invalid_body", "Invalid JSON body")
} catch (e: IllegalArgumentException) {
throw ApiException(HttpStatusCode.BadRequest, "invalid_body", e.message ?: "Invalid body")
}
@@ -1,188 +0,0 @@
package win.taptap.musicmanager.api
import io.ktor.http.ContentType
import io.ktor.http.HttpHeaders
import io.ktor.http.HttpStatusCode
import io.ktor.server.response.respond
import io.ktor.server.response.header
import io.ktor.server.response.respondFile
import io.ktor.server.response.respondOutputStream
import io.ktor.server.routing.Route
import io.ktor.server.routing.get
import java.net.URLEncoder
import java.time.Instant
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.withContext
import org.slf4j.LoggerFactory
import win.taptap.musicmanager.api.LibraryPage
import win.taptap.musicmanager.api.SearchResponse
import win.taptap.musicmanager.api.toDto
import win.taptap.musicmanager.config.LibraryConfig
import win.taptap.musicmanager.config.UsersConfig
import win.taptap.musicmanager.db.LibraryRepository
import win.taptap.musicmanager.scanner.CoverArtService
import win.taptap.musicmanager.transcoding.FfmpegTranscoder
private val log = LoggerFactory.getLogger("win.taptap.musicmanager.api.LibraryRoutes")
fun Route.libraryRoutes(
users: UsersConfig,
library: LibraryConfig,
coverArtService: CoverArtService,
transcoder: FfmpegTranscoder,
) {
/**
* Snapshot sync (`cursor`) and delta sync (`since=revision`) in one endpoint.
* Deletions are returned explicitly so offline libraries can drop them.
*/
get("/library/songs") {
val user = call.requireApiUser(users)
val limit = (call.request.queryParameters["limit"]?.toIntOrNull() ?: 500).coerceIn(1, 1000)
val scope = call.request.queryParameters["scope"] ?: "all"
val folders = user.scopeFolders(library, scope)
val now = Instant.now().toString()
val since = call.request.queryParameters["since"]?.takeIf { it.isNotBlank() }
val cursor = call.request.queryParameters["cursor"]?.takeIf { it.isNotBlank() }
if (since != null) {
val changed = LibraryRepository.changedSongs(since, folders, limit, user.name)
val deleted = LibraryRepository.tombstones(since, limit).map { it.entityId }
val complete = changed.size < limit && deleted.size < limit
call.respond(
LibraryPage(
songs = changed.map { it.toDto() },
deletedSongIds = deleted,
revision = if (complete) now else changed.lastOrNull()?.updatedAt ?: now,
nextSince = if (changed.size == limit) changed.last().updatedAt else null,
),
)
return@get
}
val songs = LibraryRepository.syncSongs(folders, cursor.orEmpty(), limit, user.name)
call.respond(
LibraryPage(
songs = songs.map { it.toDto() },
revision = now,
nextCursor = if (songs.size == limit) songs.last().id else null,
),
)
}
get("/library/search") {
val user = call.requireApiUser(users)
val query = call.request.queryParameters["q"]?.trim().orEmpty()
if (query.isEmpty()) {
call.respond(SearchResponse(emptyList()))
return@get
}
val limit = (call.request.queryParameters["limit"]?.toIntOrNull() ?: 50).coerceIn(1, 200)
val folders = user.scopeFolders(library, call.request.queryParameters["scope"] ?: "all")
val songs = LibraryRepository.searchSongs(query, limit, 0, folders, user.name)
call.respond(SearchResponse(songs.map { it.toDto() }))
}
get("/songs/{id}/stream") {
val user = call.requireApiUser(users)
val id = call.parameters["id"] ?: badRequest("Song id is required")
val parameters = call.request.queryParameters
val rawMaxBitRate = parameters["maxBitRate"]
val maxBitRate = rawMaxBitRate?.toIntOrNull()
if (rawMaxBitRate != null && (maxBitRate == null || maxBitRate < 0)) {
badRequest("Invalid maxBitRate")
}
val rawOffset = parameters["timeOffset"]
val offset = rawOffset?.toIntOrNull()
if (rawOffset != null && (offset == null || offset < 0)) {
badRequest("Invalid timeOffset")
}
val resolved = LibraryRepository.resolveSong(
id,
user.visibleFolderIds(library),
user.name,
library,
) ?: notFound("Song")
if (!transcoder.shouldTranscode(
resolved.song.bitRate,
maxBitRate,
parameters["format"],
offset ?: 0,
)
) {
call.response.header(HttpHeaders.ContentType, resolved.song.contentType)
if (parameters["download"] == "true") {
call.response.header(
HttpHeaders.ContentDisposition,
contentDisposition(resolved.file.fileName.toString()),
)
}
call.respondFile(resolved.file.toFile())
return@get
}
val process = try {
transcoder.start(resolved.file, maxBitRate, offset ?: 0)
} catch (e: Exception) {
log.error("Failed to start ffmpeg for {}", resolved.file, e)
throw ApiException(HttpStatusCode.InternalServerError, "transcode_failed", "Transcoding failed")
}
try {
call.response.headers.append(HttpHeaders.CacheControl, "no-store")
call.respondOutputStream(ContentType.parse("audio/mpeg")) {
val output = this
withContext(Dispatchers.IO) {
process.inputStream.use { input -> input.copyTo(output) }
}
}
} finally {
transcoder.stop(process)
}
}
get("/covers/{id}") {
val user = call.requireApiUser(users)
val id = call.parameters["id"] ?: badRequest("Cover id is required")
val rawSize = call.request.queryParameters["size"]
val size = rawSize?.toIntOrNull()
if (rawSize != null && (size == null || size !in 1..2048)) {
badRequest("Cover size must be between 1 and 2048")
}
val folders = user.visibleFolderIds(library)
val albumId = when {
id.startsWith("sg-") -> LibraryRepository.albumIdForSong(id, folders)
LibraryRepository.album(id, folders, user.name) != null -> id
else -> null
} ?: notFound("Cover")
val cover = if (size == null) {
coverArtService.coverFile(albumId)
} else {
coverArtService.resizedCover(albumId, size)
} ?: notFound("Cover")
call.respondFile(cover.toFile())
}
}
private fun win.taptap.musicmanager.config.UserAccount.scopeFolders(
library: LibraryConfig,
scope: String,
): Set<Int> = when (scope) {
"music" -> musicFolderIds(library)
"longform" -> longformFolderIds(library)
else -> visibleFolderIds(library)
}
private fun contentDisposition(fileName: String): String {
val encoded = URLEncoder.encode(fileName, Charsets.UTF_8).replace("+", "%20")
val ascii = fileName.map { character ->
if (character.code in 32..126 && character != '"' && character != '\\') character else '_'
}.joinToString("")
return "attachment; filename=\"$ascii\"; filename*=UTF-8''$encoded"
}
@@ -0,0 +1,142 @@
package win.taptap.musicmanager.api
import io.ktor.http.HttpStatusCode
import io.ktor.server.application.ApplicationCall
import io.ktor.server.request.contentLength
import io.ktor.server.response.respond
import io.ktor.server.routing.Route
import io.ktor.server.routing.delete
import io.ktor.server.routing.get
import io.ktor.server.routing.post
import java.time.Instant
import java.time.temporal.ChronoUnit
import org.slf4j.LoggerFactory
import win.taptap.musicmanager.config.UserAccount
import win.taptap.musicmanager.config.UsersConfig
import win.taptap.musicmanager.db.ClientLogRepository
import win.taptap.musicmanager.db.ClientLogSanitizer
import win.taptap.musicmanager.db.toDto
private val log = LoggerFactory.getLogger("win.taptap.musicmanager.api.Logs")
/** A diagnostic batch is small; anything bigger is a bug or abuse. */
private const val MAX_BODY_BYTES = 2L * 1024 * 1024
/** Keep the insert transaction short and the per-request cost predictable. */
private const val MAX_ENTRIES_PER_BATCH = 200
/** Per-user ingest ceiling, measured over the last hour. */
private const val MAX_ROWS_PER_HOUR = 20_000
/**
* Client diagnostics endpoints.
*
* The app uploads warnings/errors and a few context entries in batches. Each
* row belongs to the authenticated user (or to the user an administrator names),
* so one account can never read another account's logs. The server redacts
* credentials again before storing anything.
*/
fun Route.logRoutes(users: UsersConfig) {
post("/logs") {
val user = call.requireApiUser(users)
val length = call.request.contentLength()
if (length != null && length > MAX_BODY_BYTES) {
throw ApiException(
HttpStatusCode.PayloadTooLarge,
"logs_too_large",
"A log batch may not exceed ${MAX_BODY_BYTES / 1024} KiB",
)
}
val batch = call.receiveJson<ClientLogBatchDto>()
if (batch.entries.isEmpty()) badRequest("A log batch needs at least one entry")
if (batch.entries.size > MAX_ENTRIES_PER_BATCH) {
badRequest("A log batch may contain at most $MAX_ENTRIES_PER_BATCH entries")
}
val payloadBytes = batch.entries.sumOf { entry ->
entry.message.length + (entry.detail?.length ?: 0) + entry.tag.length
}
if (payloadBytes > MAX_BODY_BYTES) {
throw ApiException(
HttpStatusCode.PayloadTooLarge,
"logs_too_large",
"A log batch may not exceed ${MAX_BODY_BYTES / 1024} KiB",
)
}
batch.entries.forEach { entry ->
if (ClientLogSanitizer.normalizeLevel(entry.level) == null) {
badRequest("Invalid log level: ${entry.level.take(16)}")
}
}
val since = Instant.now().minus(1, ChronoUnit.HOURS)
if (ClientLogRepository.countSince(user.name, since) >= MAX_ROWS_PER_HOUR) {
throw ApiException(
HttpStatusCode.TooManyRequests,
"logs_rate_limited",
"Too many diagnostic entries in the last hour",
)
}
val accepted = ClientLogRepository.insert(user.name, batch)
// Errors should show up in the server log immediately, not only when
// someone remembers to query the diagnostics endpoint.
batch.entries
.filter { ClientLogSanitizer.normalizeLevel(it.level) == "error" }
.take(3)
.forEach { entry ->
log.warn(
"Client error [{}] {}: {}",
user.name,
ClientLogSanitizer.tag(entry.tag),
ClientLogSanitizer.message(entry.message).take(300),
)
}
call.respond(ClientLogsAcceptedDto(accepted))
}
get("/logs") {
val user = call.requireApiUser(users)
val target = call.logTargetUser(users, user)
val level = call.request.queryParameters["level"]
?.takeIf { it.isNotBlank() }
?.let {
ClientLogSanitizer.normalizeLevel(it)
?: badRequest("level must be one of ${ClientLogSanitizer.levels.sorted().joinToString()}")
}
val deviceId = call.request.queryParameters["device"]?.takeIf { it.isNotBlank() }
val beforeId = call.request.queryParameters["before"]?.let { raw ->
raw.toLongOrNull()?.takeIf { it > 0 } ?: badRequest("before must be a positive row id")
}
val sinceMillis = call.request.queryParameters["since"]?.let { raw ->
raw.toLongOrNull()?.takeIf { it > 0 } ?: badRequest("since must be epoch milliseconds")
}
val limit = call.request.queryParameters["limit"]?.let { raw ->
raw.toIntOrNull()?.takeIf { it in 1..1_000 } ?: badRequest("limit must be between 1 and 1000")
} ?: 200
val logs = ClientLogRepository.list(
userName = target,
level = level,
deviceId = deviceId,
beforeId = beforeId,
sinceMillis = sinceMillis,
limit = limit,
)
call.respond(ClientLogsDto(logs.map { it.toDto() }))
}
delete("/logs") {
val user = call.requireApiUser(users)
val target = call.logTargetUser(users, user)
val deleted = ClientLogRepository.delete(target)
call.respond(ClientLogsDeletedDto(deleted))
}
}
private fun ApplicationCall.logTargetUser(users: UsersConfig, current: UserAccount): String {
val requested = request.queryParameters["user"]?.takeIf { it.isNotBlank() }
if (requested == null || requested == current.name) return current.name
if (!current.admin) forbidden("Only administrators can read another user's logs")
return users.byName(requested)?.name ?: notFound("User")
}
@@ -7,32 +7,41 @@ import io.ktor.server.routing.delete
import io.ktor.server.routing.get
import io.ktor.server.routing.patch
import io.ktor.server.routing.post
import java.nio.file.Files
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.withContext
import kotlinx.serialization.Serializable
import win.taptap.musicmanager.api.AddSeedRequest
import win.taptap.musicmanager.api.AddToMoodResponse
import win.taptap.musicmanager.api.CreateMoodRequest
import win.taptap.musicmanager.api.MoodDetailDto
import win.taptap.musicmanager.api.MoodDto
import win.taptap.musicmanager.api.RecommendationDto
import win.taptap.musicmanager.api.SearchResponse
import win.taptap.musicmanager.api.UpdateMoodRequest
import win.taptap.musicmanager.api.seedDto
import win.taptap.musicmanager.api.toDto
import win.taptap.musicmanager.config.LibraryConfig
import win.taptap.musicmanager.config.UserAccount
import win.taptap.musicmanager.config.UsersConfig
import win.taptap.musicmanager.db.LibraryRepository
import win.taptap.musicmanager.db.DownloadRepository
import win.taptap.musicmanager.db.MoodRecord
import win.taptap.musicmanager.db.MoodRepository
import win.taptap.musicmanager.db.SongRecord
import win.taptap.musicmanager.db.SongRepository
import win.taptap.musicmanager.discovery.DiscoveryService
import win.taptap.musicmanager.discovery.MusicCatalogs
import win.taptap.musicmanager.discovery.TopUpService
import win.taptap.musicmanager.discovery.isMusicLength
import win.taptap.musicmanager.media.MediaGcService
import win.taptap.musicmanager.media.MediaStore
import win.taptap.musicmanager.radio.RadioService
import win.taptap.musicmanager.source.MusicSource
@Serializable
data class MoodsResponse(val moods: List<MoodDto>)
@Serializable
data class RecommendationsResponse(val recommendations: List<RecommendationDto>)
fun Route.moodRoutes(
users: UsersConfig,
library: LibraryConfig,
radioService: RadioService,
discoveryService: DiscoveryService,
topUpService: TopUpService,
catalogs: MusicCatalogs,
mediaStore: MediaStore,
gc: MediaGcService,
defaultSource: String,
) {
get("/moods") {
val user = call.requireApiUser(users)
@@ -44,11 +53,10 @@ fun Route.moodRoutes(
val request = call.receiveJson<CreateMoodRequest>()
val name = request.name.trim()
if (name.isEmpty()) badRequest("Mood name is required")
val folders = user.musicFolderIds(library)
val seedIds = request.seedSongIds
.distinct()
.filter { LibraryRepository.song(it, folders, user.name) != null }
val source = request.source?.let { value ->
MusicSource.normalizeOrNull(value)
?: badRequest("Unknown download source: $value")
} ?: defaultSource
val mood = MoodRepository.create(
owner = user.name,
@@ -56,18 +64,19 @@ fun Route.moodRoutes(
description = request.description?.trim()?.takeIf { it.isNotEmpty() },
query = request.query?.trim()?.takeIf { it.isNotEmpty() },
genres = request.genres,
targetSize = request.targetSize,
source = source,
collectionPercent = request.collectionPercent,
recommendationCount = request.recommendationCount,
refreshHours = request.refreshHours,
ytEnabled = request.ytEnabled,
seedSongIds = seedIds,
)
call.respond(HttpStatusCode.Created, detail(mood, user, library))
if (mood.recommendationCount > 0) topUpService.trigger(mood)
call.respond(HttpStatusCode.Created, detail(mood))
}
get("/moods/{id}") {
val user = call.requireApiUser(users)
val mood = ownedMood(call.parameters["id"], user)
call.respond(detail(mood, user, library))
call.respond(detail(mood))
}
patch("/moods/{id}") {
@@ -85,6 +94,10 @@ fun Route.moodRoutes(
request.query != null -> request.query.trim().takeIf { it.isNotEmpty() }
else -> mood.query
}
val source = request.source?.let { value ->
MusicSource.normalizeOrNull(value)
?: badRequest("Unknown download source: $value")
} ?: mood.source
val updated = MoodRepository.update(
id = mood.id,
@@ -93,44 +106,21 @@ fun Route.moodRoutes(
description = description,
query = query,
genres = request.genres ?: mood.genres,
targetSize = request.targetSize ?: mood.targetSize,
source = source,
collectionPercent = request.collectionPercent ?: mood.collectionPercent,
recommendationCount = request.recommendationCount ?: mood.recommendationCount,
refreshHours = request.refreshHours ?: mood.refreshHours,
ytEnabled = request.ytEnabled ?: mood.ytEnabled,
) ?: notFound("Mood")
call.respond(detail(updated, user, library))
topUpService.trigger(updated)
call.respond(detail(updated))
}
delete("/moods/{id}") {
val user = call.requireApiUser(users)
val mood = ownedMood(call.parameters["id"], user)
MoodRepository.delete(mood.id, user.name)
call.respond(HttpStatusCode.NoContent)
}
post("/moods/{id}/seeds") {
val user = call.requireApiUser(users)
val mood = ownedMood(call.parameters["id"], user)
val request = call.receiveJson<AddSeedRequest>()
val song = requireMusicSong(request.songId, user, library)
val added = MoodRepository.addSeed(
id = mood.id,
owner = user.name,
songId = song.id,
origin = request.origin,
)
val refreshed = MoodRepository.get(mood.id, user.name) ?: mood
call.respond(
HttpStatusCode.OK,
AddToMoodResponse(mood = refreshed.toDto(), added = added),
)
}
delete("/moods/{id}/seeds/{songId}") {
val user = call.requireApiUser(users)
val mood = ownedMood(call.parameters["id"], user)
val songId = call.parameters["songId"] ?: badRequest("Song id is required")
MoodRepository.removeSeed(mood.id, user.name, songId)
gc.collectGarbage()
call.respond(HttpStatusCode.NoContent)
}
@@ -139,14 +129,203 @@ fun Route.moodRoutes(
val user = call.requireApiUser(users)
val mood = ownedMood(call.parameters["id"], user)
val limit = (call.request.queryParameters["limit"]?.toIntOrNull() ?: 20).coerceIn(1, 50)
call.respond(SearchResponse(radioService.preview(mood, user, library, limit).map { it.toDto() }))
call.respond(TracksResponse(radioService.preview(mood, user, limit).map { it.toDto() }))
}
get("/moods/{id}/recommendations") {
val user = call.requireApiUser(users)
val mood = ownedMood(call.parameters["id"], user)
call.respond(recommendations(mood, user, library))
call.respond(RecommendationsResponse(recommendations(mood)))
}
/** Runs a discovery pass now, bypassing the mood's refresh interval. */
post("/moods/{id}/recommendations/refresh") {
val user = call.requireApiUser(users)
val mood = ownedMood(call.parameters["id"], user)
if (mood.recommendationCount <= 0) conflict("Recommendations are disabled for this mood")
val outcome = withContext(Dispatchers.IO) {
discoveryService.refresh(mood, force = true)
}
outcome.error?.let { message ->
throw ApiException(HttpStatusCode.BadGateway, "discovery_failed", message)
}
topUpService.topUp(mood, force = false)
val refreshed = MoodRepository.get(mood.id, user.name) ?: mood
call.respond(detail(refreshed))
}
post("/moods/{id}/recommendations/{externalId}/add") {
val user = call.requireApiUser(users)
val mood = ownedMood(call.parameters["id"], user)
val externalId = call.parameters["externalId"] ?: badRequest("Track id is required")
val recommendation = MoodRepository.recommendation(mood.id, externalId)
?: notFound("Recommendation")
if (recommendation.state == "failed") {
conflict("Retry the download before adding this track")
}
MoodRepository.markAdded(mood.id, externalId)
topUpService.trigger(mood)
val refreshed = MoodRepository.get(mood.id, user.name) ?: mood
val refreshedRecommendation = MoodRepository.recommendation(mood.id, externalId)
call.respond(
RecommendationDecisionDto(
detail = detail(refreshed),
recommendation = (refreshedRecommendation ?: recommendation)
.toDto(song = recommendationSong(mood.id, externalId)),
),
)
}
post("/moods/{id}/recommendations/{externalId}/reject") {
val user = call.requireApiUser(users)
val mood = ownedMood(call.parameters["id"], user)
val externalId = call.parameters["externalId"] ?: badRequest("Track id is required")
val recommendation = MoodRepository.recommendation(mood.id, externalId)
?: notFound("Recommendation")
MoodRepository.markRejected(mood.id, externalId)
DownloadRepository.cancelForMoodTrack(mood.id, externalId, "Recommendation rejected")
recommendation.songId?.let(gc::collect)
topUpService.trigger(mood)
val refreshed = MoodRepository.get(mood.id, user.name) ?: mood
call.respond(
RecommendationDecisionDto(
detail = detail(refreshed),
recommendation = recommendation.copy(state = "rejected").toDto(),
),
)
}
post("/moods/{id}/recommendations/{externalId}/retry") {
val user = call.requireApiUser(users)
val mood = ownedMood(call.parameters["id"], user)
val externalId = call.parameters["externalId"] ?: badRequest("Track id is required")
val recommendation = MoodRepository.recommendation(mood.id, externalId)
?: notFound("Recommendation")
if (recommendation.state != "failed") conflict("Only failed downloads can be retried")
MoodRepository.markRecommendationDownloading(mood.id, externalId)
DownloadRepository.enqueue(
source = recommendation.source,
externalId = externalId,
moodId = mood.id,
kind = DownloadRepository.KIND_RECOMMENDATION,
)
val refreshed = MoodRepository.get(mood.id, user.name) ?: mood
call.respond(detail(refreshed))
}
post("/moods/{id}/songs") {
val user = call.requireApiUser(users)
val mood = ownedMood(call.parameters["id"], user)
val request = call.receiveJson<AddSongRequest>()
val normalized = normalizeAdd(request, catalogs, mood.source)
val bySongId = normalized.songId?.let(SongRepository::byId)
if (normalized.songId != null && bySongId == null) notFound("Song")
val existingSong = bySongId
?: normalized.source?.let { source ->
normalized.externalId?.let { SongRepository.byExternalId(source, it) }
}
val existingFile = existingSong?.let { mediaStore.file(it) }
if (existingSong != null && existingFile != null && Files.isRegularFile(existingFile)) {
val added = MoodRepository.addSong(mood.id, existingSong.id, normalized.origin)
val refreshed = MoodRepository.get(mood.id, user.name) ?: mood
topUpService.trigger(refreshed)
call.respond(
AddSongResponse(
detail = detail(refreshed),
added = added,
downloading = false,
),
)
return@post
}
// The row may exist without its file (GC removed it), or the track may
// be new. Either way the job carries the source and external id.
val source = existingSong?.source ?: normalized.source
val externalId = existingSong?.externalId ?: normalized.externalId
if (source == null || externalId == null) {
badRequest("No song or track id was supplied")
}
DownloadRepository.enqueue(
source = source,
externalId = externalId,
moodId = mood.id,
kind = DownloadRepository.KIND_SONG,
)
val refreshed = MoodRepository.get(mood.id, user.name) ?: mood
call.respond(
HttpStatusCode.Accepted,
AddSongResponse(detail = detail(refreshed), added = false, downloading = true),
)
}
delete("/moods/{id}/songs/{songId}") {
val user = call.requireApiUser(users)
val mood = ownedMood(call.parameters["id"], user)
val songId = call.parameters["songId"] ?: badRequest("Song id is required")
MoodRepository.removeSong(mood.id, songId)
gc.collect(songId)
call.respond(HttpStatusCode.NoContent)
}
}
private data class NormalizedAdd(
val songId: String? = null,
val source: String? = null,
val externalId: String? = null,
val origin: String,
)
private fun normalizeAdd(
request: AddSongRequest,
catalogs: MusicCatalogs,
moodSource: String,
): NormalizedAdd = when {
!request.songId.isNullOrBlank() -> NormalizedAdd(songId = request.songId, origin = "search")
!request.externalId.isNullOrBlank() -> {
val source = request.source?.let { value ->
MusicSource.normalizeOrNull(value)
?: badRequest("Unknown download source: $value")
} ?: moodSource
NormalizedAdd(
source = source,
externalId = request.externalId.trim(),
origin = "search",
)
}
!request.url.isNullOrBlank() -> {
val candidate = runCatching { catalogs.resolve(request.url) }
.getOrElse { error ->
throw ApiException(
HttpStatusCode.BadGateway,
"resolve_failed",
error.message ?: "Could not resolve the URL",
)
}
?: throw ApiException(
HttpStatusCode.BadGateway,
"resolve_failed",
"The URL did not resolve to a track",
)
if (!isMusicLength(candidate.duration)) {
badRequest("Only tracks up to 10 minutes can be added")
}
NormalizedAdd(
source = candidate.source,
externalId = candidate.externalId,
origin = "url",
)
}
else -> badRequest("Supply songId, source and externalId, or url")
}
private fun ownedMood(id: String?, user: UserAccount): MoodRecord {
@@ -154,56 +333,54 @@ private fun ownedMood(id: String?, user: UserAccount): MoodRecord {
return MoodRepository.get(id, user.name) ?: notFound("Mood")
}
private fun requireMusicSong(songId: String, user: UserAccount, library: LibraryConfig) =
LibraryRepository.song(songId, user.musicFolderIds(library), user.name)
?: notFound("Song")
private fun detail(mood: MoodRecord, user: UserAccount, library: LibraryConfig): MoodDetailDto {
val folders = user.musicFolderIds(library)
val seedRecords = MoodRepository.seeds(mood.id)
val seedSongs = LibraryRepository
.songsByIds(seedRecords.map { it.songId }, folders, user.name)
.associateBy { it.id }
val seeds = seedRecords.mapNotNull { seed ->
seedSongs[seed.songId]?.let { seedDto(seed, it) }
private fun detail(mood: MoodRecord): MoodDetailDto {
val songRecords = MoodRepository.songRecords(mood.id).associateBy { it.id }
val songs = MoodRepository.songs(mood.id).mapNotNull { record ->
songRecords[record.songId]?.let { record.toDto(it) }
}
return MoodDetailDto(
mood = mood.toDto(),
seeds = seeds,
recommendations = recommendations(mood, user, library).recommendations,
songs = songs,
recommendations = recommendations(mood),
downloads = DownloadRepository.forMood(mood.id)
.take(MAX_REPORTED_DOWNLOADS)
.map { job ->
DownloadStatusDto(
source = job.source,
externalId = job.externalId,
state = job.state,
attempts = job.attempts,
error = job.error,
songId = job.songId,
)
},
)
}
private fun recommendations(
mood: MoodRecord,
user: UserAccount,
library: LibraryConfig,
): RecommendationsResponse {
private fun recommendations(mood: MoodRecord): List<RecommendationDto> {
val records = MoodRepository.recommendations(mood.id)
val matchedIds = records.mapNotNull { it.matchedSongId }
val matchedSongs = LibraryRepository
.songsByIds(matchedIds, user.visibleFolderIds(library), user.name)
.associateBy { it.id }
val songs = SongRepository.byIds(records.mapNotNull { it.songId }).associateBy { it.id }
val jobs = DownloadRepository.forMood(mood.id)
.filter { it.kind == DownloadRepository.KIND_RECOMMENDATION }
.associateBy { "${it.source}|${it.externalId}" }
return RecommendationsResponse(
recommendations = records.map { record ->
RecommendationDto(
videoId = record.videoId,
title = record.title,
artist = record.artist,
album = record.album,
duration = record.duration,
thumbnail = record.thumbnail,
state = record.state,
matchedSong = record.matchedSongId?.let { matchedSongs[it] }?.toDto(),
timesAdded = record.timesAdded,
firstSeen = record.firstSeen,
lastSeen = record.lastSeen,
)
},
)
return records.map { record ->
record.toDto(
song = record.songId?.let { songs[it] },
download = jobs["${record.source}|${record.externalId}"]?.let { job ->
DownloadDto(
state = job.state,
attempts = job.attempts,
error = job.error,
updatedAt = job.updatedAt,
)
},
)
}
}
@Serializable
data class RecommendationsResponse(val recommendations: List<RecommendationDto>)
private fun recommendationSong(moodId: String, externalId: String): SongRecord? =
MoodRepository.recommendation(moodId, externalId)?.songId?.let(SongRepository::byId)
/** Newest first; older finished jobs do not need to reach the client. */
private const val MAX_REPORTED_DOWNLOADS = 100
@@ -1,47 +1,41 @@
package win.taptap.musicmanager.api
import io.ktor.http.HttpStatusCode
import io.ktor.server.request.receiveNullable
import io.ktor.server.response.respond
import io.ktor.server.routing.Route
import io.ktor.server.routing.delete
import io.ktor.server.routing.get
import io.ktor.server.routing.post
import kotlinx.serialization.Serializable
import win.taptap.musicmanager.api.AddToMoodRequest
import win.taptap.musicmanager.api.AddToMoodResponse
import win.taptap.musicmanager.api.NextTracksRequest
import win.taptap.musicmanager.api.PlaybackEventsRequest
import win.taptap.musicmanager.api.PlaybackEventsResponse
import win.taptap.musicmanager.api.SessionDto
import win.taptap.musicmanager.api.SongDto
import win.taptap.musicmanager.api.StartSessionRequest
import win.taptap.musicmanager.api.toDto
import win.taptap.musicmanager.config.LibraryConfig
import win.taptap.musicmanager.config.UserAccount
import win.taptap.musicmanager.config.UsersConfig
import win.taptap.musicmanager.db.LibraryRepository
import win.taptap.musicmanager.db.MoodRecord
import win.taptap.musicmanager.db.MoodRepository
import win.taptap.musicmanager.db.PlaybackSessionRecord
import win.taptap.musicmanager.db.RadioRepository
import win.taptap.musicmanager.db.SongRepository
import win.taptap.musicmanager.media.MediaGcService
import win.taptap.musicmanager.radio.RadioService
@Serializable
data class TracksResponse(val songs: List<SongDto>)
fun Route.sessionRoutes(
users: UsersConfig,
library: LibraryConfig,
radioService: RadioService,
gc: MediaGcService,
) {
post("/moods/{id}/sessions") {
val user = call.requireApiUser(users)
val mood = ownedMoodFor(call.parameters["id"], user)
val request = runCatching { call.receiveJson<StartSessionRequest>() }
.getOrDefault(StartSessionRequest())
if (MoodRepository.songIds(mood.id).isEmpty()) {
throw ApiException(
HttpStatusCode.BadRequest,
"empty_mood",
"Add at least one song before playing this mood",
)
}
val request = call.receiveNullable<StartSessionRequest>() ?: StartSessionRequest()
val limit = request.limit ?: DEFAULT_BATCH
val started = radioService.start(mood, user, library, limit)
val started = radioService.start(mood, user, limit)
call.respond(
HttpStatusCode.Created,
SessionDto(
@@ -58,11 +52,8 @@ fun Route.sessionRoutes(
val user = call.requireApiUser(users)
val session = ownedSession(call.parameters["id"], user)
val mood = ownedMoodFor(session.moodId, user)
val folders = user.musicFolderIds(library)
val items = RadioRepository.items(session.id)
val songs = LibraryRepository
.songsByIds(items.map { it.songId }, folders, user.name)
.associateBy { it.id }
val songs = SongRepository.byIds(items.map { it.songId }).associateBy { it.id }
call.respond(
SessionDto(
@@ -81,13 +72,11 @@ fun Route.sessionRoutes(
val session = ownedSession(call.parameters["id"], user)
if (session.closedAt != null) conflict("Session is closed")
val mood = ownedMoodFor(session.moodId, user)
val request = runCatching { call.receiveJson<NextTracksRequest>() }
.getOrDefault(NextTracksRequest())
val request = call.receiveNullable<NextTracksRequest>() ?: NextTracksRequest()
val songs = radioService.next(
mood = mood,
user = user,
library = library,
sessionId = session.id,
excluded = request.excludeSongIds.toSet(),
limit = request.limit ?: DEFAULT_BATCH,
@@ -112,9 +101,7 @@ fun Route.sessionRoutes(
val user = call.requireApiUser(users)
val session = ownedSession(call.parameters["id"], user)
val request = call.receiveJson<AddToMoodRequest>()
val folders = user.musicFolderIds(library)
val song = LibraryRepository.song(request.songId, folders, user.name)
?: notFound("Song")
val song = SongRepository.byId(request.songId) ?: notFound("Song")
val target = if (request.moodId.isNullOrBlank()) {
ownedMoodFor(session.moodId, user)
@@ -122,7 +109,7 @@ fun Route.sessionRoutes(
ownedMoodFor(request.moodId, user)
}
val added = MoodRepository.addSeed(target.id, user.name, song.id, "manual")
val added = MoodRepository.addSong(target.id, song.id, "search")
val refreshed = MoodRepository.get(target.id, user.name) ?: target
call.respond(
AddToMoodResponse(
@@ -136,6 +123,7 @@ fun Route.sessionRoutes(
val user = call.requireApiUser(users)
val session = ownedSession(call.parameters["id"], user)
RadioRepository.close(session.id, user.name)
gc.collectGarbage()
call.respond(HttpStatusCode.NoContent)
}
}
@@ -0,0 +1,175 @@
package win.taptap.musicmanager.api
import io.ktor.http.ContentType
import io.ktor.http.HttpHeaders
import io.ktor.http.HttpStatusCode
import io.ktor.server.response.header
import io.ktor.server.response.respond
import io.ktor.server.response.respondFile
import io.ktor.server.response.respondOutputStream
import io.ktor.server.routing.Route
import io.ktor.server.routing.get
import java.net.URLEncoder
import java.nio.file.Files
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.withContext
import org.slf4j.LoggerFactory
import win.taptap.musicmanager.config.UsersConfig
import win.taptap.musicmanager.db.MoodRepository
import win.taptap.musicmanager.db.SongRecord
import win.taptap.musicmanager.db.SongRepository
import win.taptap.musicmanager.discovery.MusicCatalogs
import win.taptap.musicmanager.discovery.isMusicLength
import win.taptap.musicmanager.source.MusicSource
import win.taptap.musicmanager.media.MediaStore
import win.taptap.musicmanager.scanner.CoverArtService
import win.taptap.musicmanager.transcoding.FfmpegTranscoder
private val log = LoggerFactory.getLogger("win.taptap.musicmanager.api.SongRoutes")
/**
* YouTube search plus the shared media endpoints. Songs are server-wide, so
* any authenticated user can stream a song id they received from their own
* moods or from search.
*/
fun Route.songRoutes(
users: UsersConfig,
catalogs: MusicCatalogs,
mediaStore: MediaStore,
coverArtService: CoverArtService,
transcoder: FfmpegTranscoder,
defaultSource: String,
) {
get("/search") {
val user = call.requireApiUser(users)
val query = call.request.queryParameters["q"]?.trim().orEmpty()
if (query.isEmpty()) {
call.respond(SearchResponse())
return@get
}
val limit = (call.request.queryParameters["limit"]?.toIntOrNull() ?: 20).coerceIn(1, 25)
val moodId = call.request.queryParameters["moodId"]?.takeIf { it.isNotBlank() }
val mood = moodId?.let { MoodRepository.get(it, user.name) ?: notFound("Mood") }
val source = call.request.queryParameters["source"]?.let { value ->
MusicSource.normalizeOrNull(value)
?: badRequest("Unknown download source: $value")
} ?: mood?.source ?: defaultSource
val moodTracks = mood?.let { MoodRepository.externalIdsInMood(it.id, source) }.orEmpty()
val catalog = catalogs.forSource(source)
val results = withContext(Dispatchers.IO) {
runCatching { catalog.search(query, limit) }
.getOrElse { error ->
throw ApiException(
HttpStatusCode.BadGateway,
"search_failed",
error.message ?: "Search failed",
)
}
}
call.respond(
SearchResponse(
results = results
.filter { isMusicLength(it.duration) }
.map { candidate ->
val song = SongRepository.byExternalId(candidate.source, candidate.externalId)
val downloaded = song != null && isPresent(mediaStore, song)
SearchResultDto(
source = candidate.source,
externalId = candidate.externalId,
title = candidate.title,
artist = candidate.artist,
album = candidate.album,
duration = candidate.duration,
thumbnail = candidate.thumbnail,
songId = song?.takeIf { downloaded }?.id,
downloaded = downloaded,
inMood = candidate.externalId in moodTracks,
)
},
),
)
}
get("/songs/{id}/stream") {
call.requireApiUser(users)
val id = call.parameters["id"] ?: badRequest("Song id is required")
val song = SongRepository.byId(id) ?: notFound("Song")
val file = mediaStore.file(song)?.takeIf { Files.isRegularFile(it) } ?: notFound("Song file")
val parameters = call.request.queryParameters
val rawMaxBitRate = parameters["maxBitRate"]
val maxBitRate = rawMaxBitRate?.toIntOrNull()
if (rawMaxBitRate != null && (maxBitRate == null || maxBitRate < 0)) {
badRequest("Invalid maxBitRate")
}
val rawOffset = parameters["timeOffset"]
val offset = rawOffset?.toIntOrNull()
if (rawOffset != null && (offset == null || offset < 0)) {
badRequest("Invalid timeOffset")
}
if (!transcoder.shouldTranscode(song.bitRate, maxBitRate, parameters["format"], offset ?: 0)) {
call.response.header(HttpHeaders.ContentType, song.contentType)
if (parameters["download"] == "true") {
call.response.header(
HttpHeaders.ContentDisposition,
contentDisposition(file.fileName.toString()),
)
}
call.respondFile(file.toFile())
return@get
}
val process = try {
transcoder.start(file, maxBitRate, offset ?: 0)
} catch (e: Exception) {
log.error("Failed to start ffmpeg for {}", file, e)
throw ApiException(HttpStatusCode.InternalServerError, "transcode_failed", "Transcoding failed")
}
try {
call.response.headers.append(HttpHeaders.CacheControl, "no-store")
call.respondOutputStream(ContentType.parse("audio/mpeg")) {
val output = this
withContext(Dispatchers.IO) {
process.inputStream.use { input -> input.copyTo(output) }
}
}
} finally {
transcoder.stop(process)
}
}
get("/songs/{id}/cover") {
call.requireApiUser(users)
val id = call.parameters["id"] ?: badRequest("Song id is required")
val rawSize = call.request.queryParameters["size"]
val size = rawSize?.toIntOrNull()
if (rawSize != null && (size == null || size !in 1..2048)) {
badRequest("Cover size must be between 1 and 2048")
}
SongRepository.byId(id) ?: notFound("Song")
val cover = if (size == null) {
coverArtService.coverFile(id)
} else {
coverArtService.resizedCover(id, size)
} ?: notFound("Cover")
call.respondFile(cover.toFile())
}
}
private fun isPresent(mediaStore: MediaStore, song: SongRecord): Boolean =
mediaStore.file(song)?.let { Files.isRegularFile(it) } == true
private fun contentDisposition(fileName: String): String {
val encoded = URLEncoder.encode(fileName, Charsets.UTF_8).replace("+", "%20")
val ascii = fileName.map { character ->
if (character.code in 32..126 && character != '"' && character != '\\') character else '_'
}.joinToString("")
return "attachment; filename=\"$ascii\"; filename*=UTF-8''$encoded"
}
@@ -1,19 +1,37 @@
package win.taptap.musicmanager.config
import java.nio.file.Path
import win.taptap.musicmanager.source.MusicSource
/**
* Application-level configuration, sourced exclusively from environment variables.
*
* Library folders and users are not part of this class: they live in the YAML files
* referenced here so they can be managed by ConfigMaps/Vault.
* There is no mounted library: the server owns [mediaDir] and fills it by
* downloading from Deezer or YouTube Music. Users still live in a Vault-managed
* YAML file.
*/
data class AppConfig(
val port: Int,
val dataDir: Path,
val libraryConfigPath: Path,
val usersFile: Path,
val ffmpegPath: String,
val ytDlpPath: String = "yt-dlp",
val streamripPath: String = "rip",
val defaultSource: String = MusicSource.DEEZER,
val deezerArl: String? = null,
/** 0 = 128 kbps MP3, 1 = 320 kbps MP3. FLAC is not stored. */
val deezerQuality: Int = 0,
val deezerApiBase: String = "https://api.deezer.com",
/** Analyze downloads and backfill the library to a common loudness. */
val loudnessEnabled: Boolean = true,
/** Integrated loudness target; -16 LUFS is the ReplayGain-style value. */
val loudnessTargetLufs: Double = -16.0,
/** Client diagnostics: rows older than this are pruned. */
val clientLogRetentionDays: Int = 14,
/** Client diagnostics: newest rows kept per user. */
val clientLogMaxRowsPerUser: Int = 20_000,
val mediaDir: Path = dataDir.resolve("media"),
val downloadDir: Path = dataDir.resolve("downloads"),
) {
val databasePath: Path get() = dataDir.resolve("musicmanager.db")
@@ -27,12 +45,57 @@ data class AppConfig(
?: throw IllegalArgumentException("PORT must be an integer, got: $rawPort")
require(port in 1..65535) { "PORT must be between 1 and 65535, got: $port" }
val dataDir = Path.of(value("DATA_DIR", "./data"))
val rawSource = value("DEFAULT_SOURCE", MusicSource.DEEZER)
val defaultSource = MusicSource.normalizeOrNull(rawSource)
?: throw IllegalArgumentException(
"DEFAULT_SOURCE must be one of ${MusicSource.ALL}, got: $rawSource",
)
val rawQuality = value("DEEZER_QUALITY", "0")
val deezerQuality = rawQuality.toIntOrNull()
?: throw IllegalArgumentException("DEEZER_QUALITY must be 0 or 1, got: $rawQuality")
require(deezerQuality in 0..1) {
"DEEZER_QUALITY must be 0 (128 kbps) or 1 (320 kbps), got: $deezerQuality"
}
val loudnessEnabled = value("LOUDNESS_ENABLED", "true").toBooleanStrictOrNull()
?: throw IllegalArgumentException("LOUDNESS_ENABLED must be true or false")
val rawTarget = value("LOUDNESS_TARGET_LUFS", "-16")
val loudnessTargetLufs = rawTarget.toDoubleOrNull()
?: throw IllegalArgumentException("LOUDNESS_TARGET_LUFS must be a number, got: $rawTarget")
require(loudnessTargetLufs in -30.0..-5.0) {
"LOUDNESS_TARGET_LUFS must be between -30 and -5, got: $loudnessTargetLufs"
}
val rawRetention = value("CLIENT_LOG_RETENTION_DAYS", "14")
val clientLogRetentionDays = rawRetention.toIntOrNull()
?: throw IllegalArgumentException("CLIENT_LOG_RETENTION_DAYS must be an integer")
require(clientLogRetentionDays in 1..365) {
"CLIENT_LOG_RETENTION_DAYS must be between 1 and 365, got: $clientLogRetentionDays"
}
val rawLogRows = value("CLIENT_LOG_MAX_ROWS_PER_USER", "20000")
val clientLogMaxRowsPerUser = rawLogRows.toIntOrNull()
?: throw IllegalArgumentException("CLIENT_LOG_MAX_ROWS_PER_USER must be an integer")
require(clientLogMaxRowsPerUser in 100..1_000_000) {
"CLIENT_LOG_MAX_ROWS_PER_USER must be between 100 and 1000000, got: $clientLogMaxRowsPerUser"
}
return AppConfig(
port = port,
dataDir = Path.of(value("DATA_DIR", "./data")),
libraryConfigPath = Path.of(value("LIBRARY_CONFIG", "./config/library.yaml")),
dataDir = dataDir,
usersFile = Path.of(value("USERS_FILE", "./config/users.yaml")),
ffmpegPath = value("FFMPEG_PATH", "ffmpeg"),
ytDlpPath = value("YTDLP_PATH", "yt-dlp"),
streamripPath = value("STREAMRIP_PATH", "rip"),
defaultSource = defaultSource,
deezerArl = env["DEEZER_ARL"]?.takeIf { it.isNotBlank() },
deezerQuality = deezerQuality,
deezerApiBase = value("DEEZER_API_BASE", "https://api.deezer.com"),
loudnessEnabled = loudnessEnabled,
loudnessTargetLufs = loudnessTargetLufs,
clientLogRetentionDays = clientLogRetentionDays,
clientLogMaxRowsPerUser = clientLogMaxRowsPerUser,
mediaDir = Path.of(value("MEDIA_DIR", dataDir.resolve("media").toString())),
downloadDir = Path.of(value("DOWNLOAD_DIR", dataDir.resolve("downloads").toString())),
)
}
}
@@ -1,138 +0,0 @@
package win.taptap.musicmanager.config
import java.nio.file.Files
import java.nio.file.Path
import org.yaml.snakeyaml.LoaderOptions
import org.yaml.snakeyaml.Yaml
import org.yaml.snakeyaml.constructor.SafeConstructor
enum class FolderType {
MUSIC,
LONGFORM,
}
data class MusicFolder(
val id: Int,
val name: String,
val type: FolderType,
val path: Path,
)
/**
* Library folders from `library.yaml`. Folder IDs are stable and referenced by songs,
* so they must never be reused for a different path.
*/
data class LibraryConfig(val folders: List<MusicFolder>) {
val longformFolderIds: Set<Int> = folders
.filter { it.type == FolderType.LONGFORM }
.map { it.id }
.toSet()
fun byId(id: Int): MusicFolder? = folders.firstOrNull { it.id == id }
/** Fails fast at startup if a configured root is missing or unreadable. */
fun validatePaths() {
folders.forEach { folder ->
require(Files.isDirectory(folder.path)) {
"Library folder ${folder.id} (${folder.name}) is not a directory: ${folder.path}. " +
"Create the directory or fix the path in the library config (LIBRARY_CONFIG). " +
"For local development, run `./gradlew setupLocal` and copy " +
"config/library.example.yaml to config/library.yaml. That file is " +
"git-ignored, so `git pull` never updates it."
}
require(Files.isReadable(folder.path)) {
"Library folder ${folder.id} (${folder.name}) is not readable: ${folder.path}. " +
"Check the directory permissions."
}
}
}
companion object {
fun load(path: Path): LibraryConfig {
require(Files.isRegularFile(path)) {
"Library config not found: $path. Copy config/library.example.yaml to " +
"config/library.yaml or point LIBRARY_CONFIG at an existing file."
}
val text = Files.readString(path)
return parse(text)
}
fun parse(text: String): LibraryConfig {
val root = parseYaml(text)
val library = root["library"].asMapping("library")
val rawFolders = library["folders"].asList("library.folders")
val folders = rawFolders.mapIndexed { index, raw ->
val item = raw.asMapping("library.folders[$index]")
val id = item["id"].asInt("library.folders[$index].id")
require(id > 0) { "library.folders[$index].id must be positive" }
val name = item["name"].asString("library.folders[$index].name").trim()
require(name.isNotEmpty()) { "library.folders[$index].name must not be blank" }
val rawType = item["type"].asString("library.folders[$index].type")
val type = runCatching { FolderType.valueOf(rawType.trim().uppercase()) }
.getOrElse {
throw IllegalArgumentException(
"library.folders[$index].type must be 'music' or 'longform', got: $rawType",
)
}
val rawPath = item["path"].asString("library.folders[$index].path")
val path = Path.of(rawPath).toAbsolutePath().normalize()
MusicFolder(id = id, name = name, type = type, path = path)
}
require(folders.isNotEmpty()) { "library.folders must not be empty" }
folders.groupBy { it.id }.forEach { (id, duplicates) ->
require(duplicates.size == 1) { "Duplicate library folder id: $id" }
}
folders.groupBy { it.path }.forEach { (path, duplicates) ->
require(duplicates.size == 1) { "Duplicate library folder path: $path" }
}
return LibraryConfig(folders)
}
private fun parseYaml(text: String): Map<String, Any?> {
val loaded = Yaml(SafeConstructor(LoaderOptions())).load<Any?>(text)
?: throw IllegalArgumentException("Config must not be empty")
@Suppress("UNCHECKED_CAST")
return loaded as? Map<String, Any?>
?: throw IllegalArgumentException("Config root must be a YAML mapping")
}
}
}
internal fun Any?.asMapping(what: String): Map<String, Any?> {
@Suppress("UNCHECKED_CAST")
return this as? Map<String, Any?>
?: throw IllegalArgumentException("$what must be a mapping")
}
internal fun Any?.asList(what: String): List<Any?> =
this as? List<Any?> ?: throw IllegalArgumentException("$what must be a list")
internal fun Any?.asString(what: String): String =
this as? String ?: throw IllegalArgumentException("$what must be a string")
internal fun Any?.asInt(what: String): Int = when (this) {
is Int -> this
is Long -> toInt()
is String -> toIntOrNull()
?: throw IllegalArgumentException("$what must be an integer")
else -> throw IllegalArgumentException("$what must be an integer")
}
internal fun Any?.asBoolean(what: String): Boolean = when (this) {
is Boolean -> this
is String -> when (trim().lowercase()) {
"true", "yes", "1" -> true
"false", "no", "0" -> false
else -> throw IllegalArgumentException("$what must be a boolean")
}
else -> throw IllegalArgumentException("$what must be a boolean")
}
@@ -10,19 +10,17 @@ import org.yaml.snakeyaml.constructor.SafeConstructor
* User accounts. Passwords are supplied by a Vault-managed secret and are never
* persisted in SQLite. The `toString` implementation redacts the password so it
* cannot leak through logs or error messages.
*
* Moods are strictly per-user, so there are no folder assignments anymore. An
* old `folders:` key in users.yaml is accepted and ignored.
*/
data class UserAccount(
val name: String,
val password: String,
val admin: Boolean,
val folders: Set<Int>,
) {
/** Music folders assigned to the user plus every shared long-form folder. */
fun visibleFolderIds(library: LibraryConfig): Set<Int> =
folders + library.longformFolderIds
override fun toString(): String =
"UserAccount(name=$name, admin=$admin, folders=$folders, password=***)"
"UserAccount(name=$name, admin=$admin, password=***)"
}
data class UsersConfig(val users: List<UserAccount>) {
@@ -36,19 +34,6 @@ data class UsersConfig(val users: List<UserAccount>) {
fun byName(name: String): UserAccount? = users.firstOrNull { it.name == name }
fun validateAgainst(library: LibraryConfig) {
val knownFolderIds = library.folders.map { it.id }.toSet()
users.forEach { user ->
val unknown = user.folders - knownFolderIds
require(unknown.isEmpty()) {
"User '${user.name}' references unknown library folders: ${unknown.sorted()}"
}
require(user.visibleFolderIds(library).isNotEmpty()) {
"User '${user.name}' has no visible library folders"
}
}
}
companion object {
private val NAME_PATTERN = Regex("[A-Za-z0-9._-]{1,64}")
@@ -81,16 +66,7 @@ data class UsersConfig(val users: List<UserAccount>) {
val admin = item["admin"]?.asBoolean("users[$index].admin") ?: false
val folders = when (val rawFolders = item["folders"]) {
null -> emptySet()
else -> rawFolders.asList("users[$index].folders")
.mapIndexed { folderIndex, value ->
value.asInt("users[$index].folders[$folderIndex]")
}
.toSet()
}
UserAccount(name = name, password = password, admin = admin, folders = folders)
UserAccount(name = name, password = password, admin = admin)
}
return UsersConfig(users)
@@ -0,0 +1,33 @@
package win.taptap.musicmanager.config
/** Small YAML coercion helpers shared by the config parsers. */
internal fun Any?.asMapping(what: String): Map<String, Any?> {
@Suppress("UNCHECKED_CAST")
return this as? Map<String, Any?>
?: throw IllegalArgumentException("$what must be a mapping")
}
internal fun Any?.asList(what: String): List<Any?> =
this as? List<Any?> ?: throw IllegalArgumentException("$what must be a list")
internal fun Any?.asString(what: String): String =
this as? String ?: throw IllegalArgumentException("$what must be a string")
internal fun Any?.asInt(what: String): Int = when (this) {
is Int -> this
is Long -> toInt()
is String -> toIntOrNull()
?: throw IllegalArgumentException("$what must be an integer")
else -> throw IllegalArgumentException("$what must be an integer")
}
internal fun Any?.asBoolean(what: String): Boolean = when (this) {
is Boolean -> this
is String -> when (trim().lowercase()) {
"true", "yes", "1" -> true
"false", "no", "0" -> false
else -> throw IllegalArgumentException("$what must be a boolean")
}
else -> throw IllegalArgumentException("$what must be a boolean")
}
@@ -0,0 +1,234 @@
package win.taptap.musicmanager.db
import java.time.Instant
import java.time.temporal.ChronoUnit
import org.jetbrains.exposed.sql.Op
import org.jetbrains.exposed.sql.ResultRow
import org.jetbrains.exposed.sql.SortOrder
import org.jetbrains.exposed.sql.SqlExpressionBuilder.eq
import org.jetbrains.exposed.sql.SqlExpressionBuilder.greaterEq
import org.jetbrains.exposed.sql.SqlExpressionBuilder.less
import org.jetbrains.exposed.sql.and
import org.jetbrains.exposed.sql.deleteWhere
import org.jetbrains.exposed.sql.insert
import org.jetbrains.exposed.sql.selectAll
import org.jetbrains.exposed.sql.transactions.transaction
import win.taptap.musicmanager.api.ClientLogBatchDto
import win.taptap.musicmanager.api.ClientLogDto
data class ClientLogRecord(
val id: Long,
val deviceId: String,
val platform: String,
val appVersion: String,
val sessionId: String?,
val level: String,
val tag: String,
val message: String,
val detail: String?,
val createdAt: String,
val receivedAt: String,
)
fun ClientLogRecord.toDto(): ClientLogDto = ClientLogDto(
id = id,
deviceId = deviceId,
platform = platform,
appVersion = appVersion,
sessionId = sessionId,
level = level,
tag = tag,
message = message,
detail = detail,
createdAt = createdAt,
receivedAt = receivedAt,
)
/**
* Removes credentials before anything is persisted. The client redacts as well,
* but the server must not trust a buggy or older build: a token in a log line
* would give whoever reads the table access to the account.
*/
object ClientLogSanitizer {
private const val MAX_MESSAGE = 4_000
private const val MAX_DETAIL = 16_000
private const val MAX_TAG = 64
private const val MAX_DEVICE_ID = 80
private const val MAX_PLATFORM = 32
private const val MAX_APP_VERSION = 40
private const val MAX_SESSION_ID = 64
private val bearer = Regex("""(?i)\b(bearer\s+)[A-Za-z0-9._~+/=-]{8,}""")
private val authorization = Regex("""(?i)\b(authorization\s*:\s*)[^\r\n]+""")
private val querySecret = Regex(
"""(?i)(\b(?:token|access_token|password|passwd|arl|key)=)[^&\s"'<>]+""",
)
private val jsonSecret = Regex(
"""(?i)("(?:token|access_token|password|arl|key)"\s*:\s*")[^"]*(")""",
)
val levels = setOf("debug", "info", "warn", "error")
fun redact(value: String): String = value
.replace(bearer) { match -> match.groupValues[1] + "***" }
.replace(authorization) { match -> match.groupValues[1] + "***" }
.replace(querySecret) { match -> match.groupValues[1] + "***" }
.replace(jsonSecret) { match -> match.groupValues[1] + "***" + match.groupValues[2] }
fun normalizeLevel(value: String): String? =
value.trim().lowercase().takeIf { it in levels }
fun deviceId(value: String): String = clean(value, MAX_DEVICE_ID).ifBlank { "unknown" }
fun platform(value: String): String = clean(value, MAX_PLATFORM).ifBlank { "unknown" }
fun appVersion(value: String): String = clean(value, MAX_APP_VERSION).ifBlank { "unknown" }
fun sessionId(value: String?): String? =
value?.let { clean(it, MAX_SESSION_ID) }?.takeIf { it.isNotBlank() }
fun tag(value: String): String = clean(value, MAX_TAG).ifBlank { "app" }
fun message(value: String): String = clean(value, MAX_MESSAGE).ifBlank { "(empty)" }
fun detail(value: String?): String? =
value?.let { clean(it, MAX_DETAIL) }?.takeIf { it.isNotBlank() }
private fun clean(value: String, max: Int): String =
redact(value).trim().let { if (it.length > max) it.take(max) else it }
}
/**
* Client diagnostics storage. A batch is written in one transaction so a broken
* request cannot leave half a log behind, and retention runs on the server clock.
*/
object ClientLogRepository {
private const val MAX_CLOCK_SKEW_MILLIS = 5 * 60_000L
private const val MAX_ENTRY_AGE_MILLIS = 90L * 24 * 60 * 60_000
fun insert(userName: String, batch: ClientLogBatchDto): Int {
val now = Instant.now()
val receivedAt = now.toString()
val deviceId = ClientLogSanitizer.deviceId(batch.deviceId)
val platform = ClientLogSanitizer.platform(batch.platform)
val appVersion = ClientLogSanitizer.appVersion(batch.appVersion)
return transaction {
var inserted = 0
batch.entries.forEach { entry ->
val level = ClientLogSanitizer.normalizeLevel(entry.level) ?: return@forEach
val millis = entry.timestamp
.takeIf { it > 0 }
?.coerceIn(
now.toEpochMilli() - MAX_ENTRY_AGE_MILLIS,
now.toEpochMilli() + MAX_CLOCK_SKEW_MILLIS,
)
?: now.toEpochMilli()
ClientLogsTable.insert {
it[ClientLogsTable.userName] = userName
it[ClientLogsTable.deviceId] = deviceId
it[ClientLogsTable.platform] = platform
it[ClientLogsTable.appVersion] = appVersion
it[sessionId] = ClientLogSanitizer.sessionId(entry.sessionId)
it[ClientLogsTable.level] = level
it[tag] = ClientLogSanitizer.tag(entry.tag)
it[message] = ClientLogSanitizer.message(entry.message)
it[detail] = ClientLogSanitizer.detail(entry.detail)
it[createdAt] = Instant.ofEpochMilli(millis).toString()
it[ClientLogsTable.receivedAt] = receivedAt
}
inserted += 1
}
inserted
}
}
fun list(
userName: String,
level: String? = null,
deviceId: String? = null,
beforeId: Long? = null,
sinceMillis: Long? = null,
limit: Int = 200,
): List<ClientLogRecord> {
var condition: Op<Boolean> = ClientLogsTable.userName eq userName
level?.let { condition = condition and (ClientLogsTable.level eq it) }
deviceId?.let { condition = condition and (ClientLogsTable.deviceId eq it) }
beforeId?.let { condition = condition and (ClientLogsTable.id less it) }
sinceMillis?.let {
val since = Instant.ofEpochMilli(it).toString()
condition = condition and (ClientLogsTable.receivedAt greaterEq since)
}
return transaction {
ClientLogsTable.selectAll()
.where(condition)
.orderBy(ClientLogsTable.id to SortOrder.DESC)
.limit(limit.coerceIn(1, 1_000))
.map { it.toRecord() }
}
}
fun countSince(userName: String, since: Instant): Long = transaction {
ClientLogsTable.selectAll()
.where {
(ClientLogsTable.userName eq userName) and
(ClientLogsTable.receivedAt greaterEq since.toString())
}
.count()
}
fun delete(userName: String): Int = transaction {
ClientLogsTable.deleteWhere { ClientLogsTable.userName eq userName }
}
/**
* Deletes rows older than [retentionDays] and trims every user to
* [maxRowsPerUser], newest first. Returns the number of deleted rows.
*/
fun prune(retentionDays: Int, maxRowsPerUser: Int): Int {
val cutoff = Instant.now()
.minus(retentionDays.coerceAtLeast(1).toLong(), ChronoUnit.DAYS)
.toString()
var deleted = transaction {
ClientLogsTable.deleteWhere { ClientLogsTable.receivedAt less cutoff }
}
val users = transaction {
ClientLogsTable.select(ClientLogsTable.userName)
.withDistinct()
.map { it[ClientLogsTable.userName] }
}
users.forEach { user -> deleted += trim(user, maxRowsPerUser) }
return deleted
}
private fun trim(userName: String, maxRows: Int): Int {
if (maxRows <= 0) return delete(userName)
return transaction {
val cutoff = ClientLogsTable.select(ClientLogsTable.id)
.where { ClientLogsTable.userName eq userName }
.orderBy(ClientLogsTable.id to SortOrder.DESC)
.limit(1).offset((maxRows - 1).toLong())
.map { it[ClientLogsTable.id] }
.firstOrNull() ?: return@transaction 0
ClientLogsTable.deleteWhere {
(ClientLogsTable.userName eq userName) and
(ClientLogsTable.id less cutoff)
}
}
}
private fun ResultRow.toRecord(): ClientLogRecord = ClientLogRecord(
id = this[ClientLogsTable.id],
deviceId = this[ClientLogsTable.deviceId],
platform = this[ClientLogsTable.platform],
appVersion = this[ClientLogsTable.appVersion],
sessionId = this[ClientLogsTable.sessionId],
level = this[ClientLogsTable.level],
tag = this[ClientLogsTable.tag],
message = this[ClientLogsTable.message],
detail = this[ClientLogsTable.detail],
createdAt = this[ClientLogsTable.createdAt],
receivedAt = this[ClientLogsTable.receivedAt],
)
}
@@ -6,45 +6,20 @@ import org.jetbrains.exposed.sql.insertIgnore
import org.jetbrains.exposed.sql.selectAll
import org.jetbrains.exposed.sql.transactions.transaction
import org.jetbrains.exposed.sql.update
import win.taptap.musicmanager.config.LibraryConfig
import win.taptap.musicmanager.config.UsersConfig
/**
* Reconciles the config/Vault-managed identity tables with SQLite.
* Reconciles the config/Vault-managed identity table with SQLite.
*
* Removing a folder cascades to its songs; removing a user cascades to that
* user's play history and mood ownership.
* Removing a user cascades to that user's moods, sessions and affinity. Songs
* are shared server-wide and are not tied to users.
*/
object ConfigRepository {
fun sync(library: LibraryConfig, users: UsersConfig) {
fun sync(users: UsersConfig) {
val now = Instant.now().toString()
transaction {
val folderIds = library.folders.map { it.id }.toSet()
val obsoleteFolderIds = MusicFoldersTable.selectAll()
.map { it[MusicFoldersTable.id] }
.filterNot { it in folderIds }
obsoleteFolderIds.forEach { folderId ->
MusicFoldersTable.deleteWhere {
it.run { MusicFoldersTable.id eq folderId }
}
}
library.folders.forEach { folder ->
MusicFoldersTable.insertIgnore {
it[id] = folder.id
it[name] = folder.name
it[type] = folder.type.name.lowercase()
it[rootPath] = folder.path.toString()
}
MusicFoldersTable.update({ MusicFoldersTable.id eq folder.id }) {
it[name] = folder.name
it[type] = folder.type.name.lowercase()
it[rootPath] = folder.path.toString()
}
}
val userNames = users.users.map { it.name }.toSet()
val obsoleteUserNames = UsersTable.selectAll()
.map { it[UsersTable.name] }
@@ -62,12 +62,33 @@ object DatabaseFactory {
}
}
normalizeLegacyMoodStates(dataSource)
val database = Database.connect(dataSource)
migrate(dataSource)
return DatabaseHandle(dataSource, database)
}
/**
* V4 allowed `liked` and `downloaded` recommendations, but its CHECK
* constraint rejects the V5 `accepted` state, so an in-place rename is not
* possible. Move those rows back to `new` (allowed by both schemas) before
* Flyway runs. The table is absent on a fresh database and already uses the
* new states after V5, so both are harmless no-ops.
*/
private fun normalizeLegacyMoodStates(dataSource: HikariDataSource) {
dataSource.connection.use { connection ->
runCatching {
connection.createStatement().use { statement ->
statement.executeUpdate(
"UPDATE mood_recommendations " +
"SET state = 'new' WHERE state IN ('liked', 'downloaded')",
)
}
}
}
}
fun migrate(dataSource: HikariDataSource) {
Flyway.configure()
.dataSource(dataSource)
@@ -75,4 +96,5 @@ object DatabaseFactory {
.load()
.migrate()
}
}
@@ -0,0 +1,225 @@
package win.taptap.musicmanager.db
import java.time.Instant
import org.jetbrains.exposed.sql.ResultRow
import org.jetbrains.exposed.sql.SortOrder
import org.jetbrains.exposed.sql.SqlExpressionBuilder.eq
import org.jetbrains.exposed.sql.SqlExpressionBuilder.inList
import org.jetbrains.exposed.sql.SqlExpressionBuilder.less
import org.jetbrains.exposed.sql.and
import org.jetbrains.exposed.sql.insertIgnore
import org.jetbrains.exposed.sql.selectAll
import org.jetbrains.exposed.sql.transactions.transaction
import org.jetbrains.exposed.sql.update
data class DownloadJobRecord(
val id: String,
val source: String,
val externalId: String,
val moodId: String,
val kind: String,
val state: String,
val priority: Int,
val attempts: Int,
val error: String?,
val songId: String?,
val createdAt: String,
val updatedAt: String,
)
/**
* A single global download queue. Manual song adds use a higher priority than
* automatic recommendation downloads so a user's explicit action never waits
* behind a background refill.
*/
object DownloadRepository {
const val KIND_SONG = "song"
const val KIND_RECOMMENDATION = "recommendation"
const val PRIORITY_SONG = 100
const val PRIORITY_RECOMMENDATION = 0
fun enqueue(
source: String,
externalId: String,
moodId: String,
kind: String,
priority: Int = if (kind == KIND_SONG) PRIORITY_SONG else PRIORITY_RECOMMENDATION,
): DownloadJobRecord = transaction {
val now = Instant.now().toString()
val id = "dl-" + ApiTokenRepository.hash("$source|$moodId|$kind|$externalId").take(24)
val existing = DownloadJobsTable.selectAll()
.where { DownloadJobsTable.id eq id }
.singleOrNull()
if (existing == null) {
DownloadJobsTable.insertIgnore {
it[DownloadJobsTable.id] = id
it[DownloadJobsTable.musicSource] = source
it[DownloadJobsTable.externalId] = externalId
it[this.moodId] = moodId
it[this.kind] = kind
it[state] = "queued"
it[this.priority] = priority
it[attempts] = 0
it[createdAt] = now
it[updatedAt] = now
}
} else if (existing[DownloadJobsTable.state] != "queued" &&
existing[DownloadJobsTable.state] != "downloading"
) {
// A previous attempt failed, or a finished job no longer has a file
// (the song was removed and garbage-collected). Start it again.
DownloadJobsTable.update({ DownloadJobsTable.id eq id }) {
it[state] = "queued"
it[attempts] = 0
it[error] = null
it[this.songId] = null
it[this.priority] = priority
it[updatedAt] = now
}
}
DownloadJobsTable.selectAll()
.where { DownloadJobsTable.id eq id }
.single()
.toRecord()
}
/** Takes the highest-priority queued job and marks it as downloading. */
fun claimNext(): DownloadJobRecord? = transaction {
recoverStale()
val row = DownloadJobsTable.selectAll()
.where { DownloadJobsTable.state eq "queued" }
.orderBy(
DownloadJobsTable.priority to SortOrder.DESC,
DownloadJobsTable.createdAt to SortOrder.ASC,
)
.limit(1)
.singleOrNull()
?: return@transaction null
val attempts = row[DownloadJobsTable.attempts] + 1
DownloadJobsTable.update({ DownloadJobsTable.id eq row[DownloadJobsTable.id] }) {
it[state] = "downloading"
it[DownloadJobsTable.attempts] = attempts
it[updatedAt] = Instant.now().toString()
}
row.toRecord().copy(state = "downloading", attempts = attempts)
}
fun markDone(id: String, songId: String) {
transaction {
// Only finish a job that is still ours. A reject or recovery may
// have changed the row while the downloader was running.
DownloadJobsTable.update({
(DownloadJobsTable.id eq id) and
(DownloadJobsTable.state eq "downloading")
}) {
it[state] = "done"
it[this.songId] = songId
it[error] = null
it[updatedAt] = Instant.now().toString()
}
}
}
fun markFailed(id: String, error: String) {
transaction {
DownloadJobsTable.update({ DownloadJobsTable.id eq id }) {
it[state] = "failed"
it[this.error] = error.take(1000)
it[updatedAt] = Instant.now().toString()
}
}
}
fun retryLater(id: String, error: String) {
transaction {
DownloadJobsTable.update({ DownloadJobsTable.id eq id }) {
it[state] = "queued"
it[this.error] = error.take(1000)
it[updatedAt] = Instant.now().toString()
}
}
}
/** A reject must stop a queued or in-flight download. */
fun cancelForMoodTrack(moodId: String, externalId: String, reason: String) {
transaction {
DownloadJobsTable.update({
(DownloadJobsTable.moodId eq moodId) and
(DownloadJobsTable.externalId eq externalId) and
(DownloadJobsTable.state inList listOf("queued", "downloading"))
}) {
it[state] = "failed"
it[error] = reason.take(1000)
it[updatedAt] = Instant.now().toString()
}
}
}
fun find(moodId: String, externalId: String, kind: String): DownloadJobRecord? = transaction {
DownloadJobsTable.selectAll()
.where {
(DownloadJobsTable.moodId eq moodId) and
(DownloadJobsTable.externalId eq externalId) and
(DownloadJobsTable.kind eq kind)
}
.singleOrNull()
?.toRecord()
}
fun forMood(moodId: String): List<DownloadJobRecord> = transaction {
DownloadJobsTable.selectAll()
.where { DownloadJobsTable.moodId eq moodId }
.orderBy(DownloadJobsTable.createdAt to SortOrder.DESC)
.map { it.toRecord() }
}
fun pendingCount(): Long = transaction {
DownloadJobsTable.selectAll()
.where { DownloadJobsTable.state inList listOf("queued", "downloading") }
.count()
}
/**
* A pod restart or an uncaught bookkeeping failure can leave jobs in
* `downloading`. Requeue them after a generous timeout so they are not lost.
*/
private fun recoverStale(now: Instant = Instant.now()) {
val cutoff = now.minusSeconds(STALE_DOWNLOAD_SECONDS).toString()
val stale = DownloadJobsTable.selectAll()
.where {
(DownloadJobsTable.state eq "downloading") and
(DownloadJobsTable.updatedAt less cutoff)
}
.toList()
stale.forEach { row ->
val id = row[DownloadJobsTable.id]
DownloadJobsTable.update({ DownloadJobsTable.id eq id }) {
it[state] = "queued"
it[error] = "Download was interrupted; retrying"
it[updatedAt] = now.toString()
}
}
}
private const val STALE_DOWNLOAD_SECONDS = 30 * 60L
private fun ResultRow.toRecord() = DownloadJobRecord(
id = this[DownloadJobsTable.id],
source = this[DownloadJobsTable.musicSource],
externalId = this[DownloadJobsTable.externalId],
moodId = this[DownloadJobsTable.moodId],
kind = this[DownloadJobsTable.kind],
state = this[DownloadJobsTable.state],
priority = this[DownloadJobsTable.priority],
attempts = this[DownloadJobsTable.attempts],
error = this[DownloadJobsTable.error],
songId = this[DownloadJobsTable.songId],
createdAt = this[DownloadJobsTable.createdAt],
updatedAt = this[DownloadJobsTable.updatedAt],
)
}
@@ -1,876 +0,0 @@
package win.taptap.musicmanager.db
import java.nio.file.Files
import java.nio.file.Path
import org.jetbrains.exposed.sql.JoinType
import org.jetbrains.exposed.sql.Op
import org.jetbrains.exposed.sql.SqlExpressionBuilder.greater
import org.jetbrains.exposed.sql.ResultRow
import org.jetbrains.exposed.sql.SortOrder
import org.jetbrains.exposed.sql.Transaction
import org.jetbrains.exposed.sql.and
import org.jetbrains.exposed.sql.lowerCase
import org.jetbrains.exposed.sql.or
import org.jetbrains.exposed.sql.selectAll
import org.jetbrains.exposed.sql.transactions.transaction
import org.jetbrains.exposed.sql.union
import win.taptap.musicmanager.config.LibraryConfig
import win.taptap.musicmanager.util.IdGenerator
data class ArtistRefRecord(val id: String, val name: String)
data class UserItemState(val starredAt: String?, val rating: Int?)
data class UserSongState(
val playCount: Int,
val lastPlayedAt: String?,
val starredAt: String?,
val rating: Int?,
)
data class GenreRecord(val value: String, val songCount: Int, val albumCount: Int)
data class StarredRecords(
val artists: List<ArtistRecord>,
val albums: List<AlbumRecord>,
val songs: List<SongRecord>,
)
data class ArtistRecord(
val id: String,
val name: String,
val sortName: String,
val albumCount: Int,
val starredAt: String?,
val rating: Int?,
)
data class AlbumRecord(
val id: String,
val name: String,
val artistId: String?,
val artistName: String?,
val year: Int?,
val coverArt: String?,
val songCount: Int,
val duration: Int,
val playCount: Int,
val lastPlayedAt: String?,
val created: String,
val starredAt: String?,
val rating: Int?,
val genres: List<String>,
)
data class SongRecord(
val id: String,
val albumId: String,
val albumName: String,
val artistId: String?,
val artistName: String,
val displayArtist: String,
val title: String,
val trackNumber: Int?,
val discNumber: Int,
val year: Int?,
val duration: Int,
val bitRate: Int,
val size: Long,
val suffix: String,
val contentType: String,
val coverArt: String?,
val folderId: Int,
val relativePath: String,
val created: String,
val updatedAt: String,
val starredAt: String?,
val rating: Int?,
val playCount: Int,
val lastPlayedAt: String?,
val artists: List<ArtistRefRecord>,
val genres: List<String>,
)
data class ResolvedSong(val song: SongRecord, val file: Path)
data class TombstoneRecord(val entityId: String, val deletedAt: String)
/**
* Read-side queries for browsing and media. Every method takes the set of folder
* IDs visible to the requesting user; an album or artist without a visible song
* is never returned.
*/
object LibraryRepository {
fun artists(folderIds: Set<Int>, userName: String): List<ArtistRecord> = read {
if (folderIds.isEmpty()) return@read emptyList()
val visibleSongs = visibleSongs(folderIds)
val visibleAlbums = visibleAlbums(folderIds)
val trackArtists = SongArtistsTable
.select(SongArtistsTable.artistId)
.where { SongArtistsTable.songId inSubQuery visibleSongs }
val albumArtists = AlbumArtistsTable
.select(AlbumArtistsTable.artistId)
.where { AlbumArtistsTable.albumId inSubQuery visibleAlbums }
val albumCounts = AlbumArtistsTable
.select(AlbumArtistsTable.artistId, AlbumArtistsTable.albumId)
.withDistinct()
.where { AlbumArtistsTable.albumId inSubQuery visibleAlbums }
.groupingBy { it[AlbumArtistsTable.artistId] }
.eachCount()
val states = itemStates(userName, "artist")
return@read ArtistsTable
.selectAll()
.where { ArtistsTable.id inSubQuery trackArtists.union(albumArtists) }
.map { row ->
val id = row[ArtistsTable.id]
val state = states[id]
ArtistRecord(
id = id,
name = row[ArtistsTable.name],
sortName = row[ArtistsTable.sortName],
albumCount = albumCounts[id] ?: 0,
starredAt = state?.starredAt,
rating = state?.rating,
)
}
.sortedBy { IdGenerator.normalize(it.sortName) }
}
fun artist(artistId: String, folderIds: Set<Int>, userName: String): ArtistRecord? = read {
if (folderIds.isEmpty()) return@read null
val row = ArtistsTable.selectAll()
.where { ArtistsTable.id eq artistId }
.singleOrNull() ?: return@read null
val albumCount = AlbumArtistsTable
.select(AlbumArtistsTable.albumId)
.withDistinct()
.where {
(AlbumArtistsTable.artistId eq artistId) and
(AlbumArtistsTable.albumId inSubQuery visibleAlbums(folderIds))
}
.count()
.toInt()
val hasVisibleSong = SongArtistsTable
.select(SongArtistsTable.songId)
.where {
(SongArtistsTable.artistId eq artistId) and
(SongArtistsTable.songId inSubQuery visibleSongs(folderIds))
}
.any()
if (albumCount == 0 && !hasVisibleSong) return@read null
val state = itemStates(userName, "artist")[artistId]
ArtistRecord(
id = artistId,
name = row[ArtistsTable.name],
sortName = row[ArtistsTable.sortName],
albumCount = albumCount,
starredAt = state?.starredAt,
rating = state?.rating,
)
}
fun albumsForArtist(artistId: String, folderIds: Set<Int>, userName: String): List<AlbumRecord> = read {
if (folderIds.isEmpty()) return@read emptyList()
val albumIds = AlbumArtistsTable
.select(AlbumArtistsTable.albumId)
.withDistinct()
.where {
(AlbumArtistsTable.artistId eq artistId) and
(AlbumArtistsTable.albumId inSubQuery visibleAlbums(folderIds))
}
.map { it[AlbumArtistsTable.albumId] }
albumIds.mapNotNull { albumRecord(it, folderIds, userName) }
.sortedBy { it.name.lowercase() }
}
fun album(albumId: String, folderIds: Set<Int>, userName: String): AlbumRecord? = read {
albumRecord(albumId, folderIds, userName)
}
fun albumSongs(albumId: String, folderIds: Set<Int>, userName: String): List<SongRecord> = read {
if (folderIds.isEmpty()) return@read emptyList()
val rows = SongsTable.selectAll()
.where {
(SongsTable.albumId eq albumId) and
(SongsTable.folderId inList folderIds)
}
.orderBy(
SongsTable.discNumber to SortOrder.ASC,
SongsTable.trackNumber to SortOrder.ASC,
SongsTable.title to SortOrder.ASC,
)
.toList()
enrichSongs(rows, userName)
}
fun song(songId: String, folderIds: Set<Int>, userName: String): SongRecord? = read {
if (folderIds.isEmpty()) return@read null
val row = SongsTable.selectAll()
.where {
(SongsTable.id eq songId) and
(SongsTable.folderId inList folderIds)
}
.singleOrNull() ?: return@read null
enrichSongs(listOf(row), userName).single()
}
fun allAlbums(limit: Int, offset: Int, folderIds: Set<Int>, userName: String): List<AlbumRecord> = read {
if (folderIds.isEmpty()) return@read emptyList()
val ids = AlbumsTable
.select(AlbumsTable.id)
.where { AlbumsTable.id inSubQuery visibleAlbums(folderIds) }
.orderBy(AlbumsTable.name to SortOrder.ASC)
.limit(limit)
.offset(offset.toLong())
.map { it[AlbumsTable.id] }
ids.mapNotNull { albumRecord(it, folderIds, userName) }
}
/** Stable page of the library snapshot, ordered by song ID. */
fun syncSongs(
folderIds: Set<Int>,
afterId: String,
limit: Int,
userName: String,
): List<SongRecord> = read {
if (folderIds.isEmpty()) return@read emptyList()
val rows = SongsTable.selectAll()
.where {
(SongsTable.folderId inList folderIds) and
(SongsTable.id greater afterId)
}
.orderBy(SongsTable.id to SortOrder.ASC)
.limit(limit)
.toList()
enrichSongs(rows, userName)
}
/** Songs whose metadata changed after the client's last revision. */
fun changedSongs(
since: String,
folderIds: Set<Int>,
limit: Int,
userName: String,
): List<SongRecord> = read {
if (folderIds.isEmpty()) return@read emptyList()
val rows = SongsTable.selectAll()
.where {
(SongsTable.folderId inList folderIds) and
(SongsTable.updatedAt greater since)
}
.orderBy(
SongsTable.updatedAt to SortOrder.ASC,
SongsTable.id to SortOrder.ASC,
)
.limit(limit)
.toList()
enrichSongs(rows, userName)
}
fun tombstones(since: String, limit: Int): List<TombstoneRecord> = read {
LibraryTombstonesTable.selectAll()
.where {
(LibraryTombstonesTable.entity eq "song") and
(LibraryTombstonesTable.deletedAt greater since)
}
.orderBy(LibraryTombstonesTable.deletedAt to SortOrder.ASC)
.limit(limit)
.map {
TombstoneRecord(
entityId = it[LibraryTombstonesTable.entityId],
deletedAt = it[LibraryTombstonesTable.deletedAt],
)
}
}
fun allSongs(limit: Int, offset: Int, folderIds: Set<Int>, userName: String): List<SongRecord> = read {
if (folderIds.isEmpty()) return@read emptyList()
val rows = SongsTable.selectAll()
.where { SongsTable.folderId inList folderIds }
.orderBy(SongsTable.title to SortOrder.ASC)
.limit(limit)
.offset(offset.toLong())
.toList()
enrichSongs(rows, userName)
}
/**
* Album feed used by getAlbumList2. Sorting happens after visibility filtering so
* per-user play counts and stars are correct.
*/
fun albumList(
type: String,
size: Int,
offset: Int,
fromYear: Int?,
toYear: Int?,
genre: String?,
folderIds: Set<Int>,
userName: String,
): List<AlbumRecord> = read {
if (folderIds.isEmpty()) return@read emptyList()
val ids = AlbumsTable
.select(AlbumsTable.id)
.where { AlbumsTable.id inSubQuery visibleAlbums(folderIds) }
.map { it[AlbumsTable.id] }
// Clients may send the range inverted (e.g. descending by year), so normalize it.
val lowerYear = when {
fromYear != null && toYear != null && fromYear > toYear -> toYear
else -> fromYear
}
val upperYear = when {
fromYear != null && toYear != null && fromYear > toYear -> fromYear
else -> toYear
}
var albums = ids.mapNotNull { albumRecord(it, folderIds, userName) }
if (lowerYear != null) albums = albums.filter { (it.year ?: Int.MIN_VALUE) >= lowerYear }
if (upperYear != null) albums = albums.filter { (it.year ?: Int.MAX_VALUE) <= upperYear }
if (!genre.isNullOrBlank()) {
albums = albums.filter { album -> album.genres.any { it.equals(genre, ignoreCase = true) } }
}
val sorted = when (type) {
"newest" -> albums.sortedByDescending { it.created }
"recent" -> albums.sortedWith(
compareByDescending<AlbumRecord> { it.lastPlayedAt ?: "" }
.thenByDescending { it.created },
)
"frequent" -> albums.sortedWith(
compareByDescending<AlbumRecord> { it.playCount }
.thenBy { IdGenerator.normalize(it.name) },
)
"random" -> albums.shuffled()
"alphabeticalByName" -> albums.sortedBy { IdGenerator.normalize(it.name) }
"alphabeticalByArtist" -> albums.sortedWith(
compareBy(
{ IdGenerator.normalize(it.artistName.orEmpty()) },
{ IdGenerator.normalize(it.name) },
),
)
"starred" -> albums
.filter { it.starredAt != null }
.sortedByDescending { it.starredAt }
"byYear" -> albums.sortedWith(
compareByDescending<AlbumRecord> { it.year ?: 0 }
.thenBy { IdGenerator.normalize(it.name) },
)
"byGenre" -> albums.sortedBy { IdGenerator.normalize(it.name) }
else -> albums.sortedBy { IdGenerator.normalize(it.name) }
}
sorted.drop(offset).take(size)
}
fun randomSongs(
size: Int,
genre: String?,
fromYear: Int?,
toYear: Int?,
folderIds: Set<Int>,
userName: String,
): List<SongRecord> = read {
if (folderIds.isEmpty()) return@read emptyList()
val lowerYear = when {
fromYear != null && toYear != null && fromYear > toYear -> toYear
else -> fromYear
}
val upperYear = when {
fromYear != null && toYear != null && fromYear > toYear -> fromYear
else -> toYear
}
val rows = SongsTable.selectAll()
.where {
(SongsTable.folderId inList folderIds) and
(lowerYear?.let { SongsTable.year greaterEq it } ?: Op.TRUE) and
(upperYear?.let { SongsTable.year lessEq it } ?: Op.TRUE)
}
.toList()
var songs = enrichSongs(rows, userName)
if (!genre.isNullOrBlank()) {
songs = songs.filter { song -> song.genres.any { it.equals(genre, ignoreCase = true) } }
}
songs.shuffled().take(size)
}
fun genres(folderIds: Set<Int>): List<GenreRecord> = read {
if (folderIds.isEmpty()) return@read emptyList()
val rows = SongGenresTable
.join(SongsTable, JoinType.INNER, SongGenresTable.songId, SongsTable.id)
.select(SongGenresTable.genre, SongsTable.id, SongsTable.albumId)
.where { SongsTable.folderId inList folderIds }
.toList()
rows.groupBy { it[SongGenresTable.genre] }
.map { (genre, genreRows) ->
GenreRecord(
value = genre,
songCount = genreRows.map { it[SongsTable.id] }.distinct().size,
albumCount = genreRows.map { it[SongsTable.albumId] }.distinct().size,
)
}
.sortedBy { IdGenerator.normalize(it.value) }
}
fun songsByGenre(
genre: String,
size: Int,
offset: Int,
folderIds: Set<Int>,
userName: String,
): List<SongRecord> = read {
if (folderIds.isEmpty()) return@read emptyList()
val songIds = SongGenresTable
.select(SongGenresTable.songId)
.where { SongGenresTable.genre.lowerCase() eq genre.lowercase() }
val rows = SongsTable.selectAll()
.where {
(SongsTable.id inSubQuery songIds) and
(SongsTable.folderId inList folderIds)
}
.orderBy(SongsTable.title to SortOrder.ASC)
.limit(size)
.offset(offset.toLong())
.toList()
enrichSongs(rows, userName)
}
/** Bulk genre lookup used by mood generation to avoid one query per seed genre. */
fun songsByGenres(
genres: Collection<String>,
limit: Int,
folderIds: Set<Int>,
userName: String,
): List<SongRecord> = read {
if (genres.isEmpty() || folderIds.isEmpty()) return@read emptyList()
val normalized = genres.map { it.lowercase() }.distinct()
val songIds = SongGenresTable
.select(SongGenresTable.songId)
.where { SongGenresTable.genre.lowerCase() inList normalized }
val rows = SongsTable.selectAll()
.where {
(SongsTable.id inSubQuery songIds) and
(SongsTable.folderId inList folderIds)
}
.limit(limit)
.toList()
enrichSongs(rows, userName)
}
/** Songs by track artist or album artist, used as related-mood candidates. */
fun songsByArtists(
artistIds: Collection<String>,
limit: Int,
folderIds: Set<Int>,
userName: String,
): List<SongRecord> = read {
if (artistIds.isEmpty() || folderIds.isEmpty()) return@read emptyList()
val ids = artistIds.toList()
val songArtists = SongArtistsTable
.select(SongArtistsTable.songId)
.where { SongArtistsTable.artistId inList ids }
val albumArtists = AlbumArtistsTable
.select(AlbumArtistsTable.albumId)
.where { AlbumArtistsTable.artistId inList ids }
val rows = SongsTable.selectAll()
.where {
(SongsTable.folderId inList folderIds) and
((SongsTable.id inSubQuery songArtists) or (SongsTable.albumId inSubQuery albumArtists))
}
.limit(limit)
.toList()
enrichSongs(rows, userName)
}
fun starred(userName: String, folderIds: Set<Int>): StarredRecords = read {
if (folderIds.isEmpty()) {
return@read StarredRecords(emptyList(), emptyList(), emptyList())
}
val albumStates = itemStates(userName, "album")
val albums = albumStates
.filterValues { it.starredAt != null }
.keys
.mapNotNull { albumRecord(it, folderIds, userName) }
.sortedByDescending { it.starredAt }
val artistStates = itemStates(userName, "artist")
val artists = artistStates
.filterValues { it.starredAt != null }
.keys
.mapNotNull { artist(it, folderIds, userName) }
.sortedByDescending { it.starredAt }
val songStates = UserSongStateTable.selectAll()
.where {
(UserSongStateTable.userName eq userName) and
UserSongStateTable.starredAt.isNotNull()
}
.toList()
val songRows = if (songStates.isEmpty()) {
emptyList()
} else {
SongsTable.selectAll()
.where {
(SongsTable.id inList songStates.map { it[UserSongStateTable.songId] }) and
(SongsTable.folderId inList folderIds)
}
.toList()
}
val songs = enrichSongs(songRows, userName)
.sortedByDescending { it.starredAt }
StarredRecords(artists, albums, songs)
}
fun lyrics(songId: String, folderIds: Set<Int>): String? = read {
if (folderIds.isEmpty()) return@read null
SongsTable
.select(SongsTable.lyrics)
.where {
(SongsTable.id eq songId) and
(SongsTable.folderId inList folderIds)
}
.singleOrNull()
?.get(SongsTable.lyrics)
}
fun songsByIds(ids: List<String>, folderIds: Set<Int>, userName: String): List<SongRecord> = read {
if (ids.isEmpty() || folderIds.isEmpty()) return@read emptyList()
val rows = SongsTable.selectAll()
.where {
(SongsTable.id inList ids) and
(SongsTable.folderId inList folderIds)
}
.toList()
val byId = enrichSongs(rows, userName).associateBy { it.id }
ids.mapNotNull { byId[it] }
}
fun searchArtists(
query: String,
limit: Int,
offset: Int,
folderIds: Set<Int>,
userName: String,
): List<ArtistRecord> = read {
if (folderIds.isEmpty()) return@read emptyList()
val pattern = likePattern(query)
val visibleSongs = visibleSongs(folderIds)
val visibleAlbums = visibleAlbums(folderIds)
val trackArtists = SongArtistsTable.select(SongArtistsTable.artistId)
.where { SongArtistsTable.songId inSubQuery visibleSongs }
val albumArtists = AlbumArtistsTable.select(AlbumArtistsTable.artistId)
.where { AlbumArtistsTable.albumId inSubQuery visibleAlbums }
val ids = ArtistsTable
.select(ArtistsTable.id)
.where {
(ArtistsTable.searchText like pattern) and
(ArtistsTable.id inSubQuery trackArtists.union(albumArtists))
}
.orderBy(ArtistsTable.sortName to SortOrder.ASC)
.limit(limit)
.offset(offset.toLong())
.map { it[ArtistsTable.id] }
ids.mapNotNull { artist(it, folderIds, userName) }
}
fun searchAlbums(
query: String,
limit: Int,
offset: Int,
folderIds: Set<Int>,
userName: String,
): List<AlbumRecord> = read {
if (folderIds.isEmpty()) return@read emptyList()
val pattern = likePattern(query)
val ids = AlbumsTable
.select(AlbumsTable.id)
.where {
(AlbumsTable.searchText like pattern) and
(AlbumsTable.id inSubQuery visibleAlbums(folderIds))
}
.orderBy(AlbumsTable.name to SortOrder.ASC)
.limit(limit)
.offset(offset.toLong())
.map { it[AlbumsTable.id] }
ids.mapNotNull { albumRecord(it, folderIds, userName) }
}
fun searchSongs(
query: String,
limit: Int,
offset: Int,
folderIds: Set<Int>,
userName: String,
): List<SongRecord> = read {
if (folderIds.isEmpty()) return@read emptyList()
val pattern = likePattern(query)
val rows = SongsTable.selectAll()
.where {
(SongsTable.searchText like pattern) and
(SongsTable.folderId inList folderIds)
}
.orderBy(SongsTable.title to SortOrder.ASC)
.limit(limit)
.offset(offset.toLong())
.toList()
enrichSongs(rows, userName)
}
fun resolveSong(
songId: String,
folderIds: Set<Int>,
userName: String,
library: LibraryConfig,
): ResolvedSong? {
val song = song(songId, folderIds, userName) ?: return null
val folder = library.byId(song.folderId) ?: return null
val file = folder.path.resolve(song.relativePath).normalize()
if (!file.startsWith(folder.path)) return null
if (!Files.isRegularFile(file)) return null
return ResolvedSong(song, file)
}
fun albumIdForSong(songId: String, folderIds: Set<Int>): String? = read {
if (folderIds.isEmpty()) return@read null
SongsTable
.select(SongsTable.albumId)
.where {
(SongsTable.id eq songId) and
(SongsTable.folderId inList folderIds)
}
.singleOrNull()
?.get(SongsTable.albumId)
}
private fun albumRecord(albumId: String, folderIds: Set<Int>, userName: String): AlbumRecord? {
if (folderIds.isEmpty()) return null
val albumRow = AlbumsTable.selectAll()
.where { AlbumsTable.id eq albumId }
.singleOrNull() ?: return null
val songs = SongsTable.selectAll()
.where {
(SongsTable.albumId eq albumId) and
(SongsTable.folderId inList folderIds)
}
.toList()
if (songs.isEmpty()) return null
val primaryArtist = AlbumArtistsTable
.join(ArtistsTable, JoinType.INNER, AlbumArtistsTable.artistId, ArtistsTable.id)
.select(AlbumArtistsTable.albumId, ArtistsTable.id, ArtistsTable.name, AlbumArtistsTable.position)
.where { AlbumArtistsTable.albumId eq albumId }
.orderBy(AlbumArtistsTable.position to SortOrder.ASC)
.firstOrNull()
val states = songStates(userName, songs.map { it[SongsTable.id] })
val playCount = states.values.sumOf { it.playCount }
val lastPlayedAt = states.values.mapNotNull { it.lastPlayedAt }.maxOrNull()
val state = itemStates(userName, "album")[albumId]
return AlbumRecord(
id = albumId,
name = albumRow[AlbumsTable.name],
artistId = primaryArtist?.get(ArtistsTable.id),
artistName = primaryArtist?.get(ArtistsTable.name),
year = albumRow[AlbumsTable.year],
coverArt = albumRow[AlbumsTable.coverArt],
songCount = songs.size,
duration = songs.sumOf { it[SongsTable.duration] },
playCount = playCount,
lastPlayedAt = lastPlayedAt,
created = albumRow[AlbumsTable.createdAt],
starredAt = state?.starredAt,
rating = state?.rating,
genres = AlbumGenresTable
.select(AlbumGenresTable.genre)
.where { AlbumGenresTable.albumId eq albumId }
.map { it[AlbumGenresTable.genre] }
.sorted(),
)
}
private fun enrichSongs(rows: List<ResultRow>, userName: String): List<SongRecord> {
if (rows.isEmpty()) return emptyList()
val songIds = rows.map { it[SongsTable.id] }
val albumIds = rows.map { it[SongsTable.albumId] }.distinct()
val songArtists = SongArtistsTable
.join(ArtistsTable, JoinType.INNER, SongArtistsTable.artistId, ArtistsTable.id)
.select(SongArtistsTable.songId, ArtistsTable.id, ArtistsTable.name, SongArtistsTable.position)
.where { SongArtistsTable.songId inList songIds }
.orderBy(SongArtistsTable.position to SortOrder.ASC)
.groupBy { it[SongArtistsTable.songId] }
.mapValues { (_, artistRows) ->
artistRows.map { ArtistRefRecord(it[ArtistsTable.id], it[ArtistsTable.name]) }
}
val albumArtists = AlbumArtistsTable
.join(ArtistsTable, JoinType.INNER, AlbumArtistsTable.artistId, ArtistsTable.id)
.select(AlbumArtistsTable.albumId, ArtistsTable.id, ArtistsTable.name, AlbumArtistsTable.position)
.where { AlbumArtistsTable.albumId inList albumIds }
.orderBy(AlbumArtistsTable.position to SortOrder.ASC)
.groupBy { it[AlbumArtistsTable.albumId] }
.mapValues { (_, artistRows) ->
ArtistRefRecord(artistRows.first()[ArtistsTable.id], artistRows.first()[ArtistsTable.name])
}
val covers = AlbumsTable.selectAll()
.where { AlbumsTable.id inList albumIds }
.associate { it[AlbumsTable.id] to it[AlbumsTable.coverArt] }
val genres = SongGenresTable.selectAll()
.where { SongGenresTable.songId inList songIds }
.groupBy { it[SongGenresTable.songId] }
.mapValues { (_, genreRows) -> genreRows.map { it[SongGenresTable.genre] }.sorted() }
val states = songStates(userName, songIds)
return rows.map { row ->
val id = row[SongsTable.id]
val albumId = row[SongsTable.albumId]
val primaryArtist = albumArtists[albumId]
val state = states[id]
SongRecord(
id = id,
albumId = albumId,
albumName = row[SongsTable.albumName],
artistId = primaryArtist?.id,
artistName = row[SongsTable.displayArtist],
displayArtist = row[SongsTable.displayArtist],
title = row[SongsTable.title],
trackNumber = row[SongsTable.trackNumber],
discNumber = row[SongsTable.discNumber],
year = row[SongsTable.year],
duration = row[SongsTable.duration],
bitRate = row[SongsTable.bitRate],
size = row[SongsTable.size],
suffix = row[SongsTable.suffix],
contentType = row[SongsTable.contentType],
coverArt = covers[albumId],
folderId = row[SongsTable.folderId],
relativePath = row[SongsTable.relativePath],
created = row[SongsTable.createdAt],
updatedAt = row[SongsTable.updatedAt].ifBlank { row[SongsTable.createdAt] },
starredAt = state?.starredAt,
rating = state?.rating,
playCount = state?.playCount ?: 0,
lastPlayedAt = state?.lastPlayedAt,
artists = songArtists[id].orEmpty(),
genres = genres[id].orEmpty(),
)
}
}
private fun visibleSongs(folderIds: Set<Int>) =
SongsTable
.select(SongsTable.id)
.where { SongsTable.folderId inList folderIds }
private fun visibleAlbums(folderIds: Set<Int>) =
SongsTable
.select(SongsTable.albumId)
.withDistinct()
.where { SongsTable.folderId inList folderIds }
private fun itemStates(userName: String, itemType: String): Map<String, UserItemState> =
UserStarsTable.selectAll()
.where {
(UserStarsTable.userName eq userName) and
(UserStarsTable.itemType eq itemType)
}
.associate {
it[UserStarsTable.itemId] to UserItemState(
starredAt = it[UserStarsTable.starredAt].takeIf { value -> value.isNotBlank() },
rating = it[UserStarsTable.rating],
)
}
private fun songStates(userName: String, songIds: Collection<String>): Map<String, UserSongState> {
if (songIds.isEmpty()) return emptyMap()
return UserSongStateTable.selectAll()
.where {
(UserSongStateTable.userName eq userName) and
(UserSongStateTable.songId inList songIds.toList())
}
.associate {
it[UserSongStateTable.songId] to UserSongState(
playCount = it[UserSongStateTable.playCount],
lastPlayedAt = it[UserSongStateTable.lastPlayedAt],
starredAt = it[UserSongStateTable.starredAt],
rating = it[UserSongStateTable.rating],
)
}
}
/** Literal substring matching; `%` and `_` are removed instead of acting as wildcards. */
private fun likePattern(query: String): String {
val needle = IdGenerator.normalize(query).replace("%", "").replace("_", "")
return "%$needle%"
}
private fun <T> read(block: Transaction.() -> T): T = transaction { block() }
}
@@ -1,243 +0,0 @@
package win.taptap.musicmanager.db
import java.time.Instant
import org.jetbrains.exposed.sql.and
import org.jetbrains.exposed.sql.deleteWhere
import org.jetbrains.exposed.sql.insert
import org.jetbrains.exposed.sql.insertIgnore
import org.jetbrains.exposed.sql.selectAll
import org.jetbrains.exposed.sql.transactions.transaction
import org.jetbrains.exposed.sql.update
import win.taptap.musicmanager.scanner.ArtistRef
import win.taptap.musicmanager.scanner.ExtractedTrack
import win.taptap.musicmanager.util.IdGenerator
import win.taptap.musicmanager.util.SortNames
data class ExistingSong(
val id: String,
val fileMtime: Long,
val size: Long,
)
/** Persistence for the scanner. Per-user state is never touched when metadata is updated. */
object MediaRepository {
fun findExisting(folderId: Int, relativePath: String): ExistingSong? = transaction {
SongsTable.selectAll()
.where {
(SongsTable.folderId eq folderId) and (SongsTable.relativePath eq relativePath)
}
.singleOrNull()
?.let {
ExistingSong(
id = it[SongsTable.id],
fileMtime = it[SongsTable.fileMtime],
size = it[SongsTable.size],
)
}
}
fun upsertTrack(track: ExtractedTrack) {
val now = Instant.now().toString()
transaction {
(track.albumArtists + track.artists).distinctBy { it.id }.forEach(::upsertArtist)
val albumSearch = IdGenerator.normalize(
"${track.albumTitle} ${track.albumArtists.joinToString(" ") { it.name }}",
)
val albumExists = AlbumsTable.selectAll()
.where { AlbumsTable.id eq track.albumId }
.any()
if (albumExists) {
AlbumsTable.update({ AlbumsTable.id eq track.albumId }) {
it[name] = track.albumTitle
if (track.year != null) it[year] = track.year
it[searchText] = albumSearch
}
} else {
AlbumsTable.insert {
it[id] = track.albumId
it[name] = track.albumTitle
it[year] = track.year
it[createdAt] = now
it[searchText] = albumSearch
}
}
AlbumArtistsTable.deleteWhere {
it.run { AlbumArtistsTable.albumId eq track.albumId }
}
track.albumArtists.forEachIndexed { index, artist ->
AlbumArtistsTable.insertIgnore {
it[albumId] = track.albumId
it[artistId] = artist.id
it[position] = index
}
}
val existingByPath = SongsTable.selectAll()
.where {
(SongsTable.folderId eq track.folderId) and
(SongsTable.relativePath eq track.relativePath)
}
.singleOrNull()
val existingId = existingByPath?.get(SongsTable.id)
if (existingId != null && existingId != track.songId) {
SongsTable.deleteWhere { it.run { SongsTable.id eq existingId } }
}
val songExists = existingId == track.songId ||
SongsTable.selectAll().where { SongsTable.id eq track.songId }.any()
if (songExists) {
SongsTable.update({ SongsTable.id eq track.songId }) {
it[albumId] = track.albumId
it[folderId] = track.folderId
it[title] = track.title
it[displayArtist] = track.displayArtist
it[albumName] = track.albumTitle
it[trackNumber] = track.trackNumber
it[discNumber] = track.discNumber
it[year] = track.year
it[duration] = track.durationSeconds
it[bitRate] = track.bitRate
it[size] = track.size
it[suffix] = track.suffix
it[contentType] = track.contentType
it[relativePath] = track.relativePath
it[fileMtime] = track.fileMtime
it[searchText] = track.searchText
it[updatedAt] = now
it[lyrics] = track.lyrics
}
} else {
SongsTable.insert {
it[id] = track.songId
it[albumId] = track.albumId
it[folderId] = track.folderId
it[title] = track.title
it[displayArtist] = track.displayArtist
it[albumName] = track.albumTitle
it[trackNumber] = track.trackNumber
it[discNumber] = track.discNumber
it[year] = track.year
it[duration] = track.durationSeconds
it[bitRate] = track.bitRate
it[size] = track.size
it[suffix] = track.suffix
it[contentType] = track.contentType
it[relativePath] = track.relativePath
it[fileMtime] = track.fileMtime
it[createdAt] = now
it[searchText] = track.searchText
it[updatedAt] = now
it[lyrics] = track.lyrics
}
}
SongArtistsTable.deleteWhere { it.run { SongArtistsTable.songId eq track.songId } }
track.artists.forEachIndexed { index, artist ->
SongArtistsTable.insertIgnore {
it[songId] = track.songId
it[artistId] = artist.id
it[position] = index
}
}
SongGenresTable.deleteWhere { it.run { SongGenresTable.songId eq track.songId } }
track.genres.forEach { genre ->
SongGenresTable.insertIgnore {
it[songId] = track.songId
it[this.genre] = genre
}
}
}
}
fun setAlbumCover(albumId: String) {
transaction {
AlbumsTable.update({ AlbumsTable.id eq albumId }) {
it[coverArt] = albumId
}
}
}
fun deleteSongsNotIn(seenSongIds: Set<String>) {
transaction {
val obsolete = SongsTable.selectAll()
.map { it[SongsTable.id] }
.filterNot { it in seenSongIds }
val now = Instant.now().toString()
obsolete.forEach { songId ->
LibraryTombstonesTable.insertIgnore {
it[entity] = "song"
it[entityId] = songId
it[deletedAt] = now
}
SongsTable.deleteWhere { it.run { SongsTable.id eq songId } }
}
exec("DELETE FROM albums WHERE id NOT IN (SELECT DISTINCT album_id FROM songs)")
exec(
"""
DELETE FROM artists WHERE id NOT IN (
SELECT artist_id FROM song_artists
UNION
SELECT artist_id FROM album_artists
)
""".trimIndent(),
)
}
}
/** Recomputes album/artist genre aggregates from the song-level tags. */
fun rebuildAggregates() {
transaction {
exec("DELETE FROM album_genres")
exec(
"""
INSERT INTO album_genres (album_id, genre)
SELECT DISTINCT s.album_id, sg.genre
FROM song_genres sg
JOIN songs s ON s.id = sg.song_id
""".trimIndent(),
)
exec("DELETE FROM artist_genres")
exec(
"""
INSERT INTO artist_genres (artist_id, genre)
SELECT DISTINCT sa.artist_id, sg.genre
FROM song_artists sa
JOIN song_genres sg ON sg.song_id = sa.song_id
UNION
SELECT DISTINCT aa.artist_id, ag.genre
FROM album_artists aa
JOIN album_genres ag ON ag.album_id = aa.album_id
""".trimIndent(),
)
}
}
fun countSongs(): Long = transaction {
SongsTable.selectAll().count()
}
private fun upsertArtist(artist: ArtistRef) {
ArtistsTable.insertIgnore {
it[id] = artist.id
it[name] = artist.name
it[sortName] = SortNames.of(artist.name)
it[searchText] = IdGenerator.normalize(artist.name)
}
ArtistsTable.update({ ArtistsTable.id eq artist.id }) {
it[name] = artist.name
it[sortName] = SortNames.of(artist.name)
it[searchText] = IdGenerator.normalize(artist.name)
}
}
}
@@ -1,12 +1,16 @@
package win.taptap.musicmanager.db
import java.time.Instant
import java.time.temporal.ChronoUnit
import kotlinx.serialization.builtins.ListSerializer
import kotlinx.serialization.builtins.serializer
import kotlinx.serialization.json.Json
import org.jetbrains.exposed.sql.ResultRow
import org.jetbrains.exposed.sql.SortOrder
import org.jetbrains.exposed.sql.SqlExpressionBuilder.eq
import org.jetbrains.exposed.sql.SqlExpressionBuilder.inList
import org.jetbrains.exposed.sql.SqlExpressionBuilder.inSubQuery
import org.jetbrains.exposed.sql.SqlExpressionBuilder.notInList
import org.jetbrains.exposed.sql.and
import org.jetbrains.exposed.sql.deleteWhere
import org.jetbrains.exposed.sql.insert
@@ -14,6 +18,7 @@ import org.jetbrains.exposed.sql.insertIgnore
import org.jetbrains.exposed.sql.selectAll
import org.jetbrains.exposed.sql.transactions.transaction
import org.jetbrains.exposed.sql.update
import win.taptap.musicmanager.source.MusicSource
import win.taptap.musicmanager.util.IdGenerator
data class MoodRecord(
@@ -23,46 +28,75 @@ data class MoodRecord(
val description: String?,
val query: String?,
val genres: List<String>,
val targetSize: Int,
val source: String,
val collectionPercent: Int,
val recommendationCount: Int,
val refreshHours: Int,
val ytEnabled: Boolean,
val lastDiscoveryAt: String?,
val createdAt: String,
val changedAt: String,
val lastError: String?,
val seedCount: Int,
val newRecommendations: Int,
val songCount: Int,
val recommendationReady: Int,
val recommendationDownloading: Int,
val recommendationFailed: Int,
)
data class MoodSeedRecord(
data class MoodSongRecord(
val songId: String,
val origin: String,
val addedAt: String,
)
data class RecommendationUpsert(
val source: String,
val externalId: String,
val title: String,
val artist: String,
val album: String?,
val duration: Int?,
val thumbnail: String?,
)
data class RecommendationRecord(
val videoId: String,
val source: String,
val externalId: String,
val title: String,
val artist: String,
val album: String?,
val duration: Int?,
val thumbnail: String?,
val state: String,
val matchedSongId: String?,
val timesAdded: Int,
val songId: String?,
val attempts: Int,
val error: String?,
val rank: Int,
val firstSeen: String,
val lastSeen: String,
)
/** Result of applying a finished download to a recommendation. */
enum class RecommendationCompletion { READY, ALREADY_ADDED, REJECTED, MISSING }
/**
* Moods are the product's only collection. Seeds are permanent anchors; the
* play queue is generated by [win.taptap.musicmanager.radio.RadioService] for
* every playback session instead of being stored as a playlist.
* Moods are the product's only collection. Songs are permanent entries; the
* recommendation table also holds hidden `new` candidates and the lifecycle of
* the active slots. The play queue is generated by [win.taptap.musicmanager.radio.RadioService]
* for every playback session instead of being stored as a playlist.
*
* A mood has one download source at a time. Switching it clears the active and
* hidden recommendations of the old source; permanent songs and the
* added/rejected history stay.
*/
object MoodRepository {
private val genreSerializer = ListSerializer(String.serializer())
private val ACTIVE_STATES = listOf("downloading", "ready", "failed")
private val VISIBLE_STATES = ACTIVE_STATES
private val HIDDEN_DECISIONS = listOf("added", "rejected")
private val CLEARED_ON_SOURCE_CHANGE = listOf("new", "downloading", "ready", "failed")
fun list(owner: String): List<MoodRecord> = transaction {
MoodsTable.selectAll()
.where { MoodsTable.owner eq owner }
@@ -70,6 +104,12 @@ object MoodRepository {
.map { moodRecord(it) }
}
fun all(): List<MoodRecord> = transaction {
MoodsTable.selectAll()
.orderBy(MoodsTable.changedAt to SortOrder.DESC)
.map { moodRecord(it) }
}
fun get(id: String, owner: String): MoodRecord? = transaction {
MoodsTable.selectAll()
.where { (MoodsTable.id eq id) and (MoodsTable.owner eq owner) }
@@ -77,16 +117,24 @@ object MoodRepository {
?.let { moodRecord(it) }
}
/** Owner-agnostic lookup for background work that only has a mood id. */
fun byId(id: String): MoodRecord? = transaction {
MoodsTable.selectAll()
.where { MoodsTable.id eq id }
.singleOrNull()
?.let { moodRecord(it) }
}
fun create(
owner: String,
name: String,
description: String?,
query: String?,
genres: List<String>,
targetSize: Int,
source: String,
collectionPercent: Int,
recommendationCount: Int,
refreshHours: Int,
ytEnabled: Boolean,
seedSongIds: List<String>,
): MoodRecord = transaction {
val now = Instant.now().toString()
val id = IdGenerator.moodId(owner, name, now)
@@ -98,18 +146,15 @@ object MoodRepository {
it[MoodsTable.description] = description?.take(2000)
it[MoodsTable.query] = query?.trim()?.takeIf(String::isNotEmpty)
it[MoodsTable.genres] = encodeGenres(genres)
it[MoodsTable.targetSize] = targetSize.coerceIn(5, 500)
it[MoodsTable.musicSource] = source
it[MoodsTable.collectionPercent] = collectionPercent.coerceIn(0, 100)
it[MoodsTable.recommendationCount] = recommendationCount.coerceIn(0, 30)
it[MoodsTable.refreshHours] = refreshHours.coerceIn(1, 720)
it[MoodsTable.ytEnabled] = ytEnabled
it[createdAt] = now
it[changedAt] = now
}
insertSeeds(id, seedSongIds, "manual")
moodRecord(
MoodsTable.selectAll().where { MoodsTable.id eq id }.single(),
)
moodRecord(MoodsTable.selectAll().where { MoodsTable.id eq id }.single())
}
fun update(
@@ -119,24 +164,41 @@ object MoodRepository {
description: String?,
query: String?,
genres: List<String>,
targetSize: Int,
source: String,
collectionPercent: Int,
recommendationCount: Int,
refreshHours: Int,
ytEnabled: Boolean,
): MoodRecord? = transaction {
val exists = MoodsTable.selectAll()
val current = MoodsTable.selectAll()
.where { (MoodsTable.id eq id) and (MoodsTable.owner eq owner) }
.any()
if (!exists) return@transaction null
.singleOrNull()
?: return@transaction null
val sourceChanged = current[MoodsTable.musicSource] != source
if (sourceChanged) {
// Candidates are tied to the old catalog. Keep permanent songs and
// the added/rejected decisions, but drop the active pool so it can
// be refilled from the new source.
MoodRecommendationsTable.deleteWhere {
(MoodRecommendationsTable.moodId eq id) and
(MoodRecommendationsTable.state inList CLEARED_ON_SOURCE_CHANGE)
}
}
MoodsTable.update({ MoodsTable.id eq id }) {
it[MoodsTable.name] = name.take(200)
it[MoodsTable.description] = description?.take(2000)
it[MoodsTable.query] = query?.trim()?.takeIf(String::isNotEmpty)
it[MoodsTable.genres] = encodeGenres(genres)
it[MoodsTable.targetSize] = targetSize.coerceIn(5, 500)
it[MoodsTable.musicSource] = source
it[MoodsTable.collectionPercent] = collectionPercent.coerceIn(0, 100)
it[MoodsTable.recommendationCount] = recommendationCount.coerceIn(0, 30)
it[MoodsTable.refreshHours] = refreshHours.coerceIn(1, 720)
it[MoodsTable.ytEnabled] = ytEnabled
it[changedAt] = Instant.now().toString()
it[MoodsTable.changedAt] = Instant.now().toString()
if (sourceChanged) {
it[lastDiscoveryAt] = null
it[lastError] = null
}
}
moodRecord(MoodsTable.selectAll().where { MoodsTable.id eq id }.single())
@@ -148,85 +210,400 @@ object MoodRepository {
} > 0
}
fun seeds(id: String): List<MoodSeedRecord> = transaction {
MoodSeedsTable.selectAll()
.where { MoodSeedsTable.moodId eq id }
// ---------------------------------------------------------------- songs
fun songs(id: String): List<MoodSongRecord> = transaction {
MoodSongsTable.selectAll()
.where { MoodSongsTable.moodId eq id }
.orderBy(
MoodSeedsTable.addedAt to SortOrder.ASC,
MoodSeedsTable.songId to SortOrder.ASC,
MoodSongsTable.addedAt to SortOrder.ASC,
MoodSongsTable.songId to SortOrder.ASC,
)
.map {
MoodSeedRecord(
songId = it[MoodSeedsTable.songId],
origin = it[MoodSeedsTable.origin],
addedAt = it[MoodSeedsTable.addedAt],
MoodSongRecord(
songId = it[MoodSongsTable.songId],
origin = it[MoodSongsTable.origin],
addedAt = it[MoodSongsTable.addedAt],
)
}
}
fun seedSongIds(id: String): List<String> = seeds(id).map { it.songId }
fun songIds(id: String): List<String> = songs(id).map { it.songId }
/** Returns false when the song is already a seed or the mood is not owned. */
fun addSeed(id: String, owner: String, songId: String, origin: String): Boolean = transaction {
val owned = MoodsTable.selectAll()
.where { (MoodsTable.id eq id) and (MoodsTable.owner eq owner) }
.any()
if (!owned) return@transaction false
fun songRecords(id: String): List<SongRecord> {
val ids = songIds(id)
return SongRepository.byIds(ids)
}
val exists = MoodSeedsTable.selectAll()
.where { (MoodSeedsTable.moodId eq id) and (MoodSeedsTable.songId eq songId) }
/** Returns false when the song is already in the mood or the mood does not exist. */
fun addSong(id: String, songId: String, origin: String): Boolean = transaction {
val exists = MoodSongsTable.selectAll()
.where { (MoodSongsTable.moodId eq id) and (MoodSongsTable.songId eq songId) }
.any()
if (exists) return@transaction false
val moodExists = MoodsTable.selectAll().where { MoodsTable.id eq id }.any()
if (!moodExists) return@transaction false
val now = Instant.now().toString()
MoodSeedsTable.insertIgnore {
it[MoodSeedsTable.moodId] = id
it[MoodSeedsTable.songId] = songId
it[MoodSeedsTable.origin] = origin.takeIf { value -> value in ORIGINS } ?: "manual"
MoodSongsTable.insertIgnore {
it[moodId] = id
it[this.songId] = songId
it[this.origin] = origin.takeIf { value -> value in ORIGINS } ?: "search"
it[addedAt] = now
}
MoodsTable.update({ MoodsTable.id eq id }) {
it[changedAt] = now
}
// A track that is now permanently in the mood must not also linger as a
// hidden candidate or an active recommendation slot.
val song = SongRepository.byId(songId)
if (song != null) {
MoodRecommendationsTable.deleteWhere {
(MoodRecommendationsTable.moodId eq id) and
(MoodRecommendationsTable.musicSource eq song.source) and
(MoodRecommendationsTable.externalId eq song.externalId) and
(MoodRecommendationsTable.state eq "new")
}
MoodRecommendationsTable.update({
(MoodRecommendationsTable.moodId eq id) and
(MoodRecommendationsTable.musicSource eq song.source) and
(MoodRecommendationsTable.externalId eq song.externalId) and
(MoodRecommendationsTable.state inList listOf("downloading", "ready", "failed"))
}) {
it[state] = "added"
it[this.songId] = songId
}
}
true
}
fun removeSeed(id: String, owner: String, songId: String): Boolean = transaction {
val owned = MoodsTable.selectAll()
.where { (MoodsTable.id eq id) and (MoodsTable.owner eq owner) }
.any()
if (!owned) return@transaction false
val removed = MoodSeedsTable.deleteWhere {
(MoodSeedsTable.moodId eq id) and (MoodSeedsTable.songId eq songId)
fun removeSong(id: String, songId: String): Boolean = transaction {
val removed = MoodSongsTable.deleteWhere {
(MoodSongsTable.moodId eq id) and (MoodSongsTable.songId eq songId)
} > 0
if (removed) {
MoodsTable.update({ MoodsTable.id eq id }) {
it[changedAt] = Instant.now().toString()
}
// Detaching must not keep a permanent `added` decision pinned to the
// file, otherwise the GC could never delete it.
MoodRecommendationsTable.update({
(MoodRecommendationsTable.moodId eq id) and
(MoodRecommendationsTable.songId eq songId) and
(MoodRecommendationsTable.state eq "added")
}) {
it[MoodRecommendationsTable.songId] = null
}
}
removed
}
// ------------------------------------------------------ recommendations
/** Active recommendations, i.e. the rows the client is allowed to see. */
fun recommendations(id: String): List<RecommendationRecord> = transaction {
MoodRecommendationsTable.selectAll()
.where { MoodRecommendationsTable.moodId eq id }
.orderBy(MoodRecommendationsTable.lastSeen to SortOrder.DESC)
.map {
RecommendationRecord(
videoId = it[MoodRecommendationsTable.videoId],
title = it[MoodRecommendationsTable.title],
artist = it[MoodRecommendationsTable.artist],
album = it[MoodRecommendationsTable.album],
duration = it[MoodRecommendationsTable.duration],
thumbnail = it[MoodRecommendationsTable.thumbnail],
state = it[MoodRecommendationsTable.state],
matchedSongId = it[MoodRecommendationsTable.matchedSongId],
timesAdded = it[MoodRecommendationsTable.timesAdded],
firstSeen = it[MoodRecommendationsTable.firstSeen],
lastSeen = it[MoodRecommendationsTable.lastSeen],
)
.where {
(MoodRecommendationsTable.moodId eq id) and
(MoodRecommendationsTable.state inList VISIBLE_STATES)
}
.orderBy(
MoodRecommendationsTable.rank to SortOrder.ASC,
MoodRecommendationsTable.firstSeen to SortOrder.ASC,
)
.map { it.toRecommendationRecord() }
}
fun allRecommendations(id: String): List<RecommendationRecord> = transaction {
MoodRecommendationsTable.selectAll()
.where { MoodRecommendationsTable.moodId eq id }
.orderBy(MoodRecommendationsTable.rank to SortOrder.ASC)
.map { it.toRecommendationRecord() }
}
fun recommendation(moodId: String, externalId: String): RecommendationRecord? = transaction {
MoodRecommendationsTable.selectAll()
.where {
(MoodRecommendationsTable.moodId eq moodId) and
(MoodRecommendationsTable.externalId eq externalId)
}
.singleOrNull()
?.toRecommendationRecord()
}
fun activeCount(moodId: String): Int = transaction {
MoodRecommendationsTable.selectAll()
.where {
(MoodRecommendationsTable.moodId eq moodId) and
(MoodRecommendationsTable.state inList ACTIVE_STATES)
}
.count()
.toInt()
}
fun externalIdsInMood(moodId: String, source: String): Set<String> = transaction {
val songs = MoodSongsTable.select(MoodSongsTable.songId)
.where { MoodSongsTable.moodId eq moodId }
SongsTable.select(SongsTable.externalId)
.where {
(SongsTable.id inSubQuery songs) and
(SongsTable.musicSource eq source)
}
.map { it[SongsTable.externalId] }
.toSet()
}
fun decidedExternalIds(moodId: String, source: String): Set<String> = transaction {
MoodRecommendationsTable.selectAll()
.where {
(MoodRecommendationsTable.moodId eq moodId) and
(MoodRecommendationsTable.musicSource eq source) and
(MoodRecommendationsTable.state inList HIDDEN_DECISIONS)
}
.map { it[MoodRecommendationsTable.externalId] }
.toSet()
}
/**
* Stores a sorted discovery batch as hidden `new` candidates. Existing rows
* keep their state: active slots and decisions must survive a refresh.
*/
fun storeCandidates(
moodId: String,
source: String,
items: List<RecommendationUpsert>,
seenAt: String,
poolLimit: Int = CANDIDATE_POOL,
): Int = transaction {
val existing = MoodRecommendationsTable.selectAll()
.where {
(MoodRecommendationsTable.moodId eq moodId) and
(MoodRecommendationsTable.musicSource eq source)
}
.associateBy { it[MoodRecommendationsTable.externalId] }
var stored = 0
items.take(poolLimit.coerceAtLeast(1)).forEachIndexed { index, item ->
val row = existing[item.externalId]
if (row == null) {
MoodRecommendationsTable.insert {
it[MoodRecommendationsTable.moodId] = moodId
it[MoodRecommendationsTable.musicSource] = source
it[externalId] = item.externalId
it[title] = item.title.take(500)
it[artist] = item.artist.take(300)
it[album] = item.album?.take(300)
it[duration] = item.duration
it[thumbnail] = item.thumbnail?.take(500)
it[state] = "new"
it[rank] = index
it[firstSeen] = seenAt
it[lastSeen] = seenAt
}
} else if (row[MoodRecommendationsTable.state] == "new") {
MoodRecommendationsTable.update({
(MoodRecommendationsTable.moodId eq moodId) and
(MoodRecommendationsTable.externalId eq item.externalId)
}) {
it[title] = item.title.take(500)
it[artist] = item.artist.take(300)
it[album] = item.album?.take(300)
it[duration] = item.duration
it[thumbnail] = item.thumbnail?.take(500)
it[rank] = index
it[lastSeen] = seenAt
}
}
stored += 1
}
// Candidates that were not returned again are no longer useful.
val keep = items.take(poolLimit.coerceAtLeast(1)).map { it.externalId }.toSet()
if (keep.isNotEmpty()) {
MoodRecommendationsTable.deleteWhere {
(MoodRecommendationsTable.moodId eq moodId) and
(MoodRecommendationsTable.musicSource eq source) and
(MoodRecommendationsTable.state eq "new") and
(MoodRecommendationsTable.externalId notInList keep.toList())
}
}
stored
}
/** Moves up to [limit] hidden candidates into the downloading state. */
fun promoteCandidates(moodId: String, source: String, limit: Int): List<RecommendationRecord> =
transaction {
if (limit <= 0) return@transaction emptyList()
val rows = MoodRecommendationsTable.selectAll()
.where {
(MoodRecommendationsTable.moodId eq moodId) and
(MoodRecommendationsTable.musicSource eq source) and
(MoodRecommendationsTable.state eq "new")
}
.orderBy(
MoodRecommendationsTable.rank to SortOrder.ASC,
MoodRecommendationsTable.firstSeen to SortOrder.ASC,
)
.limit(limit)
.toList()
rows.forEach { row ->
MoodRecommendationsTable.update({
(MoodRecommendationsTable.moodId eq moodId) and
(MoodRecommendationsTable.externalId eq row[MoodRecommendationsTable.externalId])
}) {
it[state] = "downloading"
it[error] = null
}
}
rows.map { it.toRecommendationRecord().copy(state = "downloading", error = null) }
}
fun setRecommendationState(moodId: String, externalId: String, state: String, error: String? = null) {
transaction {
MoodRecommendationsTable.update({
(MoodRecommendationsTable.moodId eq moodId) and
(MoodRecommendationsTable.externalId eq externalId)
}) {
it[this.state] = state
it[this.error] = error?.take(1000)
}
}
}
fun markRecommendationDownloading(moodId: String, externalId: String) {
transaction {
MoodRecommendationsTable.update({
(MoodRecommendationsTable.moodId eq moodId) and
(MoodRecommendationsTable.externalId eq externalId)
}) {
it[state] = "downloading"
it[attempts] = 0
it[error] = null
}
}
}
fun failRecommendation(moodId: String, externalId: String, error: String, attempts: Int) {
transaction {
MoodRecommendationsTable.update({
(MoodRecommendationsTable.moodId eq moodId) and
(MoodRecommendationsTable.externalId eq externalId) and
(MoodRecommendationsTable.state eq "downloading")
}) {
it[state] = "failed"
it[this.error] = error.take(1000)
it[this.attempts] = attempts
}
}
}
/**
* Applies a finished download. A rejection that happened while the download
* was running wins: the file is left for the GC to delete.
*/
fun completeDownload(
moodId: String,
source: String,
externalId: String,
songId: String,
): RecommendationCompletion = transaction {
val row = MoodRecommendationsTable.selectAll()
.where {
(MoodRecommendationsTable.moodId eq moodId) and
(MoodRecommendationsTable.musicSource eq source) and
(MoodRecommendationsTable.externalId eq externalId)
}
.singleOrNull()
?: return@transaction RecommendationCompletion.MISSING
when (row[MoodRecommendationsTable.state]) {
"rejected" -> RecommendationCompletion.REJECTED
"added" -> {
insertRecommendationSong(moodId, songId)
MoodRecommendationsTable.update({
(MoodRecommendationsTable.moodId eq moodId) and
(MoodRecommendationsTable.externalId eq externalId)
}) {
it[this.songId] = songId
it[state] = "added"
}
RecommendationCompletion.ALREADY_ADDED
}
else -> {
MoodRecommendationsTable.update({
(MoodRecommendationsTable.moodId eq moodId) and
(MoodRecommendationsTable.externalId eq externalId)
}) {
it[this.songId] = songId
it[state] = "ready"
it[error] = null
it[lastSeen] = Instant.now().toString()
}
RecommendationCompletion.READY
}
}
}
/** Called when the user presses Add. Marks the row and attaches a ready song. */
fun markAdded(moodId: String, externalId: String): RecommendationCompletion = transaction {
val row = MoodRecommendationsTable.selectAll()
.where {
(MoodRecommendationsTable.moodId eq moodId) and
(MoodRecommendationsTable.externalId eq externalId)
}
.singleOrNull()
?: return@transaction RecommendationCompletion.MISSING
val songId = row[MoodRecommendationsTable.songId]
MoodRecommendationsTable.update({
(MoodRecommendationsTable.moodId eq moodId) and
(MoodRecommendationsTable.externalId eq externalId)
}) {
it[state] = "added"
}
if (songId != null && row[MoodRecommendationsTable.state] != "added") {
insertRecommendationSong(moodId, songId)
return@transaction RecommendationCompletion.ALREADY_ADDED
}
RecommendationCompletion.READY
}
fun markRejected(moodId: String, externalId: String) {
setRecommendationState(moodId, externalId, "rejected")
}
fun readySongs(moodId: String): List<SongRecord> = transaction {
val songIds = MoodRecommendationsTable
.select(MoodRecommendationsTable.songId)
.where {
(MoodRecommendationsTable.moodId eq moodId) and
(MoodRecommendationsTable.state eq "ready")
}
.mapNotNull { it[MoodRecommendationsTable.songId] }
if (songIds.isEmpty()) emptyList()
else SongsTable.selectAll()
.where { SongsTable.id inList songIds }
.map { it.toSongRecord() }
}
// ---------------------------------------------------------- discovery
/** Moods whose automatic discovery is due, oldest first. */
fun staleDiscovery(now: Instant, limit: Int): List<MoodRecord> = transaction {
MoodsTable.selectAll()
.orderBy(MoodsTable.lastDiscoveryAt to SortOrder.ASC)
.map { moodRecord(it) }
.filter { mood ->
mood.recommendationCount > 0 &&
(
isStale(mood.lastDiscoveryAt, mood.refreshHours, now) ||
songsAddedSince(mood.id, mood.lastDiscoveryAt)
)
}
.take(limit.coerceAtLeast(1))
}
fun markError(id: String, message: String) {
@@ -246,31 +623,90 @@ object MoodRepository {
}
}
private fun insertSeeds(id: String, songIds: List<String>, origin: String) {
val now = Instant.now().toString()
songIds.distinct().forEach { songId ->
MoodSeedsTable.insertIgnore {
it[moodId] = id
it[this.songId] = songId
it[this.origin] = origin
it[addedAt] = now
}
fun isDiscoveryDue(mood: MoodRecord, now: Instant = Instant.now()): Boolean =
isStale(mood.lastDiscoveryAt, mood.refreshHours, now) ||
songsAddedSince(mood.id, mood.lastDiscoveryAt)
/**
* The collection is the discovery seed, so a song added after the last pass
* makes the hidden candidate pool stale, even when every slot is filled.
* Both timestamps come from [Instant.toString], so the lexical comparison is
* also chronological.
*/
fun songsAddedSince(moodId: String, since: String?): Boolean {
if (since == null) return false
return transaction {
MoodSongsTable.selectAll()
.where { MoodSongsTable.moodId eq moodId }
.any { row -> row[MoodSongsTable.addedAt] > since }
}
}
// ------------------------------------------------------------ internal
private fun insertRecommendationSong(moodId: String, songId: String) {
MoodSongsTable.insertIgnore {
it[this.moodId] = moodId
it[this.songId] = songId
it[origin] = "recommendation"
it[addedAt] = Instant.now().toString()
}
MoodsTable.update({ MoodsTable.id eq moodId }) {
it[changedAt] = Instant.now().toString()
}
}
private fun isStale(lastDiscoveryAt: String?, refreshHours: Int, now: Instant): Boolean {
if (lastDiscoveryAt == null) return true
val at = runCatching { Instant.parse(lastDiscoveryAt) }.getOrNull() ?: return true
return !at.plus(refreshHours.toLong().coerceAtLeast(1), ChronoUnit.HOURS).isAfter(now)
}
private fun ResultRow.toRecommendationRecord() = RecommendationRecord(
source = this[MoodRecommendationsTable.musicSource],
externalId = this[MoodRecommendationsTable.externalId],
title = this[MoodRecommendationsTable.title],
artist = this[MoodRecommendationsTable.artist],
album = this[MoodRecommendationsTable.album],
duration = this[MoodRecommendationsTable.duration],
thumbnail = this[MoodRecommendationsTable.thumbnail],
state = this[MoodRecommendationsTable.state],
songId = this[MoodRecommendationsTable.songId],
attempts = this[MoodRecommendationsTable.attempts],
error = this[MoodRecommendationsTable.error],
rank = this[MoodRecommendationsTable.rank],
firstSeen = this[MoodRecommendationsTable.firstSeen],
lastSeen = this[MoodRecommendationsTable.lastSeen],
)
private fun ResultRow.toSongRecord() = SongRecord(
id = this[SongsTable.id],
source = this[SongsTable.musicSource],
externalId = this[SongsTable.externalId],
title = this[SongsTable.title],
artist = this[SongsTable.artist],
album = this[SongsTable.album],
duration = this[SongsTable.duration],
bitRate = this[SongsTable.bitRate],
size = this[SongsTable.size],
suffix = this[SongsTable.suffix],
contentType = this[SongsTable.contentType],
relativePath = this[SongsTable.relativePath],
fileMtime = this[SongsTable.fileMtime],
thumbnail = this[SongsTable.thumbnail],
createdAt = this[SongsTable.createdAt],
updatedAt = this[SongsTable.updatedAt],
)
private fun moodRecord(row: ResultRow): MoodRecord {
val id = row[MoodsTable.id]
val seedCount = MoodSeedsTable.selectAll()
.where { MoodSeedsTable.moodId eq id }
.count()
.toInt()
val newRecommendations = MoodRecommendationsTable.selectAll()
.where {
(MoodRecommendationsTable.moodId eq id) and
(MoodRecommendationsTable.state eq "new")
}
val songCount = MoodSongsTable.selectAll()
.where { MoodSongsTable.moodId eq id }
.count()
.toInt()
val ready = countState(id, "ready")
val downloading = countState(id, "downloading")
val failed = countState(id, "failed")
return MoodRecord(
id = id,
@@ -279,18 +715,30 @@ object MoodRepository {
description = row[MoodsTable.description],
query = row[MoodsTable.query],
genres = decodeGenres(row[MoodsTable.genres]),
targetSize = row[MoodsTable.targetSize],
source = row[MoodsTable.musicSource],
collectionPercent = row[MoodsTable.collectionPercent],
recommendationCount = row[MoodsTable.recommendationCount],
refreshHours = row[MoodsTable.refreshHours],
ytEnabled = row[MoodsTable.ytEnabled],
lastDiscoveryAt = row[MoodsTable.lastDiscoveryAt],
createdAt = row[MoodsTable.createdAt],
changedAt = row[MoodsTable.changedAt],
lastError = row[MoodsTable.lastError],
seedCount = seedCount,
newRecommendations = newRecommendations,
songCount = songCount,
recommendationReady = ready,
recommendationDownloading = downloading,
recommendationFailed = failed,
)
}
private fun countState(moodId: String, state: String): Int =
MoodRecommendationsTable.selectAll()
.where {
(MoodRecommendationsTable.moodId eq moodId) and
(MoodRecommendationsTable.state eq state)
}
.count()
.toInt()
private fun encodeGenres(genres: List<String>): String =
Json.encodeToString(
genreSerializer,
@@ -300,5 +748,6 @@ object MoodRepository {
private fun decodeGenres(value: String): List<String> =
runCatching { Json.decodeFromString(genreSerializer, value) }.getOrDefault(emptyList())
private val ORIGINS = setOf("manual", "recommendation", "yt")
private val ORIGINS = setOf("search", "recommendation", "url")
private const val CANDIDATE_POOL = 60
}
@@ -3,6 +3,9 @@ package win.taptap.musicmanager.db
import java.time.Instant
import org.jetbrains.exposed.sql.SortOrder
import org.jetbrains.exposed.sql.SqlExpressionBuilder.eq
import org.jetbrains.exposed.sql.SqlExpressionBuilder.greaterEq
import org.jetbrains.exposed.sql.SqlExpressionBuilder.inList
import org.jetbrains.exposed.sql.SqlExpressionBuilder.inSubQuery
import org.jetbrains.exposed.sql.and
import org.jetbrains.exposed.sql.insert
import org.jetbrains.exposed.sql.selectAll
@@ -25,8 +28,8 @@ data class SessionItemRecord(
)
data class AffinityRecord(val completions: Int, val skips: Int) {
/** Skips weigh double so a mood stops serving rejected tracks quickly. */
val score: Double get() = completions - skips * 2.0
/** Skips weigh triple so a mood stops serving rejected tracks quickly. */
val score: Double get() = completions - skips * 3.0
}
/**
@@ -141,6 +144,17 @@ object RadioRepository {
(PlaybackSessionsTable.userName eq userName)
}
.singleOrNull() ?: return@transaction false
if (session[PlaybackSessionsTable.closedAt] != null) return@transaction false
// Only tracks the server actually queued in this session may change
// affinity. Arbitrary/deleted ids used to violate the affinity FK.
val queued = SessionItemsTable.selectAll()
.where {
(SessionItemsTable.sessionId eq sessionId) and
(SessionItemsTable.songId eq songId)
}
.any()
if (!queued) return@transaction false
val now = Instant.now().toString()
SessionItemsTable.update({
@@ -185,6 +199,28 @@ object RadioRepository {
true
}
/**
* Songs the user started, finished or skipped since [since]. Used to avoid
* serving the same tracks every day across sessions and moods.
*/
fun recentSongIds(userName: String, since: Instant, limit: Int = 500): Set<String> = transaction {
val sessions = PlaybackSessionsTable
.select(PlaybackSessionsTable.id)
.where { PlaybackSessionsTable.userName eq userName }
SessionItemsTable
.select(SessionItemsTable.songId)
.where {
(SessionItemsTable.sessionId inSubQuery sessions) and
(SessionItemsTable.state inList listOf("started", "completed", "skipped")) and
(SessionItemsTable.updatedAt greaterEq since.toString())
}
.orderBy(SessionItemsTable.updatedAt to SortOrder.DESC)
.limit(limit)
.map { it[SessionItemsTable.songId] }
.toSet()
}
fun affinity(userName: String, songIds: Collection<String>): Map<String, AffinityRecord> {
if (songIds.isEmpty()) return emptyMap()
return transaction {
@@ -0,0 +1,243 @@
package win.taptap.musicmanager.db
import java.time.Instant
import org.jetbrains.exposed.sql.ResultRow
import org.jetbrains.exposed.sql.SortOrder
import org.jetbrains.exposed.sql.SqlExpressionBuilder.eq
import org.jetbrains.exposed.sql.SqlExpressionBuilder.inList
import org.jetbrains.exposed.sql.and
import org.jetbrains.exposed.sql.deleteWhere
import org.jetbrains.exposed.sql.insert
import org.jetbrains.exposed.sql.selectAll
import org.jetbrains.exposed.sql.transactions.transaction
import org.jetbrains.exposed.sql.update
import win.taptap.musicmanager.util.IdGenerator
data class SongRecord(
val id: String,
val source: String,
val externalId: String,
val title: String,
val artist: String,
val album: String?,
val duration: Int,
val bitRate: Int,
val size: Long,
val suffix: String,
val contentType: String,
val relativePath: String,
val fileMtime: Long,
val thumbnail: String?,
val createdAt: String,
val updatedAt: String,
/** Integrated loudness in LUFS; null until the analyzer has run. */
val loudnessLufs: Double? = null,
/** Gain in dB applied by clients to reach the server's loudness target. */
val gainDb: Double? = null,
)
data class SongUpsert(
val source: String,
val externalId: String,
val title: String,
val artist: String,
val album: String?,
val duration: Int,
val bitRate: Int,
val size: Long,
val suffix: String = "mp3",
val contentType: String = "audio/mpeg",
val relativePath: String,
val fileMtime: Long,
val thumbnail: String?,
)
/**
* The shared, server-wide song store. Rows are keyed by `(source, externalId)`,
* so downloading the same track for two moods or two users produces one file
* and one row.
*/
object SongRepository {
fun upsert(input: SongUpsert): SongRecord = transaction {
val now = Instant.now().toString()
val existing = SongsTable.selectAll()
.where {
(SongsTable.musicSource eq input.source) and
(SongsTable.externalId eq input.externalId)
}
.singleOrNull()
if (existing == null) {
val id = IdGenerator.songId(input.source, input.externalId)
SongsTable.insert {
it[this.id] = id
it[musicSource] = input.source
it[externalId] = input.externalId
it[title] = input.title
it[artist] = input.artist
it[album] = input.album
it[duration] = input.duration
it[bitRate] = input.bitRate
it[size] = input.size
it[suffix] = input.suffix
it[contentType] = input.contentType
it[relativePath] = input.relativePath
it[fileMtime] = input.fileMtime
it[thumbnail] = input.thumbnail
it[loudnessLufs] = null
it[gainDb] = null
it[createdAt] = now
it[updatedAt] = now
}
SongRecord(
id = id,
source = input.source,
externalId = input.externalId,
title = input.title,
artist = input.artist,
album = input.album,
duration = input.duration,
bitRate = input.bitRate,
size = input.size,
suffix = input.suffix,
contentType = input.contentType,
relativePath = input.relativePath,
fileMtime = input.fileMtime,
thumbnail = input.thumbnail,
createdAt = now,
updatedAt = now,
)
} else {
val current = existing.toRecord()
val id = current.id
val audioChanged = current.fileMtime != input.fileMtime ||
current.relativePath != input.relativePath
SongsTable.update({ SongsTable.id eq id }) {
it[title] = input.title
it[artist] = input.artist
it[album] = input.album
it[duration] = input.duration
it[bitRate] = input.bitRate
it[size] = input.size
it[suffix] = input.suffix
it[contentType] = input.contentType
it[relativePath] = input.relativePath
it[fileMtime] = input.fileMtime
it[thumbnail] = input.thumbnail ?: current.thumbnail
if (audioChanged) {
it[loudnessLufs] = null
it[gainDb] = null
}
it[updatedAt] = now
}
current.copy(
title = input.title,
artist = input.artist,
album = input.album,
duration = input.duration,
bitRate = input.bitRate,
size = input.size,
suffix = input.suffix,
contentType = input.contentType,
relativePath = input.relativePath,
fileMtime = input.fileMtime,
thumbnail = input.thumbnail ?: current.thumbnail,
updatedAt = now,
loudnessLufs = if (audioChanged) null else current.loudnessLufs,
gainDb = if (audioChanged) null else current.gainDb,
)
}
}
fun byId(id: String): SongRecord? = transaction {
SongsTable.selectAll()
.where { SongsTable.id eq id }
.singleOrNull()
?.toRecord()
}
fun byExternalId(source: String, externalId: String): SongRecord? = transaction {
SongsTable.selectAll()
.where {
(SongsTable.musicSource eq source) and
(SongsTable.externalId eq externalId)
}
.singleOrNull()
?.toRecord()
}
fun byIds(ids: Collection<String>): List<SongRecord> {
if (ids.isEmpty()) return emptyList()
return transaction {
// Stay below SQLite's variable limit for very large moods.
ids.toList().chunked(SQL_CHUNK).flatMap { chunk ->
SongsTable.selectAll()
.where { SongsTable.id inList chunk }
.map { it.toRecord() }
}
}
}
fun all(): List<SongRecord> = transaction {
SongsTable.selectAll()
.orderBy(SongsTable.createdAt to SortOrder.ASC)
.map { it.toRecord() }
}
fun count(): Long = transaction { SongsTable.selectAll().count() }
/** Files still waiting for loudness analysis, oldest first. */
fun pendingLoudness(limit: Int = 1): List<SongRecord> = transaction {
SongsTable.selectAll()
.where { SongsTable.loudnessLufs.isNull() }
.orderBy(SongsTable.createdAt to SortOrder.ASC)
.limit(limit)
.map { it.toRecord() }
}
fun updateLoudness(id: String, loudnessLufs: Double, gainDb: Double): Boolean = transaction {
SongsTable.update({ SongsTable.id eq id }) {
it[this.loudnessLufs] = loudnessLufs
it[this.gainDb] = gainDb
it[updatedAt] = Instant.now().toString()
} > 0
}
/** Clears the measurement so the backfill analyzes a replaced file again. */
fun clearLoudness(id: String) {
transaction {
SongsTable.update({ SongsTable.id eq id }) {
it[loudnessLufs] = null
it[gainDb] = null
}
}
}
fun delete(id: String): Boolean = transaction {
SongsTable.deleteWhere { SongsTable.id eq id } > 0
}
private fun ResultRow.toRecord(): SongRecord = SongRecord(
id = this[SongsTable.id],
source = this[SongsTable.musicSource],
externalId = this[SongsTable.externalId],
title = this[SongsTable.title],
artist = this[SongsTable.artist],
album = this[SongsTable.album],
duration = this[SongsTable.duration],
bitRate = this[SongsTable.bitRate],
size = this[SongsTable.size],
suffix = this[SongsTable.suffix],
contentType = this[SongsTable.contentType],
relativePath = this[SongsTable.relativePath],
fileMtime = this[SongsTable.fileMtime],
thumbnail = this[SongsTable.thumbnail],
createdAt = this[SongsTable.createdAt],
updatedAt = this[SongsTable.updatedAt],
loudnessLufs = this[SongsTable.loudnessLufs],
gainDb = this[SongsTable.gainDb],
)
private const val SQL_CHUNK = 500
}
@@ -2,6 +2,7 @@ package win.taptap.musicmanager.db
import org.jetbrains.exposed.sql.ReferenceOption
import org.jetbrains.exposed.sql.Table
import win.taptap.musicmanager.source.MusicSource
/** Identity tables are reconciled from config/Vault at startup; passwords are never stored. */
object UsersTable : Table("users") {
@@ -12,132 +13,40 @@ object UsersTable : Table("users") {
override val primaryKey = PrimaryKey(name)
}
object MusicFoldersTable : Table("music_folders") {
val id = integer("id")
val name = varchar("name", 255)
val type = varchar("type", 16)
val rootPath = text("root_path")
override val primaryKey = PrimaryKey(id)
}
object ArtistsTable : Table("artists") {
val id = varchar("id", 32)
val name = text("name")
val sortName = text("sort_name")
val coverArt = varchar("cover_art", 32).nullable()
val searchText = text("search_text")
override val primaryKey = PrimaryKey(id)
}
object AlbumsTable : Table("albums") {
val id = varchar("id", 32)
val name = text("name")
val year = integer("year").nullable()
val coverArt = varchar("cover_art", 32).nullable()
val createdAt = varchar("created_at", 40)
val searchText = text("search_text")
override val primaryKey = PrimaryKey(id)
}
object AlbumArtistsTable : Table("album_artists") {
val albumId = reference("album_id", AlbumsTable.id, onDelete = ReferenceOption.CASCADE)
val artistId = reference("artist_id", ArtistsTable.id, onDelete = ReferenceOption.CASCADE)
val position = integer("position")
override val primaryKey = PrimaryKey(albumId, position)
}
/**
* The shared song store. One row per `(source, externalId)`; the same file can
* be referenced by any number of users' moods. There is no per-song owner.
*
* The UNIQUE index on `external_id` predates the source column. Deezer ids and
* YouTube video ids do not overlap, so it still works as a dedupe guard.
*/
object SongsTable : Table("songs") {
val id = varchar("id", 32)
val albumId = reference("album_id", AlbumsTable.id, onDelete = ReferenceOption.CASCADE)
val folderId = reference("folder_id", MusicFoldersTable.id, onDelete = ReferenceOption.CASCADE)
val musicSource = varchar("source", 16).default(MusicSource.YOUTUBE)
val externalId = varchar("external_id", 64).uniqueIndex()
val title = text("title")
val displayArtist = text("display_artist")
val albumName = text("album_name")
val trackNumber = integer("track_number").nullable()
val discNumber = integer("disc_number").default(1)
val year = integer("year").nullable()
val duration = integer("duration")
val bitRate = integer("bit_rate")
val size = long("size")
val suffix = varchar("suffix", 16)
val contentType = varchar("content_type", 64)
val artist = text("artist")
val album = text("album").nullable()
val duration = integer("duration").default(0)
val bitRate = integer("bit_rate").default(0)
val size = long("size").default(0)
val suffix = varchar("suffix", 16).default("mp3")
val contentType = varchar("content_type", 64).default("audio/mpeg")
val relativePath = text("relative_path")
val fileMtime = long("file_mtime")
val fileMtime = long("file_mtime").default(0)
val thumbnail = text("thumbnail").nullable()
val loudnessLufs = double("loudness_lufs").nullable()
val gainDb = double("gain_db").nullable()
val createdAt = varchar("created_at", 40)
val updatedAt = varchar("updated_at", 40).default("")
val searchText = text("search_text")
val lyrics = text("lyrics").nullable()
override val primaryKey = PrimaryKey(id)
}
object SongArtistsTable : Table("song_artists") {
val songId = reference("song_id", SongsTable.id, onDelete = ReferenceOption.CASCADE)
val artistId = reference("artist_id", ArtistsTable.id, onDelete = ReferenceOption.CASCADE)
val position = integer("position")
override val primaryKey = PrimaryKey(songId, position)
}
object SongGenresTable : Table("song_genres") {
val songId = reference("song_id", SongsTable.id, onDelete = ReferenceOption.CASCADE)
val genre = varchar("genre", 255)
override val primaryKey = PrimaryKey(songId, genre)
}
object AlbumGenresTable : Table("album_genres") {
val albumId = reference("album_id", AlbumsTable.id, onDelete = ReferenceOption.CASCADE)
val genre = varchar("genre", 255)
override val primaryKey = PrimaryKey(albumId, genre)
}
object ArtistGenresTable : Table("artist_genres") {
val artistId = reference("artist_id", ArtistsTable.id, onDelete = ReferenceOption.CASCADE)
val genre = varchar("genre", 255)
override val primaryKey = PrimaryKey(artistId, genre)
}
object UserSongStateTable : Table("user_song_state") {
val userName = reference("user_name", UsersTable.name, onDelete = ReferenceOption.CASCADE)
val songId = reference("song_id", SongsTable.id, onDelete = ReferenceOption.CASCADE)
val playCount = integer("play_count").default(0)
val lastPlayedAt = varchar("last_played_at", 40).nullable()
val starredAt = varchar("starred_at", 40).nullable()
val rating = integer("rating").nullable()
override val primaryKey = PrimaryKey(userName, songId)
}
object UserStarsTable : Table("user_stars") {
val userName = reference("user_name", UsersTable.name, onDelete = ReferenceOption.CASCADE)
val itemType = varchar("item_type", 16)
val itemId = varchar("item_id", 32)
val starredAt = varchar("starred_at", 40)
val rating = integer("rating").nullable()
override val primaryKey = PrimaryKey(userName, itemType, itemId)
}
object BookmarksTable : Table("bookmarks") {
val userName = reference("user_name", UsersTable.name, onDelete = ReferenceOption.CASCADE)
val songId = reference("song_id", SongsTable.id, onDelete = ReferenceOption.CASCADE)
val positionMs = long("position_ms")
val updatedAt = varchar("updated_at", 40)
override val primaryKey = PrimaryKey(userName, songId)
override val primaryKey = PrimaryKey(id)
}
/**
* Moods are the only collection type. A mood owns seeds (permanent anchors) and
* recommendations (YT Music candidates); playback queues are generated live and
* never materialized here.
* Moods are the only collection type. A mood owns permanent songs and an
* active recommendation pool; playback queues are generated live and never
* materialized as playlists.
*/
object MoodsTable : Table("moods") {
val id = varchar("id", 32)
@@ -146,9 +55,10 @@ object MoodsTable : Table("moods") {
val description = text("description").nullable()
val query = text("query").nullable()
val genres = text("genres").default("[]")
val targetSize = integer("target_size").default(30)
val musicSource = varchar("source", 16).default(MusicSource.YOUTUBE)
val collectionPercent = integer("collection_percent").default(50)
val recommendationCount = integer("recommendation_count").default(10)
val refreshHours = integer("refresh_hours").default(24)
val ytEnabled = bool("yt_enabled").default(true)
val lastDiscoveryAt = varchar("last_discovery_at", 40).nullable()
val createdAt = varchar("created_at", 40)
val changedAt = varchar("changed_at", 40)
@@ -157,10 +67,10 @@ object MoodsTable : Table("moods") {
override val primaryKey = PrimaryKey(id)
}
object MoodSeedsTable : Table("mood_seeds") {
object MoodSongsTable : Table("mood_songs") {
val moodId = reference("mood_id", MoodsTable.id, onDelete = ReferenceOption.CASCADE)
val songId = reference("song_id", SongsTable.id, onDelete = ReferenceOption.CASCADE)
val origin = varchar("origin", 16).default("manual")
val origin = varchar("origin", 16).default("search")
val addedAt = varchar("added_at", 40)
override val primaryKey = PrimaryKey(moodId, songId)
@@ -168,23 +78,26 @@ object MoodSeedsTable : Table("mood_seeds") {
object MoodRecommendationsTable : Table("mood_recommendations") {
val moodId = reference("mood_id", MoodsTable.id, onDelete = ReferenceOption.CASCADE)
val videoId = varchar("video_id", 64)
val musicSource = varchar("source", 16).default(MusicSource.YOUTUBE)
val externalId = varchar("external_id", 64)
val title = text("title")
val artist = text("artist")
val album = text("album").nullable()
val duration = integer("duration").nullable()
val thumbnail = text("thumbnail").nullable()
val state = varchar("state", 16).default("new")
val matchedSongId = reference(
"matched_song_id",
val songId = reference(
"song_id",
SongsTable.id,
onDelete = ReferenceOption.SET_NULL,
).nullable()
val timesAdded = integer("times_added").default(0)
val attempts = integer("attempts").default(0)
val error = text("error").nullable()
val rank = integer("rank").default(0)
val firstSeen = varchar("first_seen", 40)
val lastSeen = varchar("last_seen", 40)
override val primaryKey = PrimaryKey(moodId, videoId)
override val primaryKey = PrimaryKey(moodId, externalId)
}
/** Bearer tokens are stored as hashes; the raw value only exists on the device. */
@@ -201,6 +114,23 @@ object ApiTokensTable : Table("api_tokens") {
override val primaryKey = PrimaryKey(id)
}
object DownloadJobsTable : Table("download_jobs") {
val id = varchar("id", 32)
val musicSource = varchar("source", 16).default(MusicSource.YOUTUBE)
val externalId = varchar("external_id", 64)
val moodId = reference("mood_id", MoodsTable.id, onDelete = ReferenceOption.CASCADE)
val kind = varchar("kind", 16).default("recommendation")
val state = varchar("state", 16).default("queued")
val priority = integer("priority").default(0)
val attempts = integer("attempts").default(0)
val error = text("error").nullable()
val songId = reference("song_id", SongsTable.id, onDelete = ReferenceOption.SET_NULL).nullable()
val createdAt = varchar("created_at", 40)
val updatedAt = varchar("updated_at", 40)
override val primaryKey = PrimaryKey(id)
}
object PlaybackSessionsTable : Table("playback_sessions") {
val id = varchar("id", 32)
val userName = reference("user_name", UsersTable.name, onDelete = ReferenceOption.CASCADE)
@@ -234,10 +164,23 @@ object SongAffinityTable : Table("song_affinity") {
override val primaryKey = PrimaryKey(userName, songId)
}
object LibraryTombstonesTable : Table("library_tombstones") {
val entity = varchar("entity", 20)
val entityId = varchar("entity_id", 32)
val deletedAt = varchar("deleted_at", 40)
/**
* Batches of client diagnostics. `received_at` is the server clock and is what
* retention uses; `created_at` is the device clock and is only for display.
*/
object ClientLogsTable : Table("client_logs") {
val id = long("id").autoIncrement()
val userName = reference("user_name", UsersTable.name, onDelete = ReferenceOption.CASCADE)
val deviceId = varchar("device_id", 80)
val platform = varchar("platform", 32)
val appVersion = varchar("app_version", 40)
val sessionId = varchar("session_id", 64).nullable()
val level = varchar("level", 8)
val tag = varchar("tag", 64)
val message = text("message")
val detail = text("detail").nullable()
val createdAt = varchar("created_at", 40)
val receivedAt = varchar("received_at", 40)
override val primaryKey = PrimaryKey(entity, entityId)
override val primaryKey = PrimaryKey(id)
}
@@ -0,0 +1,399 @@
package win.taptap.musicmanager.discovery
import java.net.URI
import java.net.URLEncoder
import java.net.http.HttpClient
import java.net.http.HttpRequest
import java.net.http.HttpResponse
import java.time.Duration
import java.util.concurrent.ConcurrentHashMap
import kotlinx.serialization.SerialName
import kotlinx.serialization.Serializable
import kotlinx.serialization.json.Json
import kotlinx.serialization.json.JsonObject
import kotlinx.serialization.json.contentOrNull
import kotlinx.serialization.json.jsonObject
import kotlinx.serialization.json.jsonPrimitive
import win.taptap.musicmanager.source.MusicSource
/**
* Deezer's public catalog API. No user token is needed for search, track
* metadata or the similarity graph; the ARL is only used by the downloader.
*
* Deezer returns canonical tracks with a square album cover, which is also what
* the indexer embeds into the downloaded MP3.
*/
class DeezerSearch(
private val apiBase: String = "https://api.deezer.com",
private val timeoutSeconds: Long = 20,
) : MusicCatalog {
override val source: String = MusicSource.DEEZER
private val json = Json {
ignoreUnknownKeys = true
explicitNulls = false
}
private val http = HttpClient.newBuilder()
.connectTimeout(Duration.ofSeconds(10))
.followRedirects(HttpClient.Redirect.NORMAL)
.build()
/** Artist and album lookups are stable for a release, so cache them. */
private val artistIds = ConcurrentHashMap<String, Lookup<Long>>()
private val albumRefs = ConcurrentHashMap<String, Lookup<DeezerAlbumRefDto>>()
override fun search(query: String, limit: Int): List<TrackCandidate> {
val count = limit.coerceIn(1, MAX_RESULTS)
val url = "$apiBase/search/track?q=${encode(query)}&limit=$count&output=json"
return parseDeezerSearch(get(url))
}
override fun resolve(url: String): TrackCandidate? {
val trimmed = url.trim()
if (trimmed.isEmpty()) return null
val trackId = trackIdFrom(trimmed) ?: followShortLink(trimmed)?.let(::trackIdFrom)
?: return null
return parseDeezerTrack(get("$apiBase/track/$trackId"))
}
/**
* Walks Deezer's own graph from the entire collection:
*
* 1. the most represented artists → their top tracks;
* 2. those artists' related artists → their top tracks;
* 3. the collection's albums → their deep cuts.
*
* A mood with 40 songs by 6 artists therefore still yields fresh material
* even after the obvious artist hits are already in the mood. Every request
* is best-effort; a blocked endpoint does not fail the pass.
*/
override fun discover(seeds: DiscoverySeeds, limit: Int): List<TrackCandidate> {
if (seeds.isEmpty()) return emptyList()
val found = LinkedHashMap<String, TrackCandidate>()
val seedArtists = seeds.artists
.take(MAX_SEED_ARTISTS)
.mapNotNull { seed -> artistId(seed.name)?.let { seed to it } }
seedArtists.forEach { (_, id) ->
artistTopTracks(id, SEED_TOP_LIMIT).forEach {
add(found, it, SEED_TOP_AFFINITY)
}
// Deezer's own radio is the recommendation engine at its strongest:
// a Daft Punk radio mixes Daft Punk with MGMT, Caribou, Jamiroquai.
artistRadio(id, SEED_RADIO_LIMIT).forEach {
add(found, it, RADIO_AFFINITY)
}
}
val related = LinkedHashSet<Long>()
seedArtists.take(MAX_RELATED_SEEDS).forEach { (_, id) ->
related += relatedArtists(id, MAX_RELATED_PER_SEED)
}
val seedIds = seedArtists.map { it.second }.toSet()
related.asSequence()
.filterNot { it in seedIds }
.take(MAX_RELATED_ARTISTS)
.forEach { id ->
artistTopTracks(id, RELATED_TOP_LIMIT).forEach {
add(found, it, RELATED_AFFINITY)
}
}
seeds.albums
.take(MAX_SEED_ALBUMS)
.mapNotNull(::albumRef)
.forEach { album ->
albumTracks(album, ALBUM_TRACK_LIMIT).forEach {
add(found, it, ALBUM_AFFINITY)
}
}
return found.values.take(limit.coerceAtLeast(1))
}
/** Keeps the highest weight when the same track comes from two edges. */
private fun add(
into: MutableMap<String, TrackCandidate>,
candidate: TrackCandidate,
affinity: Double,
) {
if (candidate.externalId.isEmpty()) return
val existing = into[candidate.externalId]
if (existing == null || existing.affinity < affinity) {
into[candidate.externalId] = candidate.copy(affinity = affinity)
}
}
private fun artistId(name: String): Long? = lookup(
cache = artistIds,
key = matchKey(name),
request = "$apiBase/search/artist?q=${encode(name)}&limit=$ARTIST_LOOKUP_LIMIT&output=json",
) { body ->
val parsed = json.decodeFromString<DeezerArtistListResponse>(body)
parsed.data.firstOrNull { artistMatches(it.name, name) && it.id > 0 }?.id
?: parsed.data.firstOrNull { it.id > 0 }?.id
}
private fun albumRef(seed: SeedAlbum): DeezerAlbumRefDto? {
val titleKey = matchKey(seed.title)
if (titleKey.isEmpty()) return null
val artistKey = matchKey(seed.artist)
return lookup(
cache = albumRefs,
key = "$titleKey|$artistKey",
request = "$apiBase/search/album?q=${encode("${seed.title} ${seed.artist}")}" +
"&limit=$ALBUM_LOOKUP_LIMIT&output=json",
) { body ->
val parsed = json.decodeFromString<DeezerAlbumListResponse>(body)
parsed.data.firstOrNull { album ->
album.id > 0 &&
matchKey(album.title) == titleKey &&
(album.artist == null || artistMatches(album.artist.name, seed.artist))
}
}
}
/**
* Resolves a name to an id once. Misses are cached too, so a mood full of
* obscure artists does not re-search them on every refresh.
*/
private inline fun <T : Any> lookup(
cache: MutableMap<String, Lookup<T>>,
key: String,
request: String,
crossinline parse: (String) -> T?,
): T? {
if (key.isEmpty()) return null
cache[key]?.let { return it.value }
val value = runCatching { parse(get(request)) }.getOrNull()
cache[key] = Lookup(value)
return value
}
private fun relatedArtists(id: Long, limit: Int): List<Long> = runCatching {
val body = get("$apiBase/artist/$id/related?limit=$limit&output=json")
json.decodeFromString<DeezerArtistListResponse>(body)
.data
.map { it.id }
.filter { it > 0 }
}.getOrDefault(emptyList())
private fun artistRadio(id: Long, limit: Int): List<TrackCandidate> = runCatching {
val body = get("$apiBase/artist/$id/radio?limit=$limit&output=json")
json.decodeFromString<DeezerTracksResponse>(body).data.mapNotNull { it.toCandidate() }
}.getOrDefault(emptyList())
private fun artistTopTracks(id: Long, limit: Int): List<TrackCandidate> = runCatching {
val body = get("$apiBase/artist/$id/top?limit=$limit&output=json")
json.decodeFromString<DeezerTracksResponse>(body).data.mapNotNull { it.toCandidate() }
}.getOrDefault(emptyList())
/**
* `/album/{id}/tracks` omits the album object, so inject the album the
* collection already gave us to keep the cover and the album name.
*/
private fun albumTracks(
album: DeezerAlbumRefDto,
limit: Int,
): List<TrackCandidate> = runCatching {
val body = get("$apiBase/album/${album.id}/tracks?limit=$limit&output=json")
json.decodeFromString<DeezerTracksResponse>(body).data
.mapNotNull { it.toCandidate() }
.map { candidate ->
candidate.copy(
album = candidate.album ?: album.title.takeIf { it.isNotBlank() },
thumbnail = candidate.thumbnail ?: album.cover(),
)
}
}.getOrDefault(emptyList())
private fun get(url: String): String {
throttle()
val request = HttpRequest.newBuilder(URI.create(url))
.timeout(Duration.ofSeconds(timeoutSeconds))
.header("Accept", "application/json")
.header("User-Agent", "AudioManager/0.1")
.GET()
.build()
val response = try {
http.send(request, HttpResponse.BodyHandlers.ofString())
} catch (e: Exception) {
throw IllegalStateException("Deezer request failed: ${e.message ?: e::class.simpleName}", e)
}
if (response.statusCode() != 200) {
throw IllegalStateException("Deezer API returned HTTP ${response.statusCode()}")
}
val body = response.body()
if (body.isBlank()) throw IllegalStateException("Deezer API returned an empty response")
throwIfError(body)
return body
}
/** `deezer.page.link` redirects to a normal deezer.com URL. */
private fun followShortLink(url: String): String? {
if (!url.contains("deezer.page.link", ignoreCase = true)) return null
return runCatching {
throttle()
val request = HttpRequest.newBuilder(URI.create(url))
.timeout(Duration.ofSeconds(timeoutSeconds))
.header("User-Agent", "AudioManager/0.1")
.GET()
.build()
http.send(request, HttpResponse.BodyHandlers.discarding()).uri().toString()
}.getOrNull()
}
private fun throwIfError(body: String) {
val root = runCatching { json.parseToJsonElement(body).jsonObject }.getOrNull() ?: return
val error = root["error"] as? JsonObject ?: return
val message = error["message"]?.jsonPrimitive?.contentOrNull
?: error["type"]?.jsonPrimitive?.contentOrNull
?: "Deezer request failed"
throw IllegalStateException(message)
}
/** Keeps us well below Deezer's 50 requests per 5 seconds limit. */
private fun throttle() {
synchronized(throttleLock) {
val now = System.nanoTime()
val wait = MIN_REQUEST_INTERVAL_NANOS - (now - lastRequestNanos)
if (wait > 0) {
Thread.sleep(wait / 1_000_000, (wait % 1_000_000).toInt())
}
lastRequestNanos = System.nanoTime()
}
}
/** `value == null` is a cached miss. */
private data class Lookup<T>(val value: T?)
private companion object {
const val MAX_RESULTS = 25
/** Bounds one discovery pass to roughly twenty throttled requests. */
const val MAX_SEED_ARTISTS = 5
const val MAX_RELATED_SEEDS = 2
const val MAX_RELATED_PER_SEED = 8
const val MAX_RELATED_ARTISTS = 6
const val MAX_SEED_ALBUMS = 2
const val SEED_TOP_LIMIT = 25
const val SEED_RADIO_LIMIT = 40
const val RELATED_TOP_LIMIT = 10
const val ALBUM_TRACK_LIMIT = 40
const val ARTIST_LOOKUP_LIMIT = 5
const val ALBUM_LOOKUP_LIMIT = 5
/** A hit by an artist already in the mood; the scorer adds the rest. */
const val SEED_TOP_AFFINITY = 0.5
/** A track Deezer itself puts in that artist's radio. */
const val RADIO_AFFINITY = 2.0
/** A hit by an artist Deezer considers similar to the collection. */
const val RELATED_AFFINITY = 1.5
/** Another track from an album already in the mood. */
const val ALBUM_AFFINITY = 1.0
const val MIN_REQUEST_INTERVAL_NANOS = 120_000_000L
val THROTTLE_LOCK = Any()
fun encode(value: String): String =
URLEncoder.encode(value, Charsets.UTF_8).replace("+", "%20")
fun trackIdFrom(url: String): String? {
val match = TRACK_URL.find(url) ?: return null
return match.groupValues[1].takeIf { it.isNotEmpty() }
}
val TRACK_URL = Regex("deezer\\.com/(?:[a-z]{2}/)?track/(\\d+)", RegexOption.IGNORE_CASE)
}
private val throttleLock = THROTTLE_LOCK
@Volatile
private var lastRequestNanos = 0L
}
internal fun parseDeezerSearch(json: String): List<TrackCandidate> {
val parsed = runCatching {
Json { ignoreUnknownKeys = true; explicitNulls = false }
.decodeFromString<DeezerSearchResponse>(json)
}.getOrNull() ?: return emptyList()
return parsed.data.mapNotNull { it.toCandidate() }
}
internal fun parseDeezerTrack(json: String): TrackCandidate? {
val parsed = runCatching {
Json { ignoreUnknownKeys = true; explicitNulls = false }
.decodeFromString<DeezerTrackDto>(json)
}.getOrNull() ?: return null
return parsed.toCandidate(applyLengthFilter = false)
}
@Serializable
private data class DeezerSearchResponse(val data: List<DeezerTrackDto> = emptyList())
@Serializable
private data class DeezerTracksResponse(val data: List<DeezerTrackDto> = emptyList())
@Serializable
private data class DeezerArtistListResponse(val data: List<DeezerArtistRefDto> = emptyList())
@Serializable
private data class DeezerArtistRefDto(val id: Long = 0, val name: String = "")
@Serializable
private data class DeezerAlbumListResponse(val data: List<DeezerAlbumRefDto> = emptyList())
@Serializable
private data class DeezerAlbumRefDto(
val id: Long = 0,
val title: String = "",
val artist: DeezerArtistDto? = null,
@SerialName("cover_big") val coverBig: String? = null,
@SerialName("cover_xl") val coverXl: String? = null,
) {
fun cover(): String? = coverBig ?: coverXl
}
@Serializable
private data class DeezerTrackDto(
val id: Long = 0,
val title: String = "",
val duration: Int = 0,
val readable: Boolean = true,
val artist: DeezerArtistDto? = null,
val album: DeezerAlbumDto? = null,
) {
fun toCandidate(applyLengthFilter: Boolean = true): TrackCandidate? {
if (id <= 0 || !readable) return null
if (title.isBlank()) return null
val artistName = artist?.name?.takeIf { it.isNotBlank() } ?: return null
if (applyLengthFilter && duration > 0 &&
duration !in MIN_TRACK_SECONDS..MAX_TRACK_SECONDS
) {
return null
}
return TrackCandidate(
source = MusicSource.DEEZER,
externalId = id.toString(),
title = title.trim(),
artist = artistName.trim(),
album = album?.title?.takeIf { it.isNotBlank() },
duration = duration.takeIf { it > 0 },
thumbnail = album?.coverBig ?: album?.coverXl ?: album?.coverMedium,
)
}
}
@Serializable
private data class DeezerArtistDto(val name: String = "")
@Serializable
private data class DeezerAlbumDto(
val title: String? = null,
@SerialName("cover_big") val coverBig: String? = null,
@SerialName("cover_xl") val coverXl: String? = null,
@SerialName("cover_medium") val coverMedium: String? = null,
)
@@ -0,0 +1,368 @@
package win.taptap.musicmanager.discovery
import java.time.Instant
import java.time.temporal.ChronoUnit
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.async
import kotlinx.coroutines.awaitAll
import kotlinx.coroutines.coroutineScope
import org.slf4j.LoggerFactory
import win.taptap.musicmanager.db.MoodRecord
import win.taptap.musicmanager.db.MoodRepository
import win.taptap.musicmanager.db.RecommendationUpsert
import win.taptap.musicmanager.db.SongRecord
import win.taptap.musicmanager.util.IdGenerator
data class DiscoveryOutcome(
val stored: Int,
val skipped: Boolean = false,
val error: String? = null,
)
/**
* Turns a mood's collection, genres and free text into catalog candidates.
* Candidates are stored as hidden `new` rows; [TopUpService] promotes them into
* active download slots.
*/
class DiscoveryService(
private val catalogs: MusicCatalogs,
private val clock: () -> Instant = { Instant.now() },
) {
private val log = LoggerFactory.getLogger(DiscoveryService::class.java)
suspend fun refresh(mood: MoodRecord, force: Boolean = false): DiscoveryOutcome {
if (mood.recommendationCount <= 0) return DiscoveryOutcome(0, skipped = true)
val now = clock()
if (!force && isFresh(mood.lastDiscoveryAt, mood.refreshHours, now)) {
return DiscoveryOutcome(0, skipped = true)
}
val catalog = catalogs.forSource(mood.source)
val collection = MoodRepository.songRecords(mood.id)
val artists = artistsByFrequency(collection)
val albums = albumsOf(collection)
val seeds = DiscoverySeeds(
artists = artists,
albums = albums,
// Only songs from the mood's own catalog can seed its radio.
trackIds = radioSeedIds(collection, mood.source, RADIO_SEEDS),
hint = mood.query,
genres = mood.genres,
name = mood.name,
)
val queries = buildQueries(mood, artists.map { it.name })
if (queries.isEmpty() && seeds.isEmpty()) {
// A new mood with no hint, no genres and no songs has nothing to
// base music on. Leave the slots empty instead of downloading
// whatever the mood's name happens to match.
log.debug("Skipping discovery for mood {}: no music signal yet", mood.name)
return DiscoveryOutcome(0, skipped = true)
}
// Text queries and the catalog's similarity graph run together: the
// graph knows about related artists and album deep cuts, which a title
// search cannot express.
val results = coroutineScope {
val searches = queries.map { query ->
async(Dispatchers.IO) { runCatching { catalog.search(query, QUERY_LIMIT) } }
}
val expansion = async(Dispatchers.IO) {
if (seeds.isEmpty()) {
emptyList()
} else {
runCatching { catalog.discover(seeds, DISCOVERY_LIMIT) }
.onFailure {
log.debug(
"{} expansion failed for mood {}: {}",
catalog.source,
mood.name,
it.message,
)
}
.getOrDefault(emptyList())
}
}
searches.awaitAll() to expansion.await()
}
val searches = results.first
val discovered = results.second
val failures = searches.filter { it.isFailure }
if (searches.isNotEmpty() && failures.size == searches.size && discovered.isEmpty()) {
val message = failures.firstNotNullOfOrNull { it.exceptionOrNull()?.message }
?: "Search failed"
log.warn("{} discovery failed for mood {}: {}", catalog.source, mood.name, message)
MoodRepository.markError(mood.id, message)
return DiscoveryOutcome(0, error = message)
}
val candidates = mergeCandidates(
(searches.mapNotNull { it.getOrNull() }.flatten() + discovered)
.filter { isMusicLength(it.duration) },
)
if (candidates.isEmpty()) {
val message = "${catalog.source} returned no usable results"
MoodRepository.markError(mood.id, message)
return DiscoveryOutcome(0, error = message)
}
val excluded = MoodRepository.externalIdsInMood(mood.id, mood.source) +
MoodRepository.decidedExternalIds(mood.id, mood.source) +
MoodRepository.allRecommendations(mood.id)
.filter { it.source == mood.source && it.state != "new" }
.map { it.externalId }
.toSet()
val moodText = listOfNotNull(mood.query, mood.description, mood.name).joinToString(" ")
// A different upload of a song already in the mood is still a duplicate.
val collectionRecordings = collection
.map { recordingKey(it.artist, it.title) }
.toSet()
val artistWeights = artists
.associate { matchKey(it.name) to it.songs.toDouble() }
.filterKeys { it.isNotEmpty() }
val albumKeys = albums
.map { matchKey(it.title) }
.filter { it.isNotEmpty() }
.toSet()
val ranked = rankCandidates(
candidates.values
.filterNot { it.externalId in excluded }
.filterNot { recordingKey(it.artist, it.title) in collectionRecordings }
.map { candidate ->
candidate to scoreCandidate(candidate, artistWeights, albumKeys, moodText)
},
CANDIDATE_POOL,
)
val upserts = ranked.map { candidate ->
RecommendationUpsert(
source = mood.source,
externalId = candidate.externalId,
title = candidate.title,
artist = candidate.artist,
album = candidate.album,
duration = candidate.duration,
thumbnail = candidate.thumbnail,
)
}
if (upserts.isEmpty()) {
MoodRepository.markDiscovered(mood.id)
return DiscoveryOutcome(0)
}
val stored = MoodRepository.storeCandidates(
mood.id,
mood.source,
upserts,
now.toString(),
CANDIDATE_POOL,
)
MoodRepository.markDiscovered(mood.id)
log.info(
"{} discovery for mood {} stored {} candidates from {} searches and {} expanded hits",
catalog.source,
mood.name,
stored,
queries.size,
discovered.size,
)
return DiscoveryOutcome(stored = stored)
}
private companion object {
const val QUERY_LIMIT = 10
/** The graph expands far more than a single search page does. */
const val DISCOVERY_LIMIT = 200
const val RADIO_SEEDS = 2
const val CANDIDATE_POOL = 60
}
}
internal fun isFresh(lastDiscoveryAt: String?, refreshHours: Int, now: Instant): Boolean {
if (lastDiscoveryAt == null) return false
val at = runCatching { Instant.parse(lastDiscoveryAt) }.getOrNull() ?: return false
return at.plus(refreshHours.toLong().coerceAtLeast(1), ChronoUnit.HOURS).isAfter(now)
}
/** Search and discovery only accept music-length tracks: 30 seconds to 10 minutes. */
internal fun isMusicLength(seconds: Int?): Boolean =
seconds == null || seconds in MIN_TRACK_SECONDS..MAX_TRACK_SECONDS
internal fun buildQueries(mood: MoodRecord, seedArtists: List<String>): List<String> {
val queries = LinkedHashSet<String>()
mood.query?.trim()?.takeIf { it.isNotEmpty() }?.let(queries::add)
// The collection is the strongest signal, so its artists fill the slots
// before genres do. They arrive ordered by how many songs the mood holds.
seedArtists.map { it.trim() }.filter { it.isNotEmpty() }.distinct().take(3).forEach {
queries += it
}
mood.genres.map { it.trim() }.filter { it.isNotEmpty() }.take(2).forEach {
queries += "$it mix"
}
return queries.distinctBy { it.lowercase() }.take(4)
}
/** Collection artists, most represented first. */
internal fun artistsByFrequency(collection: List<SongRecord>): List<SeedArtist> =
collection
.map { it.artist.trim() }
.filter { it.isNotEmpty() }
.groupingBy { it }
.eachCount()
.entries
.sortedWith(
compareByDescending<Map.Entry<String, Int>> { it.value }.thenBy { it.key.lowercase() },
)
.map { SeedArtist(it.key, it.value) }
/**
* The most recent `(source, externalId)` songs of the mood's catalog, one per
* artist: a radio seeded by three tracks of the same artist adds nothing.
*/
internal fun radioSeedIds(
collection: List<SongRecord>,
source: String,
limit: Int,
): List<String> = collection.asReversed()
.filter { it.source == source }
.distinctBy { matchKey(it.artist) }
.take(limit)
.map { it.externalId }
/** Distinct albums of the collection, deduplicated loosely. */
internal fun albumsOf(collection: List<SongRecord>): List<SeedAlbum> =
collection.mapNotNull { song ->
val album = song.album?.trim()?.takeIf { it.isNotEmpty() } ?: return@mapNotNull null
SeedAlbum(album, song.artist)
}.distinctBy { "${matchKey(it.title)}|${matchKey(it.artist)}" }
/**
* Keeps one copy per recording. A radio and an album endpoint may return the
* same song twice, and two uploads of one song only differ by external id; the
* copy with the stronger catalog affinity wins.
*/
internal fun mergeCandidates(
items: Iterable<TrackCandidate>,
): LinkedHashMap<String, TrackCandidate> {
val byId = LinkedHashMap<String, TrackCandidate>()
val idForRecording = HashMap<String, String>()
items.forEach { candidate ->
val recording = recordingKey(candidate.artist, candidate.title)
val currentId = idForRecording[recording]
val current = currentId?.let(byId::get)
when {
current == null -> {
byId[candidate.externalId] = candidate
idForRecording[recording] = candidate.externalId
}
candidate.affinity > current.affinity -> {
byId.remove(current.externalId)
byId[candidate.externalId] = candidate
idForRecording[recording] = candidate.externalId
}
}
}
return byId
}
/** Identity of a recording, independent of the catalog's track id. */
internal fun recordingKey(artist: String, title: String): String =
"${matchKey(artist)}|${matchKey(title)}"
/**
* Picks the best [limit] candidates while penalising an artist that keeps
* winning: without it, a mood with three songs by one artist would fill every
* slot with that artist. Each additional pick by the same artist costs
* [ARTIST_REPEAT_PENALTY], so a related artist eventually takes a slot.
*/
internal fun rankCandidates(
scored: List<Pair<TrackCandidate, Double>>,
limit: Int,
): List<TrackCandidate> {
val remaining = scored.toMutableList()
val chosen = ArrayList<TrackCandidate>(limit.coerceAtMost(remaining.size))
val perArtist = HashMap<String, Int>()
while (chosen.size < limit && remaining.isNotEmpty()) {
var bestIndex = 0
var bestScore = Double.NEGATIVE_INFINITY
for (index in remaining.indices) {
val candidate = remaining[index].first
val picked = perArtist[matchKey(candidate.artist)] ?: 0
val adjusted = remaining[index].second - ARTIST_REPEAT_PENALTY * picked
if (adjusted > bestScore) {
bestScore = adjusted
bestIndex = index
}
}
val candidate = remaining.removeAt(bestIndex).first
chosen += candidate
val key = matchKey(candidate.artist)
perArtist[key] = (perArtist[key] ?: 0) + 1
}
return chosen
}
/**
* Ranks a candidate against the whole collection:
*
* - an artist already in the mood is the strongest signal, and an artist with
* more songs in the mood ranks higher;
* - another track from an album already in the mood is a strong signal;
* - a catalog-supplied [TrackCandidate.affinity] (related artist, top track)
* is added as-is;
* - the mood's own words in the title are a tie-breaker.
*/
internal fun scoreCandidate(
candidate: TrackCandidate,
artistWeights: Map<String, Double>,
albumKeys: Set<String>,
moodText: String,
): Double {
var score = candidate.affinity
val artist = matchKey(candidate.artist)
val exact = artistWeights[artist]
when {
exact != null -> score += 3.0 + (0.5 * (exact - 1)).coerceAtMost(1.5)
artist.isNotEmpty() && artistWeights.keys.any { artistMatches(artist, it) } -> score += 1.5
}
val album = matchKey(candidate.album.orEmpty())
if (album.isNotEmpty() && album in albumKeys) score += 1.0
val text = matchKey(moodText)
if (text.isNotEmpty() && matchKey(candidate.title).contains(text)) score += 1.0
if ((candidate.duration ?: 0) > 8 * 60) score -= 1.0
return score
}
private val BRACKETS = Regex("\\([^)]*\\)|\\[[^\\]]*\\]")
private val NON_WORD = Regex("[^\\p{L}\\p{N} ]+")
private val SPACES = Regex("\\s+")
private val FEATURING = Regex("\\b(feat|ft|featuring)\\b.*$", RegexOption.IGNORE_CASE)
/** Loose key for comparing YT titles and artist names with downloaded tags. */
internal fun matchKey(value: String): String =
IdGenerator.normalize(value)
.replace(BRACKETS, " ")
.replace(FEATURING, " ")
.replace(NON_WORD, " ")
.replace(SPACES, " ")
.trim()
internal fun artistMatches(a: String, b: String): Boolean {
val left = matchKey(a)
val right = matchKey(b)
if (left.isEmpty() || right.isEmpty()) return false
if (left == right) return true
// Short names like "A" used to match every artist containing the letter.
if (left.length < 3 || right.length < 3) return false
return " $left ".contains(" $right ") || " $right ".contains(" $left ")
}
internal const val MIN_TRACK_SECONDS = 30
internal const val MAX_TRACK_SECONDS = 600
internal const val ARTIST_REPEAT_PENALTY = 1.0
@@ -0,0 +1,98 @@
package win.taptap.musicmanager.discovery
import win.taptap.musicmanager.source.MusicSource
/**
* One source-specific search hit, already split into an artist and a title.
*
* [affinity] is a catalog-supplied similarity weight: the catalog knows where a
* hit came from (a related artist, an album deep cut), which is information a
* title search cannot express.
*/
data class TrackCandidate(
val source: String,
val externalId: String,
val title: String,
val artist: String,
val album: String? = null,
val duration: Int? = null,
val thumbnail: String? = null,
val affinity: Double = 0.0,
)
/** One artist of the mood collection and how many of its songs the mood holds. */
data class SeedArtist(val name: String, val songs: Int)
/** One album of the mood collection. */
data class SeedAlbum(val title: String, val artist: String)
/**
* The whole mood collection, handed to a catalog that can expand it through its
* own similarity graph instead of title searches alone.
*/
data class DiscoverySeeds(
val artists: List<SeedArtist> = emptyList(),
val albums: List<SeedAlbum> = emptyList(),
/**
* External ids of collection songs that belong to this catalog, most recent
* first. They seed the catalog's own "radio"/play-next queue.
*/
val trackIds: List<String> = emptyList(),
val hint: String? = null,
val genres: List<String> = emptyList(),
val name: String = "",
) {
fun isEmpty(): Boolean = artists.isEmpty() && albums.isEmpty() && trackIds.isEmpty()
}
/** A music catalog that can be searched and can resolve a pasted URL. */
interface MusicCatalog {
val source: String
fun search(query: String, limit: Int): List<TrackCandidate>
/** Resolves a pasted track URL, or returns null when it is not a track. */
fun resolve(url: String): TrackCandidate?
/**
* Expands the whole collection through the catalog's own similarity graph:
* related artists, artist top tracks and album deep cuts. Catalogs without
* such an API keep the default empty list and rely on text queries.
*
* Implementations must be best-effort: one failing sub-request returns what
* the other requests found instead of failing the whole discovery pass.
*/
fun discover(seeds: DiscoverySeeds, limit: Int): List<TrackCandidate> = emptyList()
}
/**
* Dispatches by source and detects the source of a pasted URL. Deezer links are
* handled by the Deezer catalog; everything else is treated as a YouTube URL
* because that is what `yt-dlp` understands.
*/
class MusicCatalogs(private val catalogs: Map<String, MusicCatalog>) {
init {
MusicSource.ALL.forEach { source ->
require(source in catalogs) { "Missing music catalog for $source" }
}
}
fun forSource(source: String): MusicCatalog =
catalogs[source] ?: throw IllegalArgumentException("Unknown music source: $source")
fun resolve(url: String): TrackCandidate? {
val trimmed = url.trim()
if (trimmed.isEmpty()) return null
val catalog = if (isDeezerUrl(trimmed)) {
catalogs.getValue(MusicSource.DEEZER)
} else {
catalogs.getValue(MusicSource.YOUTUBE)
}
return catalog.resolve(trimmed)
}
private fun isDeezerUrl(url: String): Boolean =
url.contains("deezer.com", ignoreCase = true) ||
url.contains("deezer.page.link", ignoreCase = true)
}
@@ -0,0 +1,94 @@
package win.taptap.musicmanager.discovery
import kotlinx.coroutines.CoroutineScope
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.launch
import kotlinx.coroutines.sync.Mutex
import kotlinx.coroutines.sync.withLock
import org.slf4j.LoggerFactory
import win.taptap.musicmanager.db.DownloadRepository
import win.taptap.musicmanager.db.MoodRecord
import win.taptap.musicmanager.db.MoodRepository
data class TopUpResult(
val promoted: Int,
val discoveryError: String? = null,
)
/**
* Keeps a mood's active recommendation slots at `recommendationCount`.
*
* Order: promote hidden candidates first; only run (rate-limited) discovery
* when the pool cannot fill the missing slots. User actions pass `force = true`
* so an add/reject/retry refills immediately.
*/
class TopUpService(
private val discovery: DiscoveryService,
private val backgroundScope: CoroutineScope? = null,
) {
private val log = LoggerFactory.getLogger(TopUpService::class.java)
private val mutex = Mutex()
/** Fire-and-forget refill used by API handlers after a user decision. */
fun trigger(mood: MoodRecord) {
val scope = backgroundScope ?: return
scope.launch(Dispatchers.IO) {
runCatching { topUp(mood, force = true) }
.onFailure { log.warn("Recommendation top-up for mood {} failed", mood.name, it) }
}
}
/**
* Same, for work that only knows the id: a finished download attaches a song
* and changes the collection the next discovery pass is based on.
*/
fun trigger(moodId: String) {
val scope = backgroundScope ?: return
scope.launch(Dispatchers.IO) {
val mood = MoodRepository.byId(moodId) ?: return@launch
runCatching { topUp(mood, force = true) }
.onFailure { log.warn("Recommendation top-up for mood {} failed", mood.name, it) }
}
}
suspend fun topUp(mood: MoodRecord, force: Boolean = false): TopUpResult = mutex.withLock {
var current = MoodRepository.get(mood.id, mood.owner) ?: return@withLock TopUpResult(0)
if (current.recommendationCount <= 0) return@withLock TopUpResult(0)
var promoted = promoteMissing(current)
val stillMissing = current.recommendationCount - MoodRepository.activeCount(current.id)
var discoveryError: String? = null
val collectionChanged = MoodRepository.songsAddedSince(current.id, current.lastDiscoveryAt)
if ((stillMissing > 0 || collectionChanged) &&
(force || MoodRepository.isDiscoveryDue(current))
) {
val outcome = discovery.refresh(current, force)
discoveryError = outcome.error
current = MoodRepository.get(mood.id, mood.owner) ?: current
promoted += promoteMissing(current)
}
if (promoted > 0) {
log.info("Top-up promoted {} recommendation slots for mood {}", promoted, mood.name)
}
TopUpResult(promoted, discoveryError)
}
private fun promoteMissing(mood: MoodRecord): Int {
val active = MoodRepository.activeCount(mood.id)
val missing = mood.recommendationCount - active
if (missing <= 0) return 0
val promoted = MoodRepository.promoteCandidates(mood.id, mood.source, missing)
promoted.forEach { recommendation ->
DownloadRepository.enqueue(
source = mood.source,
externalId = recommendation.externalId,
moodId = mood.id,
kind = DownloadRepository.KIND_RECOMMENDATION,
)
}
return promoted.size
}
}
@@ -0,0 +1,36 @@
package win.taptap.musicmanager.discovery
private const val NOISE_WORDS =
"official|lyrics?|audio|video|visuali[sz]er|mv|hd|4k|remaster(?:ed)?"
private val TITLE_NOISE = Regex(
"\\((?:$NOISE_WORDS)[^)]*\\)" +
"|\\[(?:$NOISE_WORDS)[^\\]]*\\]" +
"|\\s*\\|\\s*(?:$NOISE_WORDS).*$",
RegexOption.IGNORE_CASE,
)
private val WHITESPACE = Regex("\\s+")
/**
* Removes the upload noise that YouTube titles carry ("(Official Video)",
* "[4K Remaster]", "| Lyrics"), so a search hit, a download tag and an existing
* collection row compare as the same song.
*/
internal fun cleanTrackTitle(rawTitle: String): String =
rawTitle.replace(TITLE_NOISE, "")
.replace(WHITESPACE, " ")
.trim()
.ifEmpty { rawTitle.trim() }
/**
* `"Daft Punk - Get Lucky"` tagged by the plain YouTube extractor still has the
* artist in the title. Strip it when it matches the tagged artist.
*/
internal fun stripArtistPrefix(title: String, artist: String): String {
val name = artist.trim()
if (name.isEmpty() || name.equals("Unknown artist", ignoreCase = true)) return title
val prefix = "$name - "
if (!title.startsWith(prefix, ignoreCase = true)) return title
return title.substring(prefix.length).trim().ifEmpty { title }
}
@@ -0,0 +1,294 @@
package win.taptap.musicmanager.discovery
import java.net.URLEncoder
import java.util.concurrent.TimeUnit
import kotlin.concurrent.thread
import kotlinx.serialization.json.Json
import kotlinx.serialization.json.JsonObject
import kotlinx.serialization.json.JsonPrimitive
import kotlinx.serialization.json.contentOrNull
import kotlinx.serialization.json.jsonArray
import kotlinx.serialization.json.jsonObject
import win.taptap.musicmanager.source.MusicSource
/**
* Talks to YouTube through `yt-dlp`, preferring the YouTube Music catalog.
*
* The music search returns canonical songs with an artist, album and duration,
* so discovery does not pick arbitrary user uploads, live rips or compilations.
* Each hit has to be resolved (a flat playlist only exposes ids and titles),
* which is why a search is slower than the plain YouTube search. `ytsearch` is
* kept as a fallback for bot checks or extractor changes.
*/
class YtDlpSearch(
private val executable: String,
private val timeoutSeconds: Long = 60,
) : MusicCatalog {
override val source: String = MusicSource.YOUTUBE
override fun search(query: String, limit: Int): List<TrackCandidate> {
val count = limit.coerceIn(1, MAX_RESULTS)
// yt-dlp resolves every entry of the search result page, so keep the
// retry counts low: a bot check must fail in seconds, not minutes.
val music = try {
run(
listOf(
"--playlist-end",
count.toString(),
"--dump-single-json",
"--no-warnings",
"--ignore-errors",
"--ignore-config",
"--no-cache-dir",
"--socket-timeout",
"10",
"--extractor-retries",
"1",
"--retries",
"2",
musicSearchUrl(query),
),
timeoutSeconds,
)
} catch (e: Exception) {
null
}
val fromMusic = music?.let(::parseYtSearchOutput).orEmpty()
if (fromMusic.isNotEmpty()) return fromMusic
val output = run(
listOf(
"--flat-playlist",
"--dump-single-json",
"--no-warnings",
"--ignore-errors",
"--ignore-config",
"--no-cache-dir",
"--socket-timeout",
"10",
"--extractor-retries",
"1",
"ytsearch$count:$query",
),
timeoutSeconds,
)
return parseYtSearchOutput(output)
}
/**
* YouTube Music's own radio queue: `list=RDAMVM<videoId>` is the mix the
* player builds when a song ends. It needs no account and works through the
* same yt-dlp call as search, so a YouTube mood gets real recommendations
* instead of only text matches.
*/
override fun discover(seeds: DiscoverySeeds, limit: Int): List<TrackCandidate> {
val ids = seeds.trackIds.take(MAX_RADIO_SEEDS)
if (ids.isEmpty()) return emptyList()
val found = LinkedHashMap<String, TrackCandidate>()
ids.forEach { id ->
runCatching { radio(id) }.getOrDefault(emptyList()).forEach { candidate ->
if (isMusicLength(candidate.duration)) {
found.putIfAbsent(
candidate.externalId,
candidate.copy(affinity = RADIO_AFFINITY),
)
}
}
}
return found.values.take(limit)
}
private fun radio(videoId: String): List<TrackCandidate> {
val output = run(
listOf(
"--flat-playlist",
"--playlist-end",
RADIO_LIMIT.toString(),
"--dump-single-json",
"--no-warnings",
"--ignore-errors",
"--ignore-config",
"--no-cache-dir",
"--socket-timeout",
"10",
"--extractor-retries",
"1",
"https://music.youtube.com/watch?v=$videoId&list=RDAMVM$videoId",
),
timeoutSeconds,
)
return parseYtSearchOutput(output)
}
override fun resolve(url: String): TrackCandidate? {
val trimmed = url.trim()
if (trimmed.isEmpty()) return null
val output = run(
listOf(
"--dump-single-json",
"--no-playlist",
"--no-warnings",
"--ignore-config",
"--no-cache-dir",
"--socket-timeout",
"10",
trimmed,
),
timeoutSeconds,
)
return parseYtVideo(output)
}
private fun run(args: List<String>, timeout: Long): String {
val process = ProcessBuilder(listOf(executable) + args)
// Warnings and ERROR lines must not be mixed into stdout: the JSON
// parser would fail and look like an empty result set.
.redirectErrorStream(false)
.start()
val output = StringBuilder()
val errors = StringBuilder()
val reader = thread(isDaemon = true, name = "yt-dlp-stdout") {
process.inputStream.bufferedReader().useLines { lines ->
lines.forEach { output.appendLine(it) }
}
}
val errorReader = thread(isDaemon = true, name = "yt-dlp-stderr") {
process.errorStream.bufferedReader().useLines { lines ->
lines.forEach { errors.appendLine(it) }
}
}
if (!process.waitFor(timeout, TimeUnit.SECONDS)) {
process.destroyForcibly()
reader.join(2_000)
errorReader.join(2_000)
throw IllegalStateException("yt-dlp timed out after ${timeout}s")
}
reader.join(2_000)
errorReader.join(2_000)
if (process.exitValue() != 0) {
val detail = (errors.toString().ifBlank { output.toString() }).take(400).trim()
throw IllegalStateException("yt-dlp exited with ${process.exitValue()}: $detail")
}
return output.toString()
}
private companion object {
const val MAX_RESULTS = 25
const val MAX_RADIO_SEEDS = 2
const val RADIO_LIMIT = 25
const val RADIO_AFFINITY = 2.0
/**
* YouTube Music's "Songs" filter. Without it the result page also
* contains channels, albums and playlists, and yt-dlp would spend most
* of the time resolving those instead of tracks.
*/
const val MUSIC_SONGS_FILTER = "EgWKAQIIAWoKEAkQBRAKEAMQBA%3D%3D"
fun musicSearchUrl(query: String): String {
val encoded = URLEncoder.encode(query, Charsets.UTF_8).replace("+", "%20")
return "https://music.youtube.com/search?q=$encoded&sp=$MUSIC_SONGS_FILTER"
}
}
}
internal fun parseYtSearchOutput(json: String): List<TrackCandidate> {
if (json.isBlank()) return emptyList()
val root = runCatching { Json.parseToJsonElement(json).jsonObject }.getOrNull() ?: return emptyList()
val entries = runCatching { root["entries"]?.jsonArray }.getOrNull() ?: return emptyList()
return entries.mapNotNull { element ->
runCatching { element.jsonObject }.getOrNull()?.toCandidate()
}
}
/** Parses `yt-dlp --dump-single-json <url>` for a pasted URL. */
internal fun parseYtVideo(json: String): TrackCandidate? {
if (json.isBlank()) return null
val root = runCatching { Json.parseToJsonElement(json).jsonObject }.getOrNull() ?: return null
return root.toCandidate()
}
private fun JsonObject.toCandidate(): TrackCandidate? {
val externalId = string("id")?.takeIf { it.isNotBlank() } ?: return null
// Search result pages also contain channels, albums and playlists. Their
// browse ids are longer than an 11-character video id.
if (externalId.length > MAX_VIDEO_ID_LENGTH) return null
val rawTitle = string("title")?.takeIf { it.isNotBlank() } ?: return null
val musicArtist = artistName()
if (musicArtist != null) {
// YouTube Music already separates artist and title, so do not split on
// " - " (a song title may legitimately contain a dash).
return TrackCandidate(
source = MusicSource.YOUTUBE,
externalId = externalId,
title = cleanTrackTitle(rawTitle),
artist = musicArtist,
album = string("album")?.takeIf { it.isRealValue() },
duration = number("duration")?.toInt(),
thumbnail = thumbnail(),
)
}
val channel = string("channel") ?: string("uploader").orEmpty()
val (artist, title) = splitYtTitle(rawTitle, channel)
return TrackCandidate(
source = MusicSource.YOUTUBE,
externalId = externalId,
title = title.ifEmpty { rawTitle },
artist = artist.ifEmpty { "Unknown artist" },
album = string("album")?.takeIf { it.isRealValue() },
duration = number("duration")?.toInt(),
thumbnail = thumbnail(),
)
}
/** Reads the explicit artist fields that resolved YouTube Music hits provide. */
private fun JsonObject.artistName(): String? {
string("artist")?.takeIf { it.isRealValue() }?.let { return it }
val artists = runCatching { get("artists")?.jsonArray }.getOrNull()
?.mapNotNull { (it as? JsonPrimitive)?.contentOrNull?.trim() }
?.filter { it.isNotEmpty() && it != "NA" }
.orEmpty()
return artists.takeIf { it.isNotEmpty() }?.joinToString(", ")
}
private fun String.isRealValue(): Boolean =
isNotBlank() && !equals("NA", ignoreCase = true) && !equals("null", ignoreCase = true)
private const val MAX_VIDEO_ID_LENGTH = 11
/** "Artist - Track (Official Video)" becomes ("Artist", "Track"). */
internal fun splitYtTitle(rawTitle: String, channel: String): Pair<String, String> {
val cleaned = cleanTrackTitle(rawTitle)
for (separator in listOf(" - ", " – ", " — ")) {
val index = cleaned.indexOf(separator)
if (index > 0) {
val artist = cleaned.substring(0, index).trim()
val title = cleaned.substring(index + separator.length).trim()
if (artist.isNotEmpty() && title.isNotEmpty()) return artist to title
}
}
val artist = channel.removeSuffix(" - Topic").trim()
return artist to cleaned
}
private fun JsonObject.string(key: String): String? =
(this[key] as? JsonPrimitive)?.contentOrNull
private fun JsonObject.number(key: String): Double? =
(this[key] as? JsonPrimitive)?.contentOrNull?.toDoubleOrNull()
private fun JsonObject.thumbnail(): String? {
val thumbnails = runCatching { get("thumbnails")?.jsonArray }.getOrNull()
val last = thumbnails
?.mapNotNull { runCatching { it.jsonObject.string("url") }.getOrNull() }
?.lastOrNull()
return last ?: string("thumbnail")
}
@@ -0,0 +1,127 @@
package win.taptap.musicmanager.download
import java.nio.file.Files
import java.nio.file.Path
import java.nio.file.attribute.PosixFilePermissions
import java.util.concurrent.TimeUnit
import kotlin.concurrent.thread
/** A configuration or credential problem that retrying cannot fix. */
class PermanentDownloadException(message: String) : IllegalStateException(message)
/**
* Downloads a Deezer track through `streamrip` (`rip`).
*
* Streamrip needs the account ARL and writes proper artist/album/cover tags.
* The ARL is placed in a temporary config file per job, not in the media store
* or the process arguments.
*/
class DeezerDownloader(
private val executable: String = "rip",
private val arl: String?,
private val quality: Int = 0,
private val timeoutSeconds: Long = 600,
) : AudioDownloader {
override fun download(externalId: String, targetDir: Path): Path {
if (arl.isNullOrBlank()) {
throw PermanentDownloadException(
"Deezer is not configured: set DEEZER_ARL or use YouTube Music",
)
}
if (quality !in 0..1) {
throw PermanentDownloadException("DEEZER_QUALITY must be 0 (128 kbps) or 1 (320 kbps)")
}
Files.createDirectories(targetDir)
val config = writeConfig(targetDir)
val home = Files.createDirectories(targetDir.resolve(".streamrip-home"))
try {
val process = ProcessBuilder(
executable,
"--config-path",
config.toString(),
"-f",
targetDir.toString(),
"-q",
quality.toString(),
"--no-db",
"--no-progress",
"url",
"https://www.deezer.com/track/$externalId",
)
.redirectErrorStream(false)
.apply {
environment()["HOME"] = home.toString()
environment()["XDG_CONFIG_HOME"] = home.resolve("config").toString()
environment()["XDG_CACHE_HOME"] = home.resolve("cache").toString()
environment()["XDG_DATA_HOME"] = home.resolve("data").toString()
}
.start()
val output = StringBuilder()
val errors = StringBuilder()
val reader = thread(isDaemon = true, name = "streamrip-stdout") {
process.inputStream.bufferedReader().useLines { lines ->
lines.forEach { output.appendLine(it) }
}
}
val errorReader = thread(isDaemon = true, name = "streamrip-stderr") {
process.errorStream.bufferedReader().useLines { lines ->
lines.forEach { errors.appendLine(it) }
}
}
if (!process.waitFor(timeoutSeconds, TimeUnit.SECONDS)) {
process.destroyForcibly()
reader.join(2_000)
errorReader.join(2_000)
throw IllegalStateException("streamrip timed out after ${timeoutSeconds}s")
}
reader.join(2_000)
errorReader.join(2_000)
if (process.exitValue() != 0) {
val detail = (errors.toString().ifBlank { output.toString() }).trim().takeLast(600)
throw IllegalStateException("streamrip exited with ${process.exitValue()}: $detail")
}
return Files.list(targetDir).use { stream ->
stream
.filter { it.fileName.toString().endsWith(".mp3", ignoreCase = true) }
.max(Comparator.comparingLong { runCatching { Files.size(it) }.getOrDefault(0L) })
.orElseThrow {
IllegalStateException("streamrip did not produce an MP3 file")
}
}
} finally {
Files.deleteIfExists(config)
}
}
private fun writeConfig(targetDir: Path): Path {
val template = DeezerDownloader::class.java
.getResourceAsStream("/streamrip.toml")
?.bufferedReader()
?.use { it.readText() }
?: throw IllegalStateException("streamrip.toml is missing from the application resources")
val folder = targetDir.toAbsolutePath().toString()
.replace('\\', '/')
.replace("\"", "\\\"")
val secret = arl.orEmpty()
.replace("\\", "\\\\")
.replace("\"", "\\\"")
val text = template
.replace("%%FOLDER%%", folder)
.replace("%%ARL%%", secret)
.replace("%%QUALITY%%", quality.toString())
val file = Files.createTempFile(targetDir, "streamrip-", ".toml")
runCatching {
Files.setPosixFilePermissions(file, PosixFilePermissions.fromString("rw-------"))
}
Files.writeString(file, text)
return file
}
}
@@ -0,0 +1,204 @@
package win.taptap.musicmanager.download
import java.io.IOException
import java.nio.file.AccessDeniedException
import java.nio.file.Files
import java.nio.file.Path
import java.nio.file.ReadOnlyFileSystemException
import java.nio.file.StandardCopyOption
import org.slf4j.LoggerFactory
import win.taptap.musicmanager.db.DownloadJobRecord
import win.taptap.musicmanager.db.DownloadRepository
import win.taptap.musicmanager.db.MoodRepository
import win.taptap.musicmanager.db.RecommendationCompletion
import win.taptap.musicmanager.db.SongRecord
import win.taptap.musicmanager.db.SongRepository
import win.taptap.musicmanager.media.LoudnessAnalyzer
import win.taptap.musicmanager.media.MediaGcService
import win.taptap.musicmanager.media.MediaStore
import win.taptap.musicmanager.scanner.Mp3TagExtractor
import win.taptap.musicmanager.scanner.SongIndexer
/**
* Downloads queued tracks, moves the MP3 into the shared media store, indexes it
* immediately and applies the job outcome:
*
* - `kind=song` attaches the indexed song to the requesting mood;
* - `kind=recommendation` promotes the recommendation slot to `ready`, or
* attaches it directly when the user pressed Add while it was downloading.
*
* A rejection that happens mid-download wins; the file is then collected by the
* GC because nothing references it.
*/
class DownloadService(
private val downloaders: SourceDownloaders,
private val mediaStore: MediaStore,
private val indexer: SongIndexer,
private val gc: MediaGcService,
private val tagExtractor: Mp3TagExtractor = Mp3TagExtractor(),
/** Null disables per-track normalization (for example when ffmpeg is absent). */
private val loudnessAnalyzer: LoudnessAnalyzer? = null,
private val tempDir: Path,
private val maxAttempts: Int = 3,
/**
* Called after a manual download becomes a permanent mood song. The mood
* collection is the discovery seed, so the recommendation pool has to be
* refreshed instead of waiting for the hourly worker.
*/
private val onSongAttached: (String) -> Unit = {},
) {
private val log = LoggerFactory.getLogger(DownloadService::class.java)
/** Processes at most one job. Returns false when the queue is empty. */
fun processOne(): Boolean {
val job = DownloadRepository.claimNext() ?: return false
try {
process(job)
} catch (t: Throwable) {
val message = describe(t)
log.warn(
"Download {}/{} failed (attempt {}): {}",
job.source,
job.externalId,
job.attempts,
message,
)
if (job.attempts >= maxAttempts || isPermanent(t)) {
DownloadRepository.markFailed(job.id, message)
if (job.kind == DownloadRepository.KIND_RECOMMENDATION) {
MoodRepository.failRecommendation(
job.moodId,
job.externalId,
message,
job.attempts,
)
}
} else {
DownloadRepository.retryLater(job.id, message)
}
}
return true
}
private fun process(job: DownloadJobRecord) {
// A previous job (other mood or other user) may already have the file.
val existing = SongRepository.byExternalId(job.source, job.externalId)
val existingFile = existing?.let { mediaStore.file(it) }
if (existing != null && existingFile != null && Files.isRegularFile(existingFile)) {
// The shared file may predate loudness analysis; measure it before
// finishing so this job's track is normalized too.
val analyzed = loudnessAnalyzer
?.let { analyzer ->
runCatching { analyzer.analyzeAndStore(existing, existingFile) }
.onFailure { log.warn("Loudness analysis failed for {}", existingFile, it) }
.getOrDefault(existing)
}
?: existing
finish(job, analyzed)
return
}
val working = tempDir.resolve(job.id)
runCatching { working.toFile().deleteRecursively() }
Files.createDirectories(working)
try {
val downloaded = downloaders.download(job.source, job.externalId, working)
val extracted = tagExtractor.extract(downloaded)
?: throw IllegalStateException("Downloaded file has no readable MP3 metadata")
val target = mediaStore.target(job.source, job.externalId, extracted.title)
Files.createDirectories(target.parent)
try {
Files.move(downloaded, target, StandardCopyOption.REPLACE_EXISTING)
} catch (e: IOException) {
throw IllegalStateException(
"Downloaded file could not be moved to $target: ${describe(e)}",
e,
)
}
log.info("Downloaded {}/{} to {}", job.source, job.externalId, target)
// A re-download may follow a title change; keep one file per track.
Files.list(target.parent).use { stream ->
stream.filter { it != target && it.fileName.toString().endsWith(".mp3", ignoreCase = true) }
.forEach { runCatching { Files.deleteIfExists(it) } }
}
val indexed = indexer.index(job.source, job.externalId, target)
?: throw IllegalStateException("Downloaded file could not be indexed")
// Analyze before finishing so the first play already has a gain.
// A failed measurement must never fail the download.
val song = loudnessAnalyzer
?.let { analyzer ->
runCatching { analyzer.analyzeAndStore(indexed, target) }
.onFailure { log.warn("Loudness analysis failed for {}", target, it) }
.getOrDefault(indexed)
}
?: indexed
finish(job, song)
} finally {
runCatching { working.toFile().deleteRecursively() }
}
}
private fun finish(job: DownloadJobRecord, song: SongRecord) {
when (job.kind) {
DownloadRepository.KIND_SONG -> {
MoodRepository.addSong(job.moodId, song.id, "search")
DownloadRepository.markDone(job.id, song.id)
runCatching { onSongAttached(job.moodId) }
.onFailure { log.warn("Mood {} top-up failed", job.moodId, it) }
}
else -> {
when (
MoodRepository.completeDownload(
job.moodId,
job.source,
job.externalId,
song.id,
)
) {
RecommendationCompletion.READY,
RecommendationCompletion.ALREADY_ADDED,
-> DownloadRepository.markDone(job.id, song.id)
RecommendationCompletion.REJECTED,
RecommendationCompletion.MISSING,
-> {
// The user rejected the track while it was downloading,
// or the mood was deleted. Drop the file if unused.
DownloadRepository.markFailed(job.id, "Recommendation is no longer active")
gc.collect(song.id)
}
}
}
}
}
/** Includes the exception type and cause, which plain NIO messages omit. */
private fun describe(t: Throwable): String =
generateSequence(t) { it.cause }
.map { cause ->
val text = cause.message?.takeIf { it.isNotBlank() }
if (text == null) cause::class.simpleName ?: "error"
else "${cause::class.simpleName}: $text"
}
.distinct()
.take(3)
.joinToString(" -> ")
.take(1000)
/** Retrying a permission, credential or quality error never helps. */
private fun isPermanent(t: Throwable): Boolean =
generateSequence(t) { it.cause }.any { cause ->
cause is AccessDeniedException ||
cause is ReadOnlyFileSystemException ||
cause is PermanentDownloadException ||
cause.message?.contains("Read-only", ignoreCase = true) == true ||
cause.message?.contains("Permission denied", ignoreCase = true) == true ||
cause.message?.contains("not configured", ignoreCase = true) == true ||
cause.message?.contains("subscription", ignoreCase = true) == true ||
cause.message?.contains("WrongLicense", ignoreCase = true) == true ||
cause.message?.contains("quality is not available", ignoreCase = true) == true
}
}
@@ -0,0 +1,27 @@
package win.taptap.musicmanager.download
import java.nio.file.Path
import win.taptap.musicmanager.source.MusicSource
/** Downloads a single external track id into [targetDir] and returns the file. */
fun interface AudioDownloader {
fun download(externalId: String, targetDir: Path): Path
}
/** Dispatches a download to the implementation for its source. */
class SourceDownloaders(private val downloaders: Map<String, AudioDownloader>) {
init {
MusicSource.ALL.forEach { source ->
require(source in downloaders) { "Missing downloader for $source" }
}
}
fun download(source: String, externalId: String, targetDir: Path): Path {
val downloader = downloaders[source]
?: throw IllegalStateException("Downloads from $source are not supported")
return downloader.download(externalId, targetDir)
}
fun isConfigured(source: String): Boolean = downloaders.containsKey(source)
}
@@ -0,0 +1,110 @@
package win.taptap.musicmanager.download
import java.nio.file.Files
import java.nio.file.Path
import java.util.concurrent.TimeUnit
import kotlin.concurrent.thread
/**
* Downloads from the YouTube Music player first, which is what a search hit
* points at: yt-dlp then exposes the song's artist, album and release year and
* `--embed-metadata` writes them into the MP3. Some videos only exist on the
* regular YouTube host, so that URL is tried once as a fallback.
*/
class YtDlpDownloader(
private val executable: String,
private val timeoutSeconds: Long = 600,
) : AudioDownloader {
override fun download(videoId: String, targetDir: Path): Path {
Files.createDirectories(targetDir)
return try {
runDownload("$MUSIC_WATCH_URL$videoId", videoId, targetDir)
} catch (e: YtDlpTimeoutException) {
throw e
} catch (e: IllegalStateException) {
clearPartials(videoId, targetDir)
runDownload("$YOUTUBE_WATCH_URL$videoId", videoId, targetDir)
}
}
private fun runDownload(url: String, videoId: String, targetDir: Path): Path {
val process = ProcessBuilder(
executable,
"-x",
"--audio-format",
"mp3",
"--audio-quality",
"0",
"--embed-metadata",
"--embed-thumbnail",
"--convert-thumbnails",
"jpg",
"--no-playlist",
"--ignore-config",
"--no-cache-dir",
"--no-progress",
"--print",
"after_move:filepath",
"-o",
targetDir.resolve("%(id)s.%(ext)s").toString(),
url,
)
.redirectErrorStream(false)
.start()
val output = StringBuilder()
val errors = StringBuilder()
val reader = thread(isDaemon = true, name = "yt-dlp-download-stdout") {
process.inputStream.bufferedReader().useLines { lines ->
lines.forEach { output.appendLine(it) }
}
}
val errorReader = thread(isDaemon = true, name = "yt-dlp-download-stderr") {
process.errorStream.bufferedReader().useLines { lines ->
lines.forEach { errors.appendLine(it) }
}
}
if (!process.waitFor(timeoutSeconds, TimeUnit.SECONDS)) {
process.destroyForcibly()
reader.join(2_000)
errorReader.join(2_000)
throw YtDlpTimeoutException("yt-dlp download timed out after ${timeoutSeconds}s")
}
reader.join(2_000)
errorReader.join(2_000)
if (process.exitValue() != 0) {
val detail = (errors.toString().ifBlank { output.toString() }).take(400).trim()
throw IllegalStateException("yt-dlp exited with ${process.exitValue()}: $detail")
}
val printed = output.lines()
.map { it.trim() }
.lastOrNull { it.endsWith(".mp3", ignoreCase = true) }
val file = printed?.let { Path.of(it) } ?: targetDir.resolve("$videoId.mp3")
if (!Files.isRegularFile(file)) {
throw IllegalStateException("yt-dlp did not produce an MP3 file")
}
return file
}
/** A failed first attempt may leave `.part`/`.webm` files behind. */
private fun clearPartials(videoId: String, targetDir: Path) {
runCatching {
Files.list(targetDir).use { stream ->
stream
.filter { it.fileName.toString().startsWith(videoId) }
.forEach { runCatching { Files.deleteIfExists(it) } }
}
}
}
private companion object {
const val MUSIC_WATCH_URL = "https://music.youtube.com/watch?v="
const val YOUTUBE_WATCH_URL = "https://www.youtube.com/watch?v="
}
}
/** A timeout is not retried against the other host: the network is the problem. */
class YtDlpTimeoutException(message: String) : IllegalStateException(message)
@@ -0,0 +1,118 @@
package win.taptap.musicmanager.media
import java.nio.file.Files
import java.nio.file.Path
import java.util.concurrent.TimeUnit
import org.slf4j.LoggerFactory
import win.taptap.musicmanager.db.SongRecord
import win.taptap.musicmanager.db.SongRepository
/**
* Measures a track's integrated loudness with ffmpeg's EBU R128 filter and
* stores the gain clients need to reach a common target.
*
* The analysis is one ffmpeg pass per file. It runs in the download worker for
* new tracks and in a low-priority backfill loop for the existing library, so a
* large collection is normalized gradually instead of blocking playback.
*/
class LoudnessAnalyzer(
private val ffmpegPath: String = "ffmpeg",
private val targetLufs: Double = -16.0,
private val truePeakDb: Double = -1.5,
) {
private val log = LoggerFactory.getLogger(LoudnessAnalyzer::class.java)
@Volatile private var available: Boolean? = null
data class Measurement(val lufs: Double, val truePeakDb: Double)
/** Returns null when the file cannot be analyzed; never throws. */
fun analyze(file: Path): Measurement? {
if (!Files.isRegularFile(file) || !isAvailable()) return null
return runCatching { measure(file) }
.onFailure { log.warn("Loudness analysis failed for {}: {}", file, it.message) }
.getOrNull()
}
/** Measures and persists the result, returning the updated row. */
fun analyzeAndStore(song: SongRecord, file: Path): SongRecord {
val measurement = analyze(file) ?: return song
val gain = gainFor(measurement)
SongRepository.updateLoudness(song.id, measurement.lufs, gain)
return song.copy(loudnessLufs = measurement.lufs, gainDb = gain)
}
fun gainFor(measurement: Measurement): Double {
if (measurement.lufs <= SILENCE_LUFS) return 0.0
val target = targetLufs - measurement.lufs
// Never let a boost push the measured true peak above the ceiling.
val peakLimited = truePeakDb - measurement.truePeakDb
return minOf(target, peakLimited).coerceIn(MIN_GAIN_DB, MAX_GAIN_DB)
.let { kotlin.math.round(it * 10.0) / 10.0 }
}
fun isAvailable(): Boolean {
val cached = available
if (cached != null) return cached
val result = runCatching {
val process = ProcessBuilder(ffmpegPath, "-version")
.redirectErrorStream(true)
.start()
process.inputStream.readBytes()
process.waitFor(VERSION_TIMEOUT_SECONDS, TimeUnit.SECONDS) && process.exitValue() == 0
}.getOrDefault(false)
available = result
if (!result) {
log.warn(
"ffmpeg is not available at '{}': loudness normalization is disabled",
ffmpegPath,
)
}
return result
}
private fun measure(file: Path): Measurement? {
val process = ProcessBuilder(
ffmpegPath,
"-hide_banner",
"-nostdin",
"-i",
file.toString(),
"-af",
"loudnorm=I=$targetLufs:TP=$truePeakDb:LRA=11:print_format=json",
"-f",
"null",
"-",
)
.redirectErrorStream(true)
.start()
val output = process.inputStream.bufferedReader().use { it.readText() }
if (!process.waitFor(ANALYZE_TIMEOUT_SECONDS, TimeUnit.SECONDS)) {
process.destroyForcibly()
return null
}
if (process.exitValue() != 0) return null
return parse(output)
}
companion object {
private const val SILENCE_LUFS = -70.0
private const val MIN_GAIN_DB = -30.0
private const val MAX_GAIN_DB = 12.0
private const val VERSION_TIMEOUT_SECONDS = 5L
private const val ANALYZE_TIMEOUT_SECONDS = 180L
private val INTEGRATED = Regex("\"input_i\"\\s*:\\s*\"([^\"]+)\"")
private val TRUE_PEAK = Regex("\"input_tp\"\\s*:\\s*\"([^\"]+)\"")
/** Parses the JSON block loudnorm appends to ffmpeg's stderr. */
fun parse(output: String): Measurement? {
val lufsText = INTEGRATED.find(output)?.groupValues?.get(1) ?: return null
val peakText = TRUE_PEAK.find(output)?.groupValues?.get(1) ?: return null
val lufs = lufsText.toDoubleOrNull() ?: SILENCE_LUFS
val peak = peakText.toDoubleOrNull() ?: 0.0
return Measurement(lufs = lufs, truePeakDb = peak)
}
}
}
@@ -0,0 +1,119 @@
package win.taptap.musicmanager.media
import java.nio.file.Files
import java.nio.file.attribute.FileTime
import java.time.Instant
import java.time.temporal.ChronoUnit
import org.jetbrains.exposed.sql.SqlExpressionBuilder.eq
import org.jetbrains.exposed.sql.SqlExpressionBuilder.inList
import org.jetbrains.exposed.sql.and
import org.jetbrains.exposed.sql.selectAll
import org.jetbrains.exposed.sql.transactions.transaction
import org.slf4j.LoggerFactory
import win.taptap.musicmanager.db.MoodRecommendationsTable
import win.taptap.musicmanager.db.MoodSongsTable
import win.taptap.musicmanager.db.PlaybackSessionsTable
import win.taptap.musicmanager.db.SessionItemsTable
import win.taptap.musicmanager.db.SongRecord
import win.taptap.musicmanager.db.SongRepository
import win.taptap.musicmanager.scanner.CoverArtService
/**
* Reference-counted file cleanup. A song's file is only deleted when no mood
* references it, no recommendation slot still points at it and no open playback
* session has it queued.
*/
class MediaGcService(
private val mediaStore: MediaStore,
private val coverArtService: CoverArtService,
) {
private val log = LoggerFactory.getLogger(MediaGcService::class.java)
/** Deletes every unreferenced song row, file and cover. Returns the count. */
fun collectGarbage(): Int {
var deleted = 0
SongRepository.all().forEach { song ->
if (collect(song)) deleted++
}
collectOrphanDirectories()
return deleted
}
/** Deletes [song] if it has no references. */
fun collect(song: SongRecord): Boolean {
if (isReferenced(song.id)) return false
if (!SongRepository.delete(song.id)) return false
coverArtService.deleteCover(song.id)
mediaStore.deleteSongDirectory(song.relativePath)
log.info("GC removed unreferenced song {} ({}/{})", song.id, song.source, song.externalId)
return true
}
fun collect(songId: String): Boolean {
val song = SongRepository.byId(songId) ?: return false
return collect(song)
}
fun isReferenced(songId: String): Boolean = transaction {
val inMood = MoodSongsTable.selectAll()
.where { MoodSongsTable.songId eq songId }
.any()
val inRecommendation = MoodRecommendationsTable.selectAll()
.where {
(MoodRecommendationsTable.songId eq songId) and
(MoodRecommendationsTable.state inList listOf("downloading", "ready", "added", "failed"))
}
.any()
val sessionIds = SessionItemsTable.select(SessionItemsTable.sessionId)
.where { SessionItemsTable.songId eq songId }
.map { it[SessionItemsTable.sessionId] }
val inOpenSession = if (sessionIds.isEmpty()) {
false
} else {
PlaybackSessionsTable.selectAll()
.where {
(PlaybackSessionsTable.id inList sessionIds) and
PlaybackSessionsTable.closedAt.isNull()
}
.any()
}
inMood || inRecommendation || inOpenSession
}
/**
* A crash between moving a file and indexing its row leaves a directory that
* no song points at. Only remove directories that have been idle for a while
* so a download that is still in progress is never touched.
*/
private fun collectOrphanDirectories() {
val cutoff = Instant.now().minus(ORPHAN_GRACE_MINUTES, ChronoUnit.MINUTES)
// Use the actual first path segment: new downloads are
// `<source>-<id>` while older YouTube files are `<videoId>`.
val known = SongRepository.all()
.mapNotNull { song ->
song.relativePath.replace('\\', '/').split('/').firstOrNull()?.takeIf { it.isNotBlank() }
}
.toSet()
mediaStore.songDirectories().forEach { directory ->
val name = directory.fileName?.toString() ?: return@forEach
if (name in known) return@forEach
val modified = runCatching {
Files.getLastModifiedTime(directory)
}.getOrNull() ?: return@forEach
if (modified.toInstant().isAfter(cutoff)) return@forEach
log.info("GC removed orphan media directory {}", directory)
runCatching { directory.toFile().deleteRecursively() }
}
}
private companion object {
const val ORPHAN_GRACE_MINUTES = 30L
}
}
@@ -0,0 +1,109 @@
package win.taptap.musicmanager.media
import java.nio.file.Files
import java.nio.file.Path
import win.taptap.musicmanager.db.SongRecord
import win.taptap.musicmanager.source.MusicSource
/**
* Layout of the server-owned media root:
*
* <media>/<source>-<externalId>/<sanitized-title>.mp3
*
* Older YouTube downloads live in `<media>/<videoId>/...`; those directories
* are still read and are assumed to belong to YouTube.
*/
class MediaStore(val root: Path) {
init {
Files.createDirectories(root)
}
fun file(song: SongRecord): Path? = resolveRelative(song.relativePath)
fun resolveRelative(relativePath: String): Path? {
val candidate = root.resolve(relativePath).normalize()
return candidate.takeIf { it.startsWith(root) }
}
fun directoryName(source: String, externalId: String): String = "$source-$externalId"
/** Target path for a download. The title is sanitised for the filesystem. */
fun target(source: String, externalId: String, title: String): Path =
root.resolve(directoryName(source, externalId))
.resolve("${sanitizeName(title)}.mp3")
fun relativePath(source: String, externalId: String, title: String): String =
"${directoryName(source, externalId)}/${sanitizeName(title)}.mp3"
/**
* Reads a directory name back into a source and external id. A directory
* without a known prefix is a legacy YouTube download.
*/
fun sourceAndIdFrom(directoryName: String): Pair<String, String>? {
val name = directoryName.trim()
if (name.isEmpty() || name == "." || name == "..") return null
val separator = name.indexOf('-')
if (separator > 0) {
val prefix = name.substring(0, separator)
val id = name.substring(separator + 1)
when {
prefix == MusicSource.DEEZER && id.isNotEmpty() && id.all(Char::isDigit) ->
return MusicSource.DEEZER to id
prefix == MusicSource.YOUTUBE && id.length == YOUTUBE_ID_LENGTH ->
return MusicSource.YOUTUBE to id
}
}
return MusicSource.YOUTUBE to name
}
fun sourceAndIdFromPath(relativePath: String): Pair<String, String>? {
val first = relativePath.replace('\\', '/').split('/').firstOrNull() ?: return null
return sourceAndIdFrom(first)
}
/** Deletes the directory that actually contains [relativePath]. */
fun deleteSongDirectory(relativePath: String) {
val file = resolveRelative(relativePath) ?: return
val directory = file.parent ?: return
if (directory == root || !directory.startsWith(root)) return
runCatching { directory.toFile().deleteRecursively() }
}
fun songDirectories(): List<Path> {
if (!Files.isDirectory(root)) return emptyList()
return Files.list(root).use { stream ->
stream.filter { Files.isDirectory(it) }.toList()
}
}
companion object {
private const val MAX_NAME_BYTES = 120
private const val YOUTUBE_ID_LENGTH = 11
private val INVALID = Regex("[\\\\/:*?\"<>|\\p{Cntrl}]")
private val SPACES = Regex("\\s+")
fun sanitizeName(value: String): String =
truncateUtf8(
value.replace(INVALID, "_")
.replace(SPACES, " ")
.trim()
.trim('.')
.ifEmpty { "track" },
MAX_NAME_BYTES,
)
/** Linux NAME_MAX is 255 bytes, not characters; CJK titles are 3x. */
fun truncateUtf8(value: String, maxBytes: Int): String {
var bytes = 0
val result = StringBuilder()
for (character in value) {
val length = character.toString().toByteArray(Charsets.UTF_8).size
if (bytes + length > maxBytes) break
result.append(character)
bytes += length
}
return result.toString().ifEmpty { "track" }
}
}
}
@@ -1,13 +1,12 @@
package win.taptap.musicmanager.radio
import java.time.Instant
import java.time.temporal.ChronoUnit
import java.util.UUID
import kotlin.math.max
import kotlin.random.Random
import org.slf4j.LoggerFactory
import win.taptap.musicmanager.api.musicFolderIds
import win.taptap.musicmanager.config.LibraryConfig
import win.taptap.musicmanager.config.UserAccount
import win.taptap.musicmanager.db.LibraryRepository
import win.taptap.musicmanager.db.MoodRecord
import win.taptap.musicmanager.db.MoodRepository
import win.taptap.musicmanager.db.PlaybackSessionRecord
@@ -22,25 +21,21 @@ data class StartedSession(
/**
* Server-driven mood radio.
*
* A batch is a weighted sample of the mood's world: seeds are guaranteed
* candidates, related genres/artists and free-text matches shape the pool, and
* per-user completions/skips nudge the ranking. Tracks already queued in the
* session are never repeated. The client asks for more whenever its buffer is
* low, so the queue is always generated with the latest signals.
* A batch mixes the mood's permanent collection with its ready recommendation
* songs. Every slot rolls `collectionPercent` independently, so a 70% mood gets
* roughly seven collection tracks in ten. Per-user completions/skips nudge the
* ranking. Tracks already queued in the session are never repeated until the
* pool is exhausted; then the least-recently-heard tracks replay.
*/
class RadioService {
class RadioService(
private val random: Random = Random.Default,
) {
private val log = LoggerFactory.getLogger(RadioService::class.java)
private val random = Random.Default
fun start(
mood: MoodRecord,
user: UserAccount,
library: LibraryConfig,
limit: Int,
): StartedSession {
fun start(mood: MoodRecord, user: UserAccount, limit: Int): StartedSession {
val session = RadioRepository.createSession(user.name, mood.id, UUID.randomUUID().toString())
val songs = generate(mood, user, library, session.id, emptySet(), limit, persist = true)
val songs = generate(mood, user, session.id, emptySet(), limit, persist = true)
log.info("Radio session {} started for mood {} with {} tracks", session.id, mood.name, songs.size)
return StartedSession(session, songs)
}
@@ -48,34 +43,31 @@ class RadioService {
fun next(
mood: MoodRecord,
user: UserAccount,
library: LibraryConfig,
sessionId: String,
excluded: Set<String>,
limit: Int,
): List<SongRecord> =
generate(mood, user, library, sessionId, excluded, limit, persist = true)
): List<SongRecord> = generate(mood, user, sessionId, excluded, limit, persist = true)
/** Queue preview for the mood screen; nothing is persisted. */
fun preview(
mood: MoodRecord,
user: UserAccount,
library: LibraryConfig,
limit: Int,
): List<SongRecord> =
generate(mood, user, library, sessionId = null, excluded = emptySet(), limit, persist = false)
fun preview(mood: MoodRecord, user: UserAccount, limit: Int): List<SongRecord> =
generate(mood, user, sessionId = null, excluded = emptySet(), limit, persist = false)
private fun generate(
mood: MoodRecord,
user: UserAccount,
library: LibraryConfig,
sessionId: String?,
excluded: Set<String>,
requestedLimit: Int,
persist: Boolean,
): List<SongRecord> {
val limit = requestedLimit.coerceIn(1, 50)
val folders = user.musicFolderIds(library)
if (folders.isEmpty()) return emptyList()
val collection = MoodRepository.songRecords(mood.id)
// A mood can only play once it has at least one permanent song.
if (collection.isEmpty()) return emptyList()
val collectionIds = collection.map { it.id }.toSet()
val recommendations = MoodRepository.readySongs(mood.id)
.filterNot { it.id in collectionIds }
val sessionSongs = if (sessionId != null && persist) {
RadioRepository.queuedSongIds(sessionId)
@@ -83,76 +75,44 @@ class RadioService {
emptySet()
}
val skip = sessionSongs + excluded
val seeds = LibraryRepository.songsByIds(
MoodRepository.seedSongIds(mood.id),
folders,
val recent = RadioRepository.recentSongIds(
user.name,
Instant.now().minus(RECENT_DAYS, ChronoUnit.DAYS),
)
val affinity = RadioRepository.affinity(
user.name,
(collection.map { it.id } + recommendations.map { it.id }).toSet(),
)
val seedIds = seeds.map { it.id }.toSet()
val seedGenres = (seeds.flatMap { it.genres } + mood.genres)
.map { it.trim() }
.filter { it.isNotEmpty() }
.distinctBy { it.lowercase() }
val seedArtistIds = seeds.mapNotNull { it.artistId }.distinct()
val query = mood.query?.trim()?.takeIf { it.isNotEmpty() }
val candidates = LinkedHashMap<String, SongRecord>()
seeds.forEach { candidates[it.id] = it }
if (seedGenres.isNotEmpty()) {
LibraryRepository.songsByGenres(seedGenres, POOL_LIMIT, folders, user.name)
.forEach { candidates[it.id] = it }
}
if (seedArtistIds.isNotEmpty()) {
LibraryRepository.songsByArtists(seedArtistIds, POOL_LIMIT, folders, user.name)
.forEach { candidates[it.id] = it }
}
if (query != null) {
LibraryRepository.searchSongs(query, POOL_LIMIT, 0, folders, user.name)
.forEach { candidates[it.id] = it }
}
// A mood with no seeds and no filters still needs a starting point.
LibraryRepository.randomSongs(
size = POOL_LIMIT,
genre = null,
fromYear = null,
toYear = null,
folderIds = folders,
userName = user.name,
).forEach { candidates[it.id] = it }
val freshCollection = collection.filterNot { it.id in skip }
val freshRecommendations = recommendations.filterNot { it.id in skip }
candidates.keys.removeAll(skip)
val picks = pickMixed(
collectionScored = score(freshCollection, affinity, recent, heard = false),
recommendationScored = score(freshRecommendations, affinity, recent, heard = false),
limit = limit,
collectionPercent = mood.collectionPercent,
).toMutableList()
// Small libraries can exhaust the mood. Replay tracks the session already
// heard instead of returning silence, but never repeat the current batch.
if (candidates.size < limit) {
LibraryRepository.randomSongs(
size = max(limit * 10, 200),
genre = null,
fromYear = null,
toYear = null,
folderIds = folders,
userName = user.name,
).forEach { song ->
if (song.id !in excluded) {
candidates.putIfAbsent(song.id, song)
}
// Small moods exhaust their pool quickly. Replay the least-recently
// queued tracks instead of returning silence, but never duplicate
// within one batch.
if (picks.size < limit) {
val pickedIds = picks.map { it.id }.toSet()
val heardCollection = collection.filter {
it.id in sessionSongs && it.id !in excluded && it.id !in pickedIds
}
}
val affinity = RadioRepository.affinity(user.name, candidates.keys)
val scored = candidates.values.map { song ->
song to score(
song = song,
seedIds = seedIds,
seedGenres = seedGenres,
seedArtistIds = seedArtistIds,
query = query,
affinity = affinity[song.id]?.score ?: 0.0,
val heardRecommendations = recommendations.filter {
it.id in sessionSongs && it.id !in excluded && it.id !in pickedIds
}
picks += pickMixed(
collectionScored = score(heardCollection, affinity, recent, heard = true),
recommendationScored = score(heardRecommendations, affinity, recent, heard = true),
limit = limit - picks.size,
collectionPercent = mood.collectionPercent,
)
}
val picks = pickDiverse(scored, limit)
if (persist && sessionId != null && picks.isNotEmpty()) {
RadioRepository.append(sessionId, picks.map { it.id })
}
@@ -160,49 +120,92 @@ class RadioService {
}
private fun score(
songs: List<SongRecord>,
affinity: Map<String, win.taptap.musicmanager.db.AffinityRecord>,
recent: Set<String>,
heard: Boolean,
): List<Pair<SongRecord, Double>> = songs.map { song ->
song to scoreCandidate(
song = song,
affinity = affinity[song.id]?.score ?: 0.0,
recent = song.id in recent,
jitter = random.nextDouble(0.0, 2.0),
heard = heard,
)
}
internal fun scoreCandidate(
song: SongRecord,
seedIds: Set<String>,
seedGenres: List<String>,
seedArtistIds: List<String>,
query: String?,
affinity: Double,
recent: Boolean = false,
jitter: Double = 0.0,
heard: Boolean = false,
): Double {
var score = random.nextDouble(0.0, 2.0)
if (song.id in seedIds) score += 10.0
if (song.artistId != null && song.artistId in seedArtistIds) score += 3.0
score += song.genres.count { candidate ->
seedGenres.any { it.equals(candidate, ignoreCase = true) }
} * 1.5
if (query != null && song.title.contains(query, ignoreCase = true)) score += 2.0
score += affinity.coerceIn(-6.0, 6.0)
var score = jitter + affinity.coerceIn(-8.0, 8.0)
if (recent) score -= RECENT_PENALTY
if (heard) score -= HEARD_PENALTY
return score
}
/**
* Picks the best remaining candidate, preferring a different artist than the
* previous track so a batch does not become one album on repeat.
* Picks a batch with the mood's requested collection share. Each slot rolls
* the percentage independently. An exhausted pool falls back to the other
* one instead of stalling.
*/
private fun pickDiverse(
scored: List<Pair<SongRecord, Double>>,
internal fun pickMixed(
collectionScored: List<Pair<SongRecord, Double>>,
recommendationScored: List<Pair<SongRecord, Double>>,
limit: Int,
collectionPercent: Int,
artistSpacing: Int = 2,
albumSpacing: Int = 3,
): List<SongRecord> {
val remaining = scored
.sortedWith(compareByDescending<Pair<SongRecord, Double>> { it.second })
val collection = collectionScored
.sortedByDescending { it.second }
.toMutableList()
val recommendations = recommendationScored
.sortedByDescending { it.second }
.toMutableList()
val percent = collectionPercent.coerceIn(0, 100)
val picked = ArrayList<SongRecord>(limit)
var lastArtist: String? = null
val artists = ArrayDeque<String>()
val albums = ArrayDeque<String>()
while (picked.size < limit && remaining.isNotEmpty()) {
val different = remaining.indexOfFirst { it.first.displayArtist != lastArtist }
val index = if (different >= 0) different else 0
val song = remaining.removeAt(index).first
while (picked.size < limit && (collection.isNotEmpty() || recommendations.isNotEmpty())) {
val preferCollection = when {
collection.isEmpty() -> false
recommendations.isEmpty() -> true
else -> random.nextInt(100) < percent
}
val pool = if (preferCollection) collection else recommendations
val song = pool.removeAt(diverseIndex(pool, artists, albums)).first
picked += song
lastArtist = song.displayArtist
artists.addLast(song.artist)
if (artists.size > artistSpacing) artists.removeFirst()
albums.addLast(albumKey(song))
if (albums.size > albumSpacing) albums.removeFirst()
}
return picked
}
private fun diverseIndex(
pool: List<Pair<SongRecord, Double>>,
artists: ArrayDeque<String>,
albums: ArrayDeque<String>,
): Int {
var index = pool.indexOfFirst { (song, _) ->
song.artist !in artists && albumKey(song) !in albums
}
if (index < 0) index = pool.indexOfFirst { (song, _) -> song.artist !in artists }
return if (index < 0) 0 else index
}
private fun albumKey(song: SongRecord): String = song.album?.takeIf { it.isNotBlank() } ?: song.id
private companion object {
const val POOL_LIMIT = 400
const val RECENT_DAYS = 7L
const val RECENT_PENALTY = 6.0
const val HEARD_PENALTY = 20.0
}
}
@@ -11,11 +11,10 @@ import net.coobird.thumbnailator.Thumbnails
import org.slf4j.LoggerFactory
/**
* Extracts, normalises and caches album covers under `DATA_DIR/covers`.
*
* Covers are always stored as JPEG. Original-size files are named `{albumId}.jpg`,
* resized variants `{albumId}_{size}.jpg`. When a cover changes, its variants are
* invalidated.
* Extracts, normalises and caches cover art under `DATA_DIR/covers`, keyed by
* song id. Covers are always stored as JPEG. Original-size files are named
* `{songId}.jpg`, resized variants `{songId}_{size}.jpg`. When a cover changes,
* its variants are invalidated.
*/
class CoverArtService(dataDir: Path) {
@@ -26,23 +25,23 @@ class CoverArtService(dataDir: Path) {
Files.createDirectories(coversDir)
}
fun coverFile(albumId: String): Path? =
coversDir.resolve("$albumId.jpg").takeIf { Files.isRegularFile(it) }
fun coverFile(songId: String): Path? =
coversDir.resolve("$songId.jpg").takeIf { Files.isRegularFile(it) }
/**
* Stores [bytes] as the album cover. Returns true when a usable cover exists,
* Stores [bytes] as the song cover. Returns true when a usable cover exists,
* including when the bytes were unchanged.
*/
@Synchronized
fun saveCover(albumId: String, bytes: ByteArray): Boolean {
fun saveCover(songId: String, bytes: ByteArray): Boolean {
if (bytes.isEmpty() || bytes.size > MAX_COVER_BYTES) return false
val image = runCatching {
ImageIO.read(ByteArrayInputStream(bytes))
}.getOrNull() ?: return false
val target = coversDir.resolve("$albumId.jpg")
val temp = Files.createTempFile(coversDir, "$albumId-", ".jpg")
val target = coversDir.resolve("$songId.jpg")
val temp = Files.createTempFile(coversDir, "$songId-", ".jpg")
return try {
writeJpeg(image, temp.toFile())
@@ -52,10 +51,10 @@ class CoverArtService(dataDir: Path) {
}
moveReplacing(temp, target)
deleteVariants(albumId)
deleteVariants(songId)
true
} catch (e: Exception) {
log.warn("Could not store cover for album {}: {}", albumId, e.message)
log.warn("Could not store cover for song {}: {}", songId, e.message)
false
} finally {
Files.deleteIfExists(temp)
@@ -64,14 +63,14 @@ class CoverArtService(dataDir: Path) {
/** Returns a cached `size x size` variant, generating it on first request. */
@Synchronized
fun resizedCover(albumId: String, size: Int): Path? {
fun resizedCover(songId: String, size: Int): Path? {
require(size in 1..MAX_RESIZE) { "Cover size must be between 1 and $MAX_RESIZE" }
val base = coverFile(albumId) ?: return null
val target = coversDir.resolve("${albumId}_$size.jpg")
val base = coverFile(songId) ?: return null
val target = coversDir.resolve("${songId}_$size.jpg")
if (Files.isRegularFile(target)) return target
val temp = Files.createTempFile(coversDir, "${albumId}_$size-", ".jpg")
val temp = Files.createTempFile(coversDir, "${songId}_$size-", ".jpg")
return try {
Thumbnails.of(base.toFile())
.size(size, size)
@@ -82,13 +81,18 @@ class CoverArtService(dataDir: Path) {
moveReplacing(temp, target)
target
} catch (e: Exception) {
log.warn("Could not resize cover for album {}: {}", albumId, e.message)
log.warn("Could not resize cover for song {}: {}", songId, e.message)
null
} finally {
Files.deleteIfExists(temp)
}
}
fun deleteCover(songId: String) {
Files.deleteIfExists(coversDir.resolve("$songId.jpg"))
deleteVariants(songId)
}
fun findSidecarCover(audioFile: Path): Path? {
val directory = audioFile.parent ?: return null
return SIDECAR_NAMES
@@ -116,10 +120,10 @@ class CoverArtService(dataDir: Path) {
}
}
private fun deleteVariants(albumId: String) {
private fun deleteVariants(songId: String) {
Files.list(coversDir).use { stream ->
stream
.filter { it.name.startsWith("${albumId}_") && it.name.endsWith(".jpg") }
.filter { it.name.startsWith("${songId}_") && it.name.endsWith(".jpg") }
.forEach { runCatching { Files.deleteIfExists(it) } }
}
}
@@ -1,26 +1,26 @@
package win.taptap.musicmanager.scanner
import java.nio.file.Files
import java.nio.file.Path
import java.nio.file.attribute.BasicFileAttributes
import java.util.concurrent.atomic.AtomicBoolean
import java.util.concurrent.atomic.AtomicReference
import kotlinx.coroutines.CoroutineScope
import kotlinx.coroutines.launch
import org.slf4j.LoggerFactory
import win.taptap.musicmanager.config.LibraryConfig
import win.taptap.musicmanager.config.MusicFolder
import win.taptap.musicmanager.db.MediaRepository
import win.taptap.musicmanager.db.SongRepository
import win.taptap.musicmanager.media.MediaStore
/**
* Recursive MP3 scanner with delta updates and orphan cleanup.
* Rebuilds the song index from the media root. The first path segment is the
* YouTube video id, so a file's identity never depends on its tags.
*
* Only one scan can run at a time. The file's `(mtime, size)` pair decides whether
* Only one scan can run at a time. A file's `(mtime, size)` pair decides whether
* tags need to be re-read; unchanged files are skipped.
*/
class MediaScanner(
private val library: LibraryConfig,
private val coverArtService: CoverArtService,
private val tagExtractor: Mp3TagExtractor = Mp3TagExtractor(),
private val repository: MediaRepository = MediaRepository,
private val mediaStore: MediaStore,
private val indexer: SongIndexer,
private val onScanFinished: (ScanStatus) -> Unit = {},
) {
@@ -49,80 +49,83 @@ class MediaScanner(
var count = 0L
val seenSongIds = mutableSetOf<String>()
var error: String? = null
var rootFailed = false
try {
library.folders.forEach { folder ->
try {
scanFolder(folder, seenSongIds) {
count++
status.set(ScanStatus(scanning = true, count = count))
val root = mediaStore.root
if (!Files.isDirectory(root) || !Files.isReadable(root)) {
rootFailed = true
error = "Media root is not readable: $root"
log.error("Skipping scan: media root is not readable: {}", root)
} else {
Files.walk(root).use { paths ->
paths.filter { Files.isRegularFile(it) }.forEach { file ->
if (!file.fileName.toString().endsWith(".mp3", ignoreCase = true)) {
return@forEach
}
val relativePath = root.relativize(file).joinToString("/") { it.toString() }
val sourceAndId = mediaStore.sourceAndIdFromPath(relativePath)
if (sourceAndId == null) {
log.debug("Skipping file outside a media folder: {}", file)
return@forEach
}
val (source, externalId) = sourceAndId
try {
val existing = SongRepository.byExternalId(source, externalId)
val attributes = Files.readAttributes(
file,
BasicFileAttributes::class.java,
)
val mtime = attributes.lastModifiedTime().toMillis()
val size = attributes.size()
if (existing != null && existing.fileMtime == mtime && existing.size == size) {
seenSongIds += existing.id
count++
status.set(ScanStatus(scanning = true, count = count))
return@forEach
}
val song = indexer.index(source, externalId, file)
if (song == null) {
if (existing != null) seenSongIds += existing.id
log.warn("Could not read MP3 metadata: {}", file)
return@forEach
}
seenSongIds += song.id
count++
status.set(ScanStatus(scanning = true, count = count))
} catch (e: Exception) {
SongRepository.byExternalId(source, externalId)?.let { seenSongIds += it.id }
log.warn("Could not index file: $file", e)
}
}
} catch (e: Exception) {
log.error("Could not scan folder ${folder.id} (${folder.path})", e)
}
}
repository.deleteSongsNotIn(seenSongIds)
repository.rebuildAggregates()
if (rootFailed) {
// An unmounted or temporarily unreadable root must not be
// mistaken for deleted files. Keep the last known index.
log.warn("Skipping orphan cleanup because the media root failed to scan")
} else {
SongRepository.all().forEach { song ->
if (song.id !in seenSongIds) {
SongRepository.delete(song.id)
}
}
}
} catch (t: Throwable) {
error = t.message ?: t::class.simpleName
log.error("Library scan failed after $count files", t)
log.error("Media scan failed after $count files", t)
} finally {
running.set(false)
}
val finalStatus = ScanStatus(scanning = false, count = count, error = error)
status.set(finalStatus)
log.info("Library scan finished: {} files, error={}", count, error)
log.info("Media scan finished: {} files, error={}", count, error)
runCatching { onScanFinished(finalStatus) }
.onFailure { log.warn("Scan completion hook failed", it) }
return finalStatus
}
private fun scanFolder(folder: MusicFolder, seenSongIds: MutableSet<String>, onFile: () -> Unit) {
Files.walk(folder.path).use { paths ->
paths.filter { Files.isRegularFile(it) }.forEach { file ->
if (!file.fileName.toString().endsWith(".mp3", ignoreCase = true)) {
log.debug("Skipping unsupported file: {}", file)
return@forEach
}
onFile()
val relativePath = folder.path.relativize(file).joinToString("/") { it.toString() }
val existing = repository.findExisting(folder.id, relativePath)
try {
val attributes = Files.readAttributes(file, java.nio.file.attribute.BasicFileAttributes::class.java)
val mtime = attributes.lastModifiedTime().toMillis()
val size = attributes.size()
if (existing != null && existing.fileMtime == mtime && existing.size == size) {
seenSongIds += existing.id
return@forEach
}
val track = tagExtractor.extract(folder.id, folder.path, file)
if (track == null) {
if (existing != null) seenSongIds += existing.id
log.warn("Could not read MP3 metadata: {}", file)
return@forEach
}
val coverBytes = track.coverBytes
?: coverArtService.findSidecarCover(file)?.let(coverArtService::readCover)
repository.upsertTrack(track)
if (coverBytes != null && coverArtService.saveCover(track.albumId, coverBytes)) {
repository.setAlbumCover(track.albumId)
}
seenSongIds += track.songId
} catch (e: Exception) {
if (existing != null) seenSongIds += existing.id
log.warn("Could not index file: $file", e)
}
}
}
}
}
@@ -8,25 +8,13 @@ import java.util.logging.Logger
import org.jaudiotagger.audio.AudioFileIO
import org.jaudiotagger.tag.FieldKey
import org.jaudiotagger.tag.Tag
import win.taptap.musicmanager.util.IdGenerator
import win.taptap.musicmanager.discovery.cleanTrackTitle
import win.taptap.musicmanager.discovery.stripArtistPrefix
data class ArtistRef(val id: String, val name: String)
data class ExtractedTrack(
val songId: String,
val albumId: String,
val albumArtistId: String,
val albumArtists: List<ArtistRef>,
val artists: List<ArtistRef>,
data class ExtractedSong(
val title: String,
val displayArtist: String,
val albumTitle: String,
val folderId: Int,
val relativePath: String,
val trackNumber: Int?,
val discNumber: Int,
val year: Int?,
val genres: List<String>,
val artist: String,
val album: String?,
val durationSeconds: Int,
val bitRate: Int,
val size: Long,
@@ -34,15 +22,11 @@ data class ExtractedTrack(
val contentType: String,
val fileMtime: Long,
val coverBytes: ByteArray?,
val lyrics: String?,
) {
val searchText: String =
IdGenerator.normalize("$title $displayArtist $albumTitle ${albumArtists.joinToString(" ") { it.name }}")
}
)
/**
* Reads MP3 metadata with jaudiotagger and applies the documented fallbacks:
* album artist -> artist -> folder name, title -> file name.
* Reads the tags yt-dlp embeds into downloaded MP3s. Files are always MP3s
* inside the media store; anything else is ignored.
*/
class Mp3TagExtractor {
@@ -55,7 +39,7 @@ class Mp3TagExtractor {
).forEach { Logger.getLogger(it).level = Level.WARNING }
}
fun extract(folderId: Int, root: Path, file: Path): ExtractedTrack? {
fun extract(file: Path): ExtractedSong? {
if (!file.fileName.toString().endsWith(".mp3", ignoreCase = true)) return null
val attributes = Files.readAttributes(file, BasicFileAttributes::class.java)
@@ -67,54 +51,24 @@ class Mp3TagExtractor {
val tag = audioFile.tag
val header = audioFile.audioHeader
val relativePath = root.relativize(file).joinToString("/") { it.toString() }
val fallbackArtist = file.parent
?.takeIf { it != root }
?.parent
?.takeIf { it != root }
?.fileName
?.toString()
?.takeIf { it.isNotBlank() }
?: "Unknown Artist"
val fallbackTitle = file.fileName.toString().removeSuffix(".mp3").removeSuffix(".MP3")
val fallbackAlbum = file.parent
?.takeIf { it != root }
?.fileName
?.toString()
?.takeIf { it.isNotBlank() }
?: "Unknown Album"
val tagArtists = (tag.values(FieldKey.ARTIST) + tag.values(FieldKey.ARTISTS)).distinct()
val selectedArtists = tagArtists.ifEmpty { listOf(fallbackArtist) }
val albumArtistNames =
(tag.values(FieldKey.ALBUM_ARTIST) + tag.values(FieldKey.ALBUM_ARTISTS))
.ifEmpty { selectedArtists }
val albumArtist = albumArtistNames.first()
val title = tag.values(FieldKey.TITLE).firstOrNull()
?: file.fileName.toString().removeSuffix(".mp3").removeSuffix(".MP3")
val albumTitle = tag.values(FieldKey.ALBUM).firstOrNull() ?: fallbackAlbum
val year = (tag.values(FieldKey.YEAR) + tag.values(FieldKey.ALBUM_YEAR))
.firstNotNullOfOrNull { YEAR_PATTERN.find(it)?.value?.toIntOrNull() }
val trackNumber = tag.values(FieldKey.TRACK)
val rawTitle = tag.values(FieldKey.TITLE).firstOrNull() ?: fallbackTitle
val artists = (tag.values(FieldKey.ARTIST) + tag.values(FieldKey.ARTISTS)).distinct()
val albumTitle = tag.values(FieldKey.ALBUM).firstOrNull()
val albumArtist = (tag.values(FieldKey.ALBUM_ARTIST) + tag.values(FieldKey.ALBUM_ARTISTS))
.firstOrNull()
?.substringBefore('/')
?.trim()
?.toIntOrNull()
val discNumber = tag.values(FieldKey.DISC_NO)
.firstOrNull()
?.substringBefore('/')
?.trim()
?.toIntOrNull()
?: 1
val artist = when {
artists.isNotEmpty() -> artists.joinToString(", ")
albumArtist != null -> albumArtist
else -> "Unknown artist"
}.removeSuffix(" - Topic").trim()
val genres = tag.values(FieldKey.GENRE)
// The plain YouTube extractor embeds the raw upload title, which keeps
// the artist prefix and the "(Official Video)" noise in the store.
val title = stripArtistPrefix(cleanTrackTitle(rawTitle), artist)
val duration = header.trackLength.takeIf { it > 0 }
?: header.preciseTrackLength.toInt().coerceAtLeast(0)
@@ -126,26 +80,10 @@ class Mp3TagExtractor {
.getOrNull()
?.takeIf { it.isNotEmpty() }
val lyrics = tag.values(FieldKey.LYRICS).firstOrNull() ?: readSidecarLyrics(file)
val albumArtistRefs = albumArtistNames.map { ArtistRef(IdGenerator.artistId(it), it) }
val artistRefs = selectedArtists.map { ArtistRef(IdGenerator.artistId(it), it) }
return ExtractedTrack(
songId = IdGenerator.songId(folderId, relativePath),
albumId = IdGenerator.albumId(albumArtist, albumTitle),
albumArtistId = IdGenerator.artistId(albumArtist),
albumArtists = albumArtistRefs,
artists = artistRefs,
return ExtractedSong(
title = title,
displayArtist = selectedArtists.joinToString(", "),
albumTitle = albumTitle,
folderId = folderId,
relativePath = relativePath,
trackNumber = trackNumber,
discNumber = discNumber,
year = year,
genres = genres,
artist = artist,
album = albumTitle,
durationSeconds = duration,
bitRate = bitRate,
size = attributes.size(),
@@ -153,7 +91,6 @@ class Mp3TagExtractor {
contentType = "audio/mpeg",
fileMtime = attributes.lastModifiedTime().toMillis(),
coverBytes = coverBytes,
lyrics = lyrics,
)
}
@@ -164,15 +101,4 @@ class Mp3TagExtractor {
.map { it.trim() }
.filter { it.isNotEmpty() }
.distinct()
private fun readSidecarLyrics(file: Path): String? {
val lrc = file.resolveSibling(file.fileName.toString().substringBeforeLast('.') + ".lrc")
return runCatching {
if (Files.isRegularFile(lrc)) Files.readString(lrc) else null
}.getOrNull()
}
private companion object {
val YEAR_PATTERN = Regex("\\d{4}")
}
}
@@ -0,0 +1,59 @@
package win.taptap.musicmanager.scanner
import java.nio.file.Files
import java.nio.file.Path
import win.taptap.musicmanager.db.SongRecord
import win.taptap.musicmanager.db.SongRepository
import win.taptap.musicmanager.db.SongUpsert
import win.taptap.musicmanager.media.MediaStore
/**
* Single-file indexing shared by the scanner and the downloader. The
* downloader calls this immediately after moving a fresh MP3 into the media
* root, so it does not wait for a full scan.
*/
class SongIndexer(
private val mediaStore: MediaStore,
private val coverArtService: CoverArtService,
private val tagExtractor: Mp3TagExtractor = Mp3TagExtractor(),
) {
fun index(
source: String,
externalId: String,
file: Path,
thumbnail: String? = null,
): SongRecord? {
val normalized = file.normalize()
if (!normalized.startsWith(mediaStore.root)) return null
if (!Files.isRegularFile(normalized)) return null
val extracted = tagExtractor.extract(normalized) ?: return null
val relativePath = mediaStore.root.relativize(normalized).joinToString("/") { it.toString() }
val song = SongRepository.upsert(
SongUpsert(
source = source,
externalId = externalId,
title = extracted.title,
artist = extracted.artist,
album = extracted.album,
duration = extracted.durationSeconds,
bitRate = extracted.bitRate,
size = extracted.size,
suffix = extracted.suffix,
contentType = extracted.contentType,
relativePath = relativePath,
fileMtime = extracted.fileMtime,
thumbnail = thumbnail,
),
)
val coverBytes = extracted.coverBytes
?: coverArtService.findSidecarCover(normalized)?.let(coverArtService::readCover)
if (coverBytes != null) {
coverArtService.saveCover(song.id, coverBytes)
}
return song
}
}
@@ -0,0 +1,18 @@
package win.taptap.musicmanager.source
/**
* The two supported download sources. `externalId` is namespaced by the source:
* a YouTube video id or a Deezer track id.
*/
object MusicSource {
const val YOUTUBE = "youtube"
const val DEEZER = "deezer"
val ALL = setOf(YOUTUBE, DEEZER)
/** Returns the normalized source, or null when it is not supported. */
fun normalizeOrNull(value: String?): String? =
value?.trim()?.lowercase()?.takeIf { it in ALL }
fun isKnown(value: String?): Boolean = normalizeOrNull(value) != null
}
@@ -19,22 +19,23 @@ object IdGenerator {
.replace(WHITESPACE, " ")
.lowercase(Locale.ROOT)
fun artistId(albumArtist: String): String = "ar-" + hash(normalize(albumArtist))
/** Stable across restarts and independent of the downloaded file's title. */
fun songId(source: String, externalId: String): String =
"sg-" + hash("${sourcePrefix(source)}:${normalize(externalId)}")
fun albumId(albumArtist: String, albumTitle: String): String =
"al-" + hash("${normalize(albumArtist)}|${normalize(albumTitle)}")
fun songId(folderId: Int, relativePath: String): String =
"sg-" + hash("$folderId:${normalizePath(relativePath)}")
/**
* YouTube ids were created before sources existed and must not change, so
* `youtube` keeps the historical `yt:` prefix.
*/
private fun sourcePrefix(source: String): String = when (source) {
"youtube" -> "yt"
"deezer" -> "dz"
else -> source.lowercase(Locale.ROOT)
}
fun moodId(owner: String, name: String, createdAt: String): String =
"mo-" + hash("${normalize(owner)}|${normalize(name)}|$createdAt")
private fun normalizePath(path: String): String = path
.replace('\\', '/')
.removePrefix("./")
.trimStart('/')
private fun hash(value: String): String {
val digest = MessageDigest.getInstance("SHA-256").digest(value.toByteArray(Charsets.UTF_8))
val hex = digest.joinToString("") { "%02x".format(it) }
@@ -0,0 +1,8 @@
-- Per-track loudness, measured once per file with ffmpeg's EBU R128 filter.
--
-- `loudness_lufs` is the integrated loudness; `gain_db` is the attenuation or
-- boost the clients apply so tracks from Deezer and YouTube sound equally loud.
-- Both stay NULL for files that have not been analyzed yet; the backfill worker
-- picks those up without blocking downloads.
ALTER TABLE songs ADD COLUMN loudness_lufs REAL;
ALTER TABLE songs ADD COLUMN gain_db REAL;
@@ -0,0 +1,24 @@
-- Client diagnostics. The apps keep a small local ring buffer of warnings and
-- errors (plus an occasional info entry) and upload it in batches so a bug can
-- be investigated with the server-side request id and the client context.
--
-- Rows are per user and pruned by retention days and a per-user row cap. The
-- client timestamp is informational; ordering uses the server-assigned id so a
-- device with a wrong clock cannot reorder the log.
CREATE TABLE client_logs (
id INTEGER PRIMARY KEY AUTOINCREMENT,
user_name TEXT NOT NULL REFERENCES users(name) ON DELETE CASCADE,
device_id TEXT NOT NULL,
platform TEXT NOT NULL,
app_version TEXT NOT NULL,
session_id TEXT,
level TEXT NOT NULL CHECK (level IN ('debug', 'info', 'warn', 'error')),
tag TEXT NOT NULL,
message TEXT NOT NULL,
detail TEXT,
created_at TEXT NOT NULL,
received_at TEXT NOT NULL
);
CREATE INDEX idx_client_logs_user_id ON client_logs (user_name, id);
CREATE INDEX idx_client_logs_received ON client_logs (received_at);
@@ -0,0 +1,33 @@
-- Accepted YT recommendations that are not in the library are downloaded into
-- the first visible music folder and indexed by a scan.
CREATE TABLE download_jobs (
id TEXT PRIMARY KEY,
user_name TEXT NOT NULL REFERENCES users(name) ON DELETE CASCADE,
mood_id TEXT NOT NULL REFERENCES moods(id) ON DELETE CASCADE,
video_id TEXT NOT NULL,
title TEXT NOT NULL,
artist TEXT NOT NULL,
state TEXT NOT NULL DEFAULT 'queued'
CHECK (state IN ('queued', 'downloading', 'done', 'failed')),
attempts INTEGER NOT NULL DEFAULT 0,
error TEXT,
song_id TEXT REFERENCES songs(id) ON DELETE SET NULL,
created_at TEXT NOT NULL,
updated_at TEXT NOT NULL,
UNIQUE (user_name, mood_id, video_id)
);
CREATE INDEX idx_download_jobs_state ON download_jobs (state, created_at);
-- Long-form resume points, separate from mood playback.
CREATE TABLE longform_progress (
user_name TEXT NOT NULL REFERENCES users(name) ON DELETE CASCADE,
song_id TEXT NOT NULL REFERENCES songs(id) ON DELETE CASCADE,
position_seconds INTEGER NOT NULL DEFAULT 0,
duration_seconds INTEGER NOT NULL DEFAULT 0,
completed BOOLEAN NOT NULL DEFAULT 0,
updated_at TEXT NOT NULL,
PRIMARY KEY (user_name, song_id)
);
CREATE INDEX idx_longform_progress_updated ON longform_progress (user_name, updated_at);
@@ -0,0 +1,3 @@
-- Share of each radio batch that should come from the mood's seed songs.
-- The rest is filled from the generated recommendation pool.
ALTER TABLE moods ADD COLUMN seed_percent INTEGER NOT NULL DEFAULT 50;
@@ -0,0 +1,162 @@
-- Mood-first redesign: destructive migration.
--
-- The mounted-library model is gone. Songs enter only through downloads, so
-- every content table from the old design is dropped. Users and device tokens
-- are preserved; the media root starts empty and is rebuilt by the downloader.
--
-- Drop order matters because SQLite enforces foreign keys on DROP TABLE.
DROP TABLE IF EXISTS playback_session_items;
DROP TABLE IF EXISTS playback_sessions;
DROP TABLE IF EXISTS song_affinity;
DROP TABLE IF EXISTS download_jobs;
DROP TABLE IF EXISTS mood_recommendations;
DROP TABLE IF EXISTS mood_seeds;
DROP TABLE IF EXISTS moods;
DROP TABLE IF EXISTS song_genres;
DROP TABLE IF EXISTS song_artists;
DROP TABLE IF EXISTS album_genres;
DROP TABLE IF EXISTS artist_genres;
DROP TABLE IF EXISTS album_artists;
DROP TABLE IF EXISTS user_song_state;
DROP TABLE IF EXISTS user_stars;
DROP TABLE IF EXISTS bookmarks;
DROP TABLE IF EXISTS library_tombstones;
DROP TABLE IF EXISTS longform_progress;
DROP TABLE IF EXISTS songs;
DROP TABLE IF EXISTS albums;
DROP TABLE IF EXISTS artists;
DROP TABLE IF EXISTS music_folders;
-- One row per YouTube video. The file lives at <media>/<video_id>/<title>.mp3,
-- so the video id is both the dedupe key and the on-disk folder name.
CREATE TABLE songs (
id TEXT PRIMARY KEY,
video_id TEXT NOT NULL UNIQUE,
title TEXT NOT NULL,
artist TEXT NOT NULL,
album TEXT,
duration INTEGER NOT NULL DEFAULT 0,
bit_rate INTEGER NOT NULL DEFAULT 0,
size INTEGER NOT NULL DEFAULT 0,
suffix TEXT NOT NULL DEFAULT 'mp3',
content_type TEXT NOT NULL DEFAULT 'audio/mpeg',
relative_path TEXT NOT NULL,
file_mtime INTEGER NOT NULL DEFAULT 0,
thumbnail TEXT,
created_at TEXT NOT NULL,
updated_at TEXT NOT NULL
);
CREATE INDEX idx_songs_updated ON songs (updated_at);
CREATE TABLE moods (
id TEXT PRIMARY KEY,
owner TEXT NOT NULL REFERENCES users(name) ON DELETE CASCADE,
name TEXT NOT NULL,
description TEXT,
query TEXT,
genres TEXT NOT NULL DEFAULT '[]',
collection_percent INTEGER NOT NULL DEFAULT 50
CHECK (collection_percent BETWEEN 0 AND 100),
recommendation_count INTEGER NOT NULL DEFAULT 10
CHECK (recommendation_count BETWEEN 0 AND 30),
refresh_hours INTEGER NOT NULL DEFAULT 24,
last_discovery_at TEXT,
created_at TEXT NOT NULL,
changed_at TEXT NOT NULL,
last_error TEXT
);
CREATE INDEX idx_moods_owner ON moods (owner);
CREATE TABLE mood_songs (
mood_id TEXT NOT NULL REFERENCES moods(id) ON DELETE CASCADE,
song_id TEXT NOT NULL REFERENCES songs(id) ON DELETE CASCADE,
origin TEXT NOT NULL DEFAULT 'search'
CHECK (origin IN ('search', 'recommendation', 'url')),
added_at TEXT NOT NULL,
PRIMARY KEY (mood_id, song_id)
);
CREATE INDEX idx_mood_songs_song ON mood_songs (song_id);
-- new hidden candidate pool used to fill active slots
-- downloading an active slot with a queued/running download job
-- ready an active slot whose MP3 is indexed and playable
-- failed an active slot whose download exhausted its retries
-- added the user promoted it to mood_songs
-- rejected the user does not want it in this mood again
CREATE TABLE mood_recommendations (
mood_id TEXT NOT NULL REFERENCES moods(id) ON DELETE CASCADE,
video_id TEXT NOT NULL,
title TEXT NOT NULL,
artist TEXT NOT NULL,
album TEXT,
duration INTEGER,
thumbnail TEXT,
state TEXT NOT NULL DEFAULT 'new'
CHECK (state IN ('new', 'downloading', 'ready', 'failed', 'added', 'rejected')),
song_id TEXT REFERENCES songs(id) ON DELETE SET NULL,
attempts INTEGER NOT NULL DEFAULT 0,
error TEXT,
rank INTEGER NOT NULL DEFAULT 0,
first_seen TEXT NOT NULL,
last_seen TEXT NOT NULL,
PRIMARY KEY (mood_id, video_id)
);
CREATE INDEX idx_mood_recommendations_state ON mood_recommendations (mood_id, state);
CREATE INDEX idx_mood_recommendations_song ON mood_recommendations (song_id);
CREATE TABLE download_jobs (
id TEXT PRIMARY KEY,
video_id TEXT NOT NULL,
mood_id TEXT NOT NULL REFERENCES moods(id) ON DELETE CASCADE,
kind TEXT NOT NULL CHECK (kind IN ('song', 'recommendation')),
state TEXT NOT NULL DEFAULT 'queued'
CHECK (state IN ('queued', 'downloading', 'done', 'failed')),
priority INTEGER NOT NULL DEFAULT 0,
attempts INTEGER NOT NULL DEFAULT 0,
error TEXT,
song_id TEXT REFERENCES songs(id) ON DELETE SET NULL,
created_at TEXT NOT NULL,
updated_at TEXT NOT NULL,
UNIQUE (mood_id, video_id, kind)
);
CREATE INDEX idx_download_jobs_state ON download_jobs (state, priority, created_at);
CREATE TABLE playback_sessions (
id TEXT PRIMARY KEY,
user_name TEXT NOT NULL REFERENCES users(name) ON DELETE CASCADE,
mood_id TEXT NOT NULL REFERENCES moods(id) ON DELETE CASCADE,
next_position INTEGER NOT NULL DEFAULT 0,
created_at TEXT NOT NULL,
last_seen_at TEXT NOT NULL,
closed_at TEXT
);
CREATE INDEX idx_playback_sessions_user ON playback_sessions (user_name, last_seen_at);
CREATE TABLE playback_session_items (
session_id TEXT NOT NULL REFERENCES playback_sessions(id) ON DELETE CASCADE,
position INTEGER NOT NULL,
song_id TEXT NOT NULL REFERENCES songs(id) ON DELETE CASCADE,
state TEXT NOT NULL DEFAULT 'queued'
CHECK (state IN ('queued', 'started', 'completed', 'skipped')),
queued_at TEXT NOT NULL,
updated_at TEXT NOT NULL,
PRIMARY KEY (session_id, position)
);
CREATE INDEX idx_session_items_song ON playback_session_items (song_id);
CREATE TABLE song_affinity (
user_name TEXT NOT NULL REFERENCES users(name) ON DELETE CASCADE,
song_id TEXT NOT NULL REFERENCES songs(id) ON DELETE CASCADE,
completions INTEGER NOT NULL DEFAULT 0,
skips INTEGER NOT NULL DEFAULT 0,
last_event_at TEXT NOT NULL,
PRIMARY KEY (user_name, song_id)
);
@@ -0,0 +1,22 @@
-- Two download sources: Deezer (default for new moods) and YouTube Music.
--
-- Existing rows predate the source column and are YouTube Music. The old
-- `video_id` column becomes the source-neutral `external_id`; a Deezer track id
-- and a YouTube video id do not overlap, so the existing UNIQUE constraint
-- remains valid. The source is stored next to it for display and dispatch.
ALTER TABLE songs RENAME COLUMN video_id TO external_id;
ALTER TABLE songs ADD COLUMN source TEXT NOT NULL DEFAULT 'youtube'
CHECK (source IN ('youtube', 'deezer'));
CREATE INDEX idx_songs_source_external ON songs (source, external_id);
ALTER TABLE moods ADD COLUMN source TEXT NOT NULL DEFAULT 'youtube'
CHECK (source IN ('youtube', 'deezer'));
ALTER TABLE mood_recommendations RENAME COLUMN video_id TO external_id;
ALTER TABLE mood_recommendations ADD COLUMN source TEXT NOT NULL DEFAULT 'youtube'
CHECK (source IN ('youtube', 'deezer'));
ALTER TABLE download_jobs RENAME COLUMN video_id TO external_id;
ALTER TABLE download_jobs ADD COLUMN source TEXT NOT NULL DEFAULT 'youtube'
CHECK (source IN ('youtube', 'deezer'));
+1 -1
View File
@@ -1,7 +1,7 @@
<configuration>
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>%d{yyyy-MM-dd'T'HH:mm:ss.SSSXXX} %-5level [%thread] %logger{36} - %msg%n</pattern>
<pattern>%d{yyyy-MM-dd'T'HH:mm:ss.SSSXXX} %-5level [%thread] %logger{36} %X{requestId:-} - %msg%n</pattern>
</encoder>
</appender>
+102
View File
@@ -0,0 +1,102 @@
# Streamrip configuration used by DeezerDownloader.
#
# `%%ARL%%`, `%%QUALITY%%` and `%%FOLDER%%` are replaced per download. The file
# is written into the job's temporary directory and deleted afterwards, so the
# ARL never touches the persistent media store.
[downloads]
folder = "%%FOLDER%%"
source_subdirectories = false
disc_subdirectories = false
concurrency = false
max_connections = 1
requests_per_minute = 60
verify_ssl = true
[deezer]
# 0: 128 kbps MP3, 1: 320 kbps MP3. FLAC (2) is not used: the store is MP3-only.
quality = %%QUALITY%%
arl = "%%ARL%%"
use_deezloader = false
deezloader_warnings = false
[qobuz]
quality = 1
download_booklets = false
use_auth_token = false
email_or_userid = ""
password_or_token = ""
app_id = ""
secrets = []
[tidal]
quality = 1
download_videos = false
user_id = ""
country_code = ""
access_token = ""
refresh_token = ""
token_expiry = ""
[soundcloud]
quality = 0
client_id = ""
app_version = ""
[youtube]
quality = 0
download_videos = false
video_downloads_folder = ""
[database]
downloads_enabled = false
downloads_path = ""
failed_downloads_enabled = false
failed_downloads_path = ""
[conversion]
enabled = false
codec = "MP3"
sampling_rate = 48000
bit_depth = 16
lossy_bitrate = 320
[qobuz_filters]
extras = false
repeats = false
non_albums = false
features = false
non_studio_albums = false
non_remaster = false
[artwork]
embed = true
embed_size = "large"
embed_max_width = -1
save_artwork = false
saved_max_width = -1
[metadata]
set_playlist_to_album = false
renumber_playlist_tracks = false
exclude = []
[filepaths]
add_singles_to_folder = false
folder_format = "{albumartist} - {title} ({year})"
track_format = "{title}"
restrict_characters = false
truncate_to = 120
[lastfm]
source = "deezer"
fallback_source = ""
[cli]
text_output = false
progress_bars = false
max_search_results = 10
[misc]
version = "2.0.6"
check_for_updates = false
File diff suppressed because it is too large. Load diff
@@ -13,9 +13,16 @@ class AppConfigTest {
assertEquals(8080, config.port)
assertEquals(Path.of("./data"), config.dataDir)
assertEquals(Path.of("./config/library.yaml"), config.libraryConfigPath)
assertEquals(Path.of("./config/users.yaml"), config.usersFile)
assertEquals("ffmpeg", config.ffmpegPath)
assertEquals("yt-dlp", config.ytDlpPath)
assertEquals("rip", config.streamripPath)
assertEquals("deezer", config.defaultSource)
assertEquals(null, config.deezerArl)
assertEquals(0, config.deezerQuality)
assertEquals("https://api.deezer.com", config.deezerApiBase)
assertEquals(Path.of("./data/media"), config.mediaDir)
assertEquals(Path.of("./data/downloads"), config.downloadDir)
assertEquals(Path.of("./data/musicmanager.db"), config.databasePath)
}
@@ -25,17 +32,41 @@ class AppConfigTest {
mapOf(
"PORT" to "9090",
"DATA_DIR" to "/var/lib/musicmanager",
"LIBRARY_CONFIG" to "/etc/musicmanager/library.yaml",
"USERS_FILE" to "/run/secrets/users.yaml",
"FFMPEG_PATH" to "/usr/bin/ffmpeg",
"YTDLP_PATH" to "/usr/local/bin/yt-dlp",
"STREAMRIP_PATH" to "/usr/local/bin/rip",
"DEFAULT_SOURCE" to "youtube",
"DEEZER_ARL" to "test-arl",
"DEEZER_QUALITY" to "1",
"DEEZER_API_BASE" to "http://deezer.test",
"MEDIA_DIR" to "/mnt/media",
"DOWNLOAD_DIR" to "/var/tmp/downloads",
),
)
assertEquals(9090, config.port)
assertEquals(Path.of("/var/lib/musicmanager"), config.dataDir)
assertEquals(Path.of("/etc/musicmanager/library.yaml"), config.libraryConfigPath)
assertEquals(Path.of("/run/secrets/users.yaml"), config.usersFile)
assertEquals("/usr/bin/ffmpeg", config.ffmpegPath)
assertEquals("/usr/local/bin/yt-dlp", config.ytDlpPath)
assertEquals("/usr/local/bin/rip", config.streamripPath)
assertEquals("youtube", config.defaultSource)
assertEquals("test-arl", config.deezerArl)
assertEquals(1, config.deezerQuality)
assertEquals("http://deezer.test", config.deezerApiBase)
assertEquals(Path.of("/mnt/media"), config.mediaDir)
assertEquals(Path.of("/var/tmp/downloads"), config.downloadDir)
}
@Test
fun `rejects unknown sources and qualities`() {
assertFailsWith<IllegalArgumentException> {
AppConfig.fromEnv(mapOf("DEFAULT_SOURCE" to "spotify"))
}
assertFailsWith<IllegalArgumentException> {
AppConfig.fromEnv(mapOf("DEEZER_QUALITY" to "2"))
}
}
@Test
@@ -1,127 +0,0 @@
package win.taptap.musicmanager.config
import java.nio.file.Files
import kotlin.io.path.createTempDirectory
import kotlin.test.Test
import kotlin.test.assertEquals
import kotlin.test.assertFailsWith
import kotlin.test.assertTrue
class LibraryConfigTest {
private val validYaml = """
library:
folders:
- id: 1
name: "Music"
type: music
path: "./media/music"
- id: 10
name: "Long-form"
type: longform
path: "./media/longform"
""".trimIndent()
@Test
fun `parses folders and resolves absolute paths`() {
val config = LibraryConfig.parse(validYaml)
assertEquals(2, config.folders.size)
assertEquals(setOf(10), config.longformFolderIds)
assertEquals(FolderType.MUSIC, config.folders[0].type)
assertEquals(FolderType.LONGFORM, config.folders[1].type)
assertTrue(config.folders[0].path.isAbsolute)
assertEquals(1, config.byId(1)?.id)
}
@Test
fun `rejects duplicate folder ids`() {
val yaml = """
library:
folders:
- id: 1
name: A
type: music
path: /music/a
- id: 1
name: B
type: music
path: /music/b
""".trimIndent()
assertFailsWith<IllegalArgumentException> { LibraryConfig.parse(yaml) }
}
@Test
fun `rejects duplicate folder paths`() {
val yaml = """
library:
folders:
- id: 1
name: A
type: music
path: /music/a
- id: 2
name: B
type: music
path: /music/a
""".trimIndent()
assertFailsWith<IllegalArgumentException> { LibraryConfig.parse(yaml) }
}
@Test
fun `rejects unknown folder types`() {
val yaml = """
library:
folders:
- id: 1
name: A
type: video
path: /music/a
""".trimIndent()
assertFailsWith<IllegalArgumentException> { LibraryConfig.parse(yaml) }
}
@Test
fun `validatePaths fails for a missing root`() {
val root = createTempDirectory("musicmanager-test")
try {
val yaml = """
library:
folders:
- id: 1
name: A
type: music
path: "${root.resolve("missing")}"
""".trimIndent()
assertFailsWith<IllegalArgumentException> {
LibraryConfig.parse(yaml).validatePaths()
}
} finally {
root.toFile().deleteRecursively()
}
}
@Test
fun `validatePaths accepts existing readable roots`() {
val root = createTempDirectory("musicmanager-test")
try {
val music = Files.createDirectories(root.resolve("music"))
val yaml = """
library:
folders:
- id: 1
name: A
type: music
path: "$music"
""".trimIndent()
LibraryConfig.parse(yaml).validatePaths()
} finally {
root.toFile().deleteRecursively()
}
}
}
@@ -8,48 +8,26 @@ import kotlin.test.assertTrue
class UsersConfigTest {
private val library = LibraryConfig.parse(
"""
library:
folders:
- id: 1
name: "Music A"
type: music
path: /music/a
- id: 2
name: "Music B"
type: music
path: /music/b
- id: 10
name: "Long-form"
type: longform
path: /longform
""".trimIndent(),
)
private val usersYaml = """
users:
- name: alice
password: "secret-a"
admin: true
folders: [1]
- name: bob
password: "secret-b"
folders: [1, 2]
""".trimIndent()
@Test
fun `parses users and shares long-form folders`() {
val users = UsersConfig.parse(usersYaml)
users.validateAgainst(library)
fun `parses users and ignores old folder assignments`() {
val users = UsersConfig.parse(
"""
users:
- name: alice
password: "secret-a"
admin: true
folders: [1]
- name: bob
password: "secret-b"
""".trimIndent(),
)
val alice = users.byName("alice")!!
val bob = users.byName("bob")!!
assertTrue(alice.admin)
assertFalse(bob.admin)
assertEquals(setOf(1, 10), alice.visibleFolderIds(library))
assertEquals(setOf(1, 2, 10), bob.visibleFolderIds(library))
assertEquals(2, users.users.size)
}
@Test
@@ -57,7 +35,6 @@ class UsersConfigTest {
val yaml = """
users:
- name: alice
folders: [1]
""".trimIndent()
assertFailsWith<IllegalArgumentException> { UsersConfig.parse(yaml) }
@@ -69,56 +46,27 @@ class UsersConfigTest {
users:
- name: alice
password: a
folders: [1]
- name: alice
password: b
folders: [2]
""".trimIndent()
assertFailsWith<IllegalArgumentException> { UsersConfig.parse(yaml) }
}
@Test
fun `rejects unknown folder references`() {
val users = UsersConfig.parse(
"""
users:
- name: alice
password: a
folders: [99]
""".trimIndent(),
)
assertFailsWith<IllegalArgumentException> { users.validateAgainst(library) }
}
@Test
fun `rejects a user with no visible folders`() {
val emptyLibrary = LibraryConfig.parse(
"""
library:
folders:
- id: 1
name: "Music"
type: music
path: /music
""".trimIndent(),
)
val users = UsersConfig.parse(
"""
users:
- name: alice
password: a
folders: []
""".trimIndent(),
)
assertFailsWith<IllegalArgumentException> { users.validateAgainst(emptyLibrary) }
fun `rejects an empty user list`() {
assertFailsWith<IllegalArgumentException> { UsersConfig.parse("users: []") }
}
@Test
fun `does not expose passwords in toString`() {
val alice = UsersConfig.parse(usersYaml).byName("alice")!!
val alice = UsersConfig.parse(
"""
users:
- name: alice
password: "secret-a"
""".trimIndent(),
).byName("alice")!!
assertFalse(alice.toString().contains("secret-a"))
assertTrue(alice.toString().contains("password=***"))
@@ -0,0 +1,110 @@
package win.taptap.musicmanager.db
import java.nio.file.Files
import java.time.Instant
import java.time.temporal.ChronoUnit
import kotlin.test.Test
import kotlin.test.assertEquals
import kotlin.test.assertFalse
import kotlin.test.assertTrue
import org.jetbrains.exposed.sql.SqlExpressionBuilder.eq
import org.jetbrains.exposed.sql.transactions.transaction
import org.jetbrains.exposed.sql.update
import win.taptap.musicmanager.api.ClientLogBatchDto
import win.taptap.musicmanager.api.ClientLogEntryDto
import win.taptap.musicmanager.config.AppConfig
import win.taptap.musicmanager.config.UserAccount
import win.taptap.musicmanager.config.UsersConfig
class ClientLogRepositoryTest {
@Test
fun `credentials are redacted before they are stored`() {
withDatabase { db ->
assertEquals(1, ClientLogRepository.insert("alice", batch("playback", "Authorization: Bearer supersecret")))
val row = ClientLogRepository.list("alice").single()
assertFalse(row.message.contains("supersecret"))
assertFalse(row.detail.orEmpty().contains("supersecret"))
assertTrue(row.detail.orEmpty().contains("***"))
}
}
@Test
fun `retention removes old rows and the per-user cap keeps the newest`() {
withDatabase { db ->
ClientLogRepository.insert("alice", batch("one", "first"))
ClientLogRepository.insert("alice", batch("two", "second"))
ClientLogRepository.insert("alice", batch("three", "third"))
// Three rows, keep two: the newest two survive.
assertEquals(1, ClientLogRepository.prune(365, 2))
assertEquals(listOf("third", "second"), ClientLogRepository.list("alice").map { it.message })
// Age the remaining rows past a one-day retention window.
transaction {
ClientLogsTable.update({ ClientLogsTable.userName eq "alice" }) {
it[receivedAt] = Instant.now().minus(10, ChronoUnit.DAYS).toString()
}
}
assertEquals(2, ClientLogRepository.prune(7, 2))
assertTrue(ClientLogRepository.list("alice").isEmpty())
}
}
@Test
fun `users cannot see each other's diagnostics`() {
withDatabase { db ->
ClientLogRepository.insert("alice", batch("a", "alice line"))
ClientLogRepository.insert("bob", batch("b", "bob line"))
assertEquals(listOf("alice line"), ClientLogRepository.list("alice").map { it.message })
assertEquals(listOf("bob line"), ClientLogRepository.list("bob").map { it.message })
assertEquals(1, ClientLogRepository.delete("alice"))
assertTrue(ClientLogRepository.list("alice").isEmpty())
assertEquals(1, ClientLogRepository.list("bob").size)
}
}
private fun withDatabase(block: (DatabaseHandle) -> Unit) {
val dir = Files.createTempDirectory("client-logs-test")
val users = UsersConfig(
listOf(
UserAccount("alice", "alice-pw", admin = true),
UserAccount("bob", "bob-pw", admin = false),
),
)
val db = DatabaseFactory.connect(
AppConfig(
port = 0,
dataDir = dir,
usersFile = dir.resolve("users.yaml"),
ffmpegPath = "ffmpeg",
),
)
try {
ConfigRepository.sync(users)
block(db)
} finally {
db.close()
}
}
private fun batch(tag: String, message: String): ClientLogBatchDto =
ClientLogBatchDto(
deviceId = "test-device",
platform = "desktop",
appVersion = "0.4.3",
entries = listOf(
ClientLogEntryDto(
timestamp = System.currentTimeMillis(),
level = "warn",
tag = tag,
message = message,
detail = "request failed: Authorization: Bearer supersecret",
sessionId = "run-test",
),
),
)
}
@@ -2,18 +2,19 @@ package win.taptap.musicmanager.db
import java.nio.file.Files
import java.nio.file.Path
import java.sql.DriverManager
import java.time.Instant
import kotlin.test.Test
import kotlin.test.assertEquals
import kotlin.test.assertFalse
import kotlin.test.assertTrue
import org.flywaydb.core.Flyway
import org.flywaydb.core.api.MigrationVersion
import org.jetbrains.exposed.sql.insert
import org.jetbrains.exposed.sql.selectAll
import org.jetbrains.exposed.sql.transactions.transaction
import org.junit.jupiter.api.io.TempDir
import win.taptap.musicmanager.config.AppConfig
import win.taptap.musicmanager.config.FolderType
import win.taptap.musicmanager.config.LibraryConfig
import win.taptap.musicmanager.config.MusicFolder
import win.taptap.musicmanager.config.UserAccount
import win.taptap.musicmanager.config.UsersConfig
@@ -29,82 +30,119 @@ class DatabaseFactoryTest {
assertEquals("wal", pragma(db, "journal_mode"))
assertEquals("1", pragma(db, "foreign_keys"))
val migrations = db.dataSource.connection.use { connection ->
connection.createStatement().use { statement ->
statement.executeQuery("SELECT COUNT(*) FROM flyway_schema_history").use { result ->
result.next()
result.getInt(1)
}
}
}
assertTrue(migrations >= 2, "Expected at least the V1 and V2 migrations")
val migrations = queryInt(db, "SELECT COUNT(*) FROM flyway_schema_history")
assertTrue(migrations >= 9, "Expected at least the V1..V9 migrations")
// The new schema exists and the mounted-library tables do not.
assertTrue(tableExists(db, "songs"))
assertTrue(tableExists(db, "mood_songs"))
assertTrue(tableExists(db, "mood_recommendations"))
assertTrue(tableExists(db, "download_jobs"))
assertFalse(tableExists(db, "music_folders"))
assertFalse(tableExists(db, "artists"))
assertFalse(tableExists(db, "longform_progress"))
} finally {
db.close()
}
}
@Test
fun `syncs folders and users idempotently`() {
val db = DatabaseFactory.connect(appConfig())
try {
val library = library()
val users = UsersConfig(
listOf(
UserAccount("alice", "pw", admin = true, folders = setOf(1)),
UserAccount("bob", "pw", admin = false, folders = setOf(1)),
),
)
fun `V8 wipes content but keeps users and device tokens`() {
val config = appConfig()
Files.createDirectories(config.dataDir)
val url = "jdbc:sqlite:${config.databasePath.toAbsolutePath()}"
ConfigRepository.sync(library, users)
ConfigRepository.sync(library, users)
Flyway.configure()
.dataSource(url, null, null)
.locations("classpath:db/migration")
.target(MigrationVersion.fromVersion("7"))
.load()
.migrate()
transaction(db.database) {
assertEquals(2L, MusicFoldersTable.selectAll().count())
assertEquals(2L, UsersTable.selectAll().count())
assertEquals(
"longform",
MusicFoldersTable.selectAll()
.where { MusicFoldersTable.id eq 10 }
.single()[MusicFoldersTable.type],
DriverManager.getConnection(url).use { connection ->
connection.createStatement().use { statement ->
statement.executeUpdate(
"INSERT INTO users (name, is_admin, updated_at) VALUES ('alice', 1, 'now')",
)
statement.executeUpdate(
"INSERT INTO api_tokens " +
"(id, user_name, token_hash, device_name, platform, created_at, last_used_at, expires_at) " +
"VALUES ('tk-1', 'alice', 'hash', 'phone', 'android', 'now', 'now', 'later')",
)
statement.executeUpdate(
"INSERT INTO music_folders (id, name, type, root_path) VALUES (1, 'Music', 'music', '/m')",
)
statement.executeUpdate(
"INSERT INTO artists (id, name, sort_name, search_text) VALUES ('ar-1', 'Artist', 'artist', 'artist')",
)
statement.executeUpdate(
"INSERT INTO albums (id, name, created_at, search_text) VALUES ('al-1', 'Album', 'now', 'album')",
)
statement.executeUpdate(
"INSERT INTO songs " +
"(id, album_id, folder_id, title, display_artist, album_name, duration, bit_rate, " +
"size, suffix, content_type, relative_path, file_mtime, created_at, updated_at, search_text) " +
"VALUES ('sg-1', 'al-1', 1, 'Song', 'Artist', 'Album', 120, 128, 10, 'mp3', " +
"'audio/mpeg', 'a/b.mp3', 0, 'now', 'now', 'song')",
)
statement.executeUpdate(
"INSERT INTO moods " +
"(id, owner, name, genres, target_size, refresh_hours, yt_enabled, created_at, changed_at, seed_percent) " +
"VALUES ('mo-1', 'alice', 'Focus', '[]', 30, 24, 1, 'now', 'now', 50)",
)
statement.executeUpdate(
"INSERT INTO mood_seeds (mood_id, song_id, origin, added_at) VALUES ('mo-1', 'sg-1', 'manual', 'now')",
)
statement.executeUpdate(
"INSERT INTO playback_sessions (id, user_name, mood_id, created_at, last_seen_at) " +
"VALUES ('ps-1', 'alice', 'mo-1', 'now', 'now')",
)
statement.executeUpdate(
"INSERT INTO playback_session_items " +
"(session_id, position, song_id, state, queued_at, updated_at) " +
"VALUES ('ps-1', 0, 'sg-1', 'queued', 'now', 'now')",
)
}
}
val db = DatabaseFactory.connect(config)
try {
assertEquals(
1,
queryInt(db, "SELECT COUNT(*) FROM users WHERE name = 'alice'"),
"Users must survive the wipe",
)
assertEquals(
1,
queryInt(db, "SELECT COUNT(*) FROM api_tokens WHERE id = 'tk-1'"),
"Device tokens must survive the wipe",
)
assertEquals(0, queryInt(db, "SELECT COUNT(*) FROM songs"))
assertEquals(0, queryInt(db, "SELECT COUNT(*) FROM moods"))
assertEquals(0, queryInt(db, "SELECT COUNT(*) FROM playback_sessions"))
assertTrue(tableExists(db, "mood_songs"))
assertFalse(tableExists(db, "music_folders"))
} finally {
db.close()
}
}
@Test
fun `removing a user cascades their per-user state`() {
fun `syncs users idempotently`() {
val db = DatabaseFactory.connect(appConfig())
try {
val library = library()
val aliceAndBob = UsersConfig(
val users = UsersConfig(
listOf(
UserAccount("alice", "pw", admin = true, folders = setOf(1)),
UserAccount("bob", "pw", admin = false, folders = setOf(1)),
UserAccount("alice", "pw", admin = true),
UserAccount("bob", "pw", admin = false),
),
)
ConfigRepository.sync(library, aliceAndBob)
insertSong(db, folderId = 1)
ConfigRepository.sync(users)
ConfigRepository.sync(users)
transaction(db.database) {
UserSongStateTable.insert {
it[userName] = "alice"
it[songId] = "sg-1"
it[playCount] = 3
it[starredAt] = Instant.now().toString()
}
assertEquals(1L, UserSongStateTable.selectAll().count())
}
ConfigRepository.sync(
library,
UsersConfig(listOf(UserAccount("bob", "pw", admin = false, folders = setOf(1)))),
)
transaction(db.database) {
assertEquals(0L, UserSongStateTable.selectAll().count())
assertEquals(1L, UsersTable.selectAll().count())
assertEquals(2L, UsersTable.selectAll().count())
}
} finally {
db.close()
@@ -112,26 +150,69 @@ class DatabaseFactoryTest {
}
@Test
fun `removing a folder cascades its songs`() {
fun `removing a user cascades their moods and playback state`() {
val db = DatabaseFactory.connect(appConfig())
try {
val library = library()
val users = UsersConfig(listOf(UserAccount("alice", "pw", admin = true, folders = setOf(1))))
ConfigRepository.sync(library, users)
insertSong(db, folderId = 1)
ConfigRepository.sync(
UsersConfig(
listOf(
UserAccount("alice", "pw", admin = true),
UserAccount("bob", "pw", admin = false),
),
),
)
val now = Instant.now().toString()
val moodId = "mo-1"
transaction(db.database) {
assertEquals(1L, SongsTable.selectAll().count())
SongsTable.insert {
it[id] = "sg-1"
it[musicSource] = "youtube"
it[externalId] = "video-1"
it[title] = "Song"
it[artist] = "Artist"
it[relativePath] = "video-1/song.mp3"
it[createdAt] = now
it[updatedAt] = now
}
MoodsTable.insert {
it[id] = moodId
it[owner] = "alice"
it[name] = "Focus"
it[createdAt] = now
it[changedAt] = now
}
MoodSongsTable.insert {
it[this.moodId] = moodId
it[songId] = "sg-1"
it[addedAt] = now
}
PlaybackSessionsTable.insert {
it[id] = "ps-1"
it[userName] = "alice"
it[this.moodId] = moodId
it[createdAt] = now
it[lastSeenAt] = now
}
SessionItemsTable.insert {
it[sessionId] = "ps-1"
it[position] = 0
it[songId] = "sg-1"
it[queuedAt] = now
it[updatedAt] = now
}
}
ConfigRepository.sync(
LibraryConfig(library.folders.filter { it.id == 10 }),
UsersConfig(listOf(UserAccount("alice", "pw", admin = true, folders = emptySet()))),
UsersConfig(listOf(UserAccount("bob", "pw", admin = false))),
)
transaction(db.database) {
assertEquals(0L, SongsTable.selectAll().count())
assertEquals(1L, MusicFoldersTable.selectAll().count())
assertEquals(1L, UsersTable.selectAll().count())
assertEquals(0L, MoodsTable.selectAll().count())
assertEquals(0L, MoodSongsTable.selectAll().count())
assertEquals(0L, PlaybackSessionsTable.selectAll().count())
// Songs are shared and have no owner, so they survive.
assertEquals(1L, SongsTable.selectAll().count())
}
} finally {
db.close()
@@ -141,57 +222,10 @@ class DatabaseFactoryTest {
private fun appConfig(): AppConfig = AppConfig(
port = 8080,
dataDir = tempDir.resolve("data"),
libraryConfigPath = tempDir.resolve("library.yaml"),
usersFile = tempDir.resolve("users.yaml"),
ffmpegPath = "ffmpeg",
)
private fun library(): LibraryConfig {
val music = Files.createDirectories(tempDir.resolve("music"))
val longform = Files.createDirectories(tempDir.resolve("longform"))
return LibraryConfig(
listOf(
MusicFolder(1, "Music", FolderType.MUSIC, music),
MusicFolder(10, "Long-form", FolderType.LONGFORM, longform),
),
)
}
private fun insertSong(db: DatabaseHandle, folderId: Int) {
val now = Instant.now().toString()
transaction(db.database) {
ArtistsTable.insert {
it[id] = "ar-1"
it[name] = "Artist"
it[sortName] = "artist"
it[searchText] = "artist"
}
AlbumsTable.insert {
it[id] = "al-1"
it[name] = "Album"
it[createdAt] = now
it[searchText] = "album"
}
SongsTable.insert {
it[id] = "sg-1"
it[albumId] = "al-1"
it[this.folderId] = folderId
it[title] = "Song"
it[displayArtist] = "Artist"
it[albumName] = "Album"
it[duration] = 120
it[bitRate] = 128
it[size] = 1024
it[suffix] = "mp3"
it[contentType] = "audio/mpeg"
it[relativePath] = "artist/album/song.mp3"
it[fileMtime] = 0
it[createdAt] = now
it[searchText] = "song"
}
}
}
private fun pragma(db: DatabaseHandle, name: String): String =
db.dataSource.connection.use { connection ->
connection.createStatement().use { statement ->
@@ -201,4 +235,20 @@ class DatabaseFactoryTest {
}
}
}
private fun queryInt(db: DatabaseHandle, sql: String): Int =
db.dataSource.connection.use { connection ->
connection.createStatement().use { statement ->
statement.executeQuery(sql).use { result ->
assertTrue(result.next(), "$sql returned no row")
result.getInt(1)
}
}
}
private fun tableExists(db: DatabaseHandle, name: String): Boolean =
queryInt(
db,
"SELECT COUNT(*) FROM sqlite_master WHERE type = 'table' AND name = '$name'",
) > 0
}
@@ -0,0 +1,439 @@
package win.taptap.musicmanager.db
import java.nio.file.Files
import java.nio.file.Path
import java.time.Instant
import kotlin.test.Test
import kotlin.test.assertEquals
import kotlin.test.assertFalse
import kotlin.test.assertNotNull
import kotlin.test.assertNull
import kotlin.test.assertTrue
import org.junit.jupiter.api.io.TempDir
import win.taptap.musicmanager.config.AppConfig
import win.taptap.musicmanager.config.UserAccount
import win.taptap.musicmanager.config.UsersConfig
import win.taptap.musicmanager.media.MediaGcService
import win.taptap.musicmanager.media.MediaStore
import win.taptap.musicmanager.scanner.CoverArtService
import win.taptap.musicmanager.source.MusicSource
class MoodRepositoryTest {
@TempDir
lateinit var tempDir: Path
@Test
fun `recommendation slots are promoted completed added and refilled`() {
withDatabase { mediaStore, _ ->
val mood = createMood(recommendationCount = 3)
storeCandidates(mood.id, 5)
val promoted = MoodRepository.promoteCandidates(mood.id, MusicSource.YOUTUBE, 3)
assertEquals(3, promoted.size)
assertEquals(3, MoodRepository.activeCount(mood.id))
assertEquals(
2,
MoodRepository.allRecommendations(mood.id).count { it.state == "new" },
)
val first = promoted.first()
val song = insertSong(
mediaStore,
MusicSource.YOUTUBE,
"vid-${first.externalId}",
"Track ${first.externalId}",
)
assertEquals(
RecommendationCompletion.READY,
MoodRepository.completeDownload(
mood.id,
MusicSource.YOUTUBE,
first.externalId,
song.id,
),
)
assertEquals(
"ready",
MoodRepository.recommendation(mood.id, first.externalId)?.state,
)
// Add turns the slot into a permanent mood song and frees it.
assertEquals(
RecommendationCompletion.ALREADY_ADDED,
MoodRepository.markAdded(mood.id, first.externalId),
)
assertTrue(MoodRepository.songIds(mood.id).contains(song.id))
assertEquals(
"added",
MoodRepository.recommendation(mood.id, first.externalId)?.state,
)
assertEquals(2, MoodRepository.activeCount(mood.id))
val refill = MoodRepository.promoteCandidates(mood.id, MusicSource.YOUTUBE, 1)
assertEquals(1, refill.size)
assertEquals(3, MoodRepository.activeCount(mood.id))
// Reject is permanent and frees the slot again.
MoodRepository.markRejected(mood.id, refill.first().externalId)
assertEquals(2, MoodRepository.activeCount(mood.id))
assertEquals(
"rejected",
MoodRepository.recommendation(mood.id, refill.first().externalId)?.state,
)
assertTrue(
MoodRepository.decidedExternalIds(mood.id, MusicSource.YOUTUBE)
.contains(refill.first().externalId),
)
assertEquals(
1,
MoodRepository.allRecommendations(mood.id).count { it.state == "new" },
)
}
}
@Test
fun `adding a song converts matching recommendations and drops hidden candidates`() {
withDatabase { mediaStore, _ ->
val mood = createMood(recommendationCount = 2)
val one = insertSong(mediaStore, MusicSource.YOUTUBE, "v-1", "One")
val two = insertSong(mediaStore, MusicSource.YOUTUBE, "v-2", "Two")
storeCandidates(mood.id, 2)
MoodRepository.promoteCandidates(mood.id, MusicSource.YOUTUBE, 2)
assertTrue(MoodRepository.addSong(mood.id, one.id, "search"))
assertTrue(MoodRepository.addSong(mood.id, two.id, "search"))
val first = assertNotNull(MoodRepository.recommendation(mood.id, "v-1"))
val second = assertNotNull(MoodRepository.recommendation(mood.id, "v-2"))
assertEquals("added", first.state)
assertEquals(one.id, first.songId)
assertEquals("added", second.state)
assertEquals(two.id, second.songId)
assertEquals(2, MoodRepository.songIds(mood.id).size)
assertEquals(0, MoodRepository.activeCount(mood.id))
// A still-hidden candidate for a video added manually is removed.
val another = createMood(recommendationCount = 2, name = "Another")
val three = insertSong(mediaStore, MusicSource.YOUTUBE, "v-3", "Three")
storeCandidates(another.id, 1, startAt = 3)
assertNotNull(MoodRepository.recommendation(another.id, "v-3"))
MoodRepository.addSong(another.id, three.id, "search")
assertNull(MoodRepository.recommendation(another.id, "v-3"))
}
}
@Test
fun `decisions survive a discovery refresh`() {
withDatabase { _, _ ->
val mood = createMood(recommendationCount = 3)
storeCandidates(mood.id, 3)
val first = MoodRepository
.promoteCandidates(mood.id, MusicSource.YOUTUBE, 1)
.single()
MoodRepository.markRejected(mood.id, first.externalId)
// The same video must not come back as a hidden candidate.
storeCandidates(mood.id, 3)
assertEquals(
"rejected",
MoodRepository.recommendation(mood.id, first.externalId)?.state,
)
}
}
@Test
fun `switching source clears active recommendations but keeps songs`() {
withDatabase { mediaStore, _ ->
val mood = createMood(recommendationCount = 2)
val song = insertSong(mediaStore, MusicSource.YOUTUBE, "vid-1", "One")
MoodRepository.addSong(mood.id, song.id, "search")
storeCandidates(mood.id, 2)
MoodRepository.promoteCandidates(mood.id, MusicSource.YOUTUBE, 2)
assertEquals(2, MoodRepository.activeCount(mood.id))
val updated = assertNotNull(
MoodRepository.update(
id = mood.id,
owner = "alice",
name = mood.name,
description = mood.description,
query = mood.query,
genres = mood.genres,
source = MusicSource.DEEZER,
collectionPercent = mood.collectionPercent,
recommendationCount = mood.recommendationCount,
refreshHours = mood.refreshHours,
),
)
assertEquals(MusicSource.DEEZER, updated.source)
assertEquals(0, MoodRepository.activeCount(mood.id))
assertTrue(MoodRepository.songIds(mood.id).contains(song.id))
assertNull(MoodRepository.recommendation(mood.id, "v-1"))
storeCandidates(mood.id, 2, source = MusicSource.DEEZER)
val promoted = MoodRepository
.promoteCandidates(mood.id, MusicSource.DEEZER, 2)
assertEquals(2, promoted.size)
assertEquals(MusicSource.DEEZER, promoted.first().source)
}
}
@Test
fun `adding a song makes discovery due again`() {
withDatabase { mediaStore, _ ->
val mood = createMood(recommendationCount = 2)
assertTrue(MoodRepository.isDiscoveryDue(mood), "A never-seen mood is due")
MoodRepository.markDiscovered(mood.id)
val fresh = assertNotNull(MoodRepository.get(mood.id, "alice"))
assertFalse(MoodRepository.isDiscoveryDue(fresh), "Nothing changed since the pass")
// Timestamps are compared strictly; keep them distinguishable.
Thread.sleep(5)
val song = insertSong(mediaStore, MusicSource.YOUTUBE, "vid-1", "One")
MoodRepository.addSong(mood.id, song.id, "search")
val changed = assertNotNull(MoodRepository.get(mood.id, "alice"))
assertTrue(
MoodRepository.isDiscoveryDue(changed),
"The collection is the discovery seed, so the pool is stale",
)
}
}
@Test
fun `download completion after a rejection does not attach the song`() {
withDatabase { mediaStore, _ ->
val mood = createMood(recommendationCount = 2)
storeCandidates(mood.id, 2)
val recommendation = MoodRepository
.promoteCandidates(mood.id, MusicSource.YOUTUBE, 1)
.single()
val song = insertSong(mediaStore, MusicSource.YOUTUBE, "vid-1", "One")
MoodRepository.markRejected(mood.id, recommendation.externalId)
assertEquals(
RecommendationCompletion.REJECTED,
MoodRepository.completeDownload(
mood.id,
MusicSource.YOUTUBE,
recommendation.externalId,
song.id,
),
)
assertTrue(MoodRepository.songIds(mood.id).isEmpty())
}
}
@Test
fun `adding while downloading attaches exactly one mood song`() {
withDatabase { mediaStore, _ ->
val mood = createMood(recommendationCount = 2)
storeCandidates(mood.id, 2)
val recommendation = MoodRepository
.promoteCandidates(mood.id, MusicSource.YOUTUBE, 1)
.single()
val song = insertSong(mediaStore, MusicSource.YOUTUBE, "vid-1", "One")
// User presses Add while the download is still running.
MoodRepository.markAdded(mood.id, recommendation.externalId)
assertTrue(MoodRepository.songIds(mood.id).isEmpty())
assertEquals(
RecommendationCompletion.ALREADY_ADDED,
MoodRepository.completeDownload(
mood.id,
MusicSource.YOUTUBE,
recommendation.externalId,
song.id,
),
)
assertEquals(listOf(song.id), MoodRepository.songIds(mood.id))
assertFalse(MoodRepository.addSong(mood.id, song.id, "recommendation"))
}
}
@Test
fun `mood counts reflect songs and active recommendation states`() {
withDatabase { mediaStore, _ ->
val mood = createMood(recommendationCount = 3)
val song = insertSong(mediaStore, MusicSource.YOUTUBE, "vid-1", "One")
MoodRepository.addSong(mood.id, song.id, "search")
storeCandidates(mood.id, 3)
val promoted = MoodRepository.promoteCandidates(mood.id, MusicSource.YOUTUBE, 2)
val ready = insertSong(mediaStore, MusicSource.YOUTUBE, "vid-ready", "Ready")
MoodRepository.completeDownload(
mood.id,
MusicSource.YOUTUBE,
promoted.first().externalId,
ready.id,
)
val record = assertNotNull(MoodRepository.get(mood.id, "alice"))
assertEquals(1, record.songCount)
assertEquals(1, record.recommendationReady)
assertEquals(1, record.recommendationDownloading)
assertEquals(0, record.recommendationFailed)
MoodRepository.failRecommendation(
mood.id,
promoted.last().externalId,
"boom",
3,
)
val failed = assertNotNull(MoodRepository.get(mood.id, "alice"))
assertEquals(0, failed.recommendationDownloading)
assertEquals(1, failed.recommendationFailed)
}
}
@Test
fun `removing a song only detaches it`() {
withDatabase { mediaStore, _ ->
val mood = createMood(recommendationCount = 0)
val song = insertSong(mediaStore, MusicSource.YOUTUBE, "vid-1", "One")
MoodRepository.addSong(mood.id, song.id, "search")
assertTrue(MoodRepository.removeSong(mood.id, song.id))
assertTrue(MoodRepository.songIds(mood.id).isEmpty())
assertNotNull(SongRepository.byId(song.id))
}
}
@Test
fun `GC keeps shared or playing files and deletes unreferenced ones`() {
withDatabase { mediaStore, gc ->
val firstMood = createMood(recommendationCount = 0, name = "First")
val secondMood = createMood(recommendationCount = 0, name = "Second")
val song = insertSong(mediaStore, MusicSource.YOUTUBE, "vid-1", "One")
val file = assertNotNull(mediaStore.file(song))
assertTrue(Files.isRegularFile(file))
MoodRepository.addSong(firstMood.id, song.id, "search")
MoodRepository.addSong(secondMood.id, song.id, "search")
MoodRepository.removeSong(firstMood.id, song.id)
assertFalse(gc.collect(song.id), "The second mood still references the song")
assertTrue(Files.isRegularFile(file))
// An open playback session protects the track even after detach.
val now = Instant.now().toString()
val session = RadioRepository.createSession("alice", secondMood.id, now)
RadioRepository.append(session.id, listOf(song.id))
MoodRepository.removeSong(secondMood.id, song.id)
assertFalse(gc.collect(song.id), "An open session still references the song")
RadioRepository.close(session.id, "alice")
assertTrue(gc.collect(song.id))
assertNull(SongRepository.byId(song.id))
assertFalse(Files.exists(mediaStore.root.resolve("youtube-vid-1")))
}
}
@Test
fun `a ready recommendation protects its file`() {
withDatabase { mediaStore, gc ->
val mood = createMood(recommendationCount = 2)
storeCandidates(mood.id, 2)
val recommendation = MoodRepository
.promoteCandidates(mood.id, MusicSource.YOUTUBE, 1)
.single()
val song = insertSong(mediaStore, MusicSource.YOUTUBE, "vid-1", "One")
MoodRepository.completeDownload(
mood.id,
MusicSource.YOUTUBE,
recommendation.externalId,
song.id,
)
assertFalse(gc.collect(song.id))
MoodRepository.markRejected(mood.id, recommendation.externalId)
assertTrue(gc.collect(song.id))
}
}
private fun withDatabase(block: (MediaStore, MediaGcService) -> Unit) {
val handle = DatabaseFactory.connect(appConfig())
try {
ConfigRepository.sync(UsersConfig(listOf(UserAccount("alice", "pw", admin = true))))
val store = MediaStore(Files.createDirectories(tempDir.resolve("media")))
val gc = MediaGcService(store, CoverArtService(tempDir.resolve("data")))
block(store, gc)
} finally {
handle.close()
}
}
private fun createMood(
recommendationCount: Int,
name: String = "Focus",
source: String = MusicSource.YOUTUBE,
): MoodRecord = MoodRepository.create(
owner = "alice",
name = name,
description = null,
query = null,
genres = emptyList(),
source = source,
collectionPercent = 50,
recommendationCount = recommendationCount,
refreshHours = 24,
)
private fun storeCandidates(
moodId: String,
count: Int,
startAt: Int = 1,
source: String = MusicSource.YOUTUBE,
) {
MoodRepository.storeCandidates(
moodId,
source,
(startAt until startAt + count).map { index ->
RecommendationUpsert(
source = source,
externalId = "v-$index",
title = "Track $index",
artist = "Artist $index",
album = null,
duration = 180,
thumbnail = null,
)
},
Instant.now().toString(),
)
}
private fun insertSong(
mediaStore: MediaStore,
source: String,
externalId: String,
title: String,
): SongRecord {
val file = mediaStore.target(source, externalId, title)
Files.createDirectories(file.parent)
Files.write(file, "fake-mp3".toByteArray())
return SongRepository.upsert(
SongUpsert(
source = source,
externalId = externalId,
title = title,
artist = "Artist",
album = null,
duration = 180,
bitRate = 128,
size = Files.size(file),
relativePath = mediaStore.root.relativize(file).joinToString("/") { it.toString() },
fileMtime = Files.getLastModifiedTime(file).toMillis(),
thumbnail = null,
),
)
}
private fun appConfig(): AppConfig = AppConfig(
port = 8080,
dataDir = tempDir.resolve("data"),
usersFile = tempDir.resolve("users.yaml"),
ffmpegPath = "ffmpeg",
)
}
@@ -0,0 +1,147 @@
package win.taptap.musicmanager.discovery
import com.sun.net.httpserver.HttpServer
import java.net.InetSocketAddress
import java.util.concurrent.CopyOnWriteArrayList
import kotlin.test.Test
import kotlin.test.assertEquals
import kotlin.test.assertTrue
/**
* Exercises the Deezer similarity walk against a local stand-in API, so the
* request shapes and the affinity weights stay pinned without network access.
*/
class DeezerDiscoverTest {
@Test
fun `expands the collection through related artists and album deep cuts`() {
val requests = CopyOnWriteArrayList<String>()
val server = serve { path ->
requests += path
when (path) {
"/search/artist" ->
ok("""{"data":[{"id":27,"name":"Daft Punk"}]}""")
"/search/album" ->
ok(
"""{"data":[{"id":5,"title":"Random Access Memories",
"artist":{"name":"Daft Punk"}}]}""",
)
"/artist/27/top" ->
ok(
"""{"data":[
{"id":1,"title":"Instant Crush","duration":337,"readable":true,
"artist":{"name":"Daft Punk"},
"album":{"title":"Random Access Memories"}},
{"id":2,"title":"Too Long","duration":900,"readable":true,
"artist":{"name":"Daft Punk"},
"album":{"title":"Random Access Memories"}}]}""",
)
"/artist/27/radio" ->
ok(
"""{"data":[
{"id":5,"title":"Kids","duration":303,"readable":true,
"artist":{"name":"MGMT"},
"album":{"title":"Oracular Spectacular"}}]}""",
)
"/artist/27/related" ->
ok("""{"data":[{"id":99,"name":"Daft Punk Tribute"}]}""")
"/artist/99/top" ->
ok(
"""{"data":[
{"id":3,"title":"Robot Rock","duration":300,"readable":true,
"artist":{"name":"Daft Punk Tribute"}}]}""",
)
"/album/5/tracks" ->
ok(
"""{"data":[
{"id":4,"title":"Contact","duration":382,"readable":true,
"artist":{"name":"Daft Punk"},
"album":{"title":"Random Access Memories"}}]}""",
)
else -> 404 to """{"error":{"type":"DataException","message":"not found"}}"""
}
}
try {
val found = catalog(server).discover(seeds(), limit = 50).associateBy { it.externalId }
// The 15-minute track is not music-length and never reaches the pool.
assertEquals(setOf("1", "3", "4", "5"), found.keys)
assertEquals(0.5, found.getValue("1").affinity)
// Deezer's own radio is the strongest affinity signal we can get.
assertEquals(2.0, found.getValue("5").affinity)
assertEquals("MGMT", found.getValue("5").artist)
assertEquals(1.5, found.getValue("3").affinity)
assertEquals(1.0, found.getValue("4").affinity)
// `/album/{id}/tracks` has no album object; the search result fills it in.
assertEquals("Random Access Memories", found.getValue("4").album)
assertTrue("/artist/27/related" in requests)
assertTrue("/artist/99/top" in requests)
assertTrue("/album/5/tracks" in requests)
} finally {
server.stop(0)
}
}
@Test
fun `a failing endpoint does not fail the whole expansion`() {
val server = serve { path ->
when (path) {
"/search/artist" -> ok("""{"data":[{"id":27,"name":"Daft Punk"}]}""")
"/artist/27/top" ->
ok(
"""{"data":[
{"id":1,"title":"Instant Crush","duration":337,"readable":true,
"artist":{"name":"Daft Punk"}}]}""",
)
else -> 500 to """{"error":{"type":"DataException","message":"boom"}}"""
}
}
try {
val found = catalog(server).discover(seeds(), limit = 50)
assertEquals(listOf("1"), found.map { it.externalId })
} finally {
server.stop(0)
}
}
@Test
fun `an empty collection makes no requests`() {
val requests = CopyOnWriteArrayList<String>()
val server = serve { path ->
requests += path
404 to """{"error":{"type":"DataException","message":"not found"}}"""
}
try {
assertTrue(catalog(server).discover(DiscoverySeeds(), limit = 50).isEmpty())
assertTrue(requests.isEmpty())
} finally {
server.stop(0)
}
}
private fun seeds() = DiscoverySeeds(
artists = listOf(SeedArtist("Daft Punk", 4)),
albums = listOf(SeedAlbum("Random Access Memories", "Daft Punk")),
)
private fun catalog(server: HttpServer) = DeezerSearch(
apiBase = "http://127.0.0.1:${server.address.port}",
timeoutSeconds = 5,
)
private fun ok(body: String) = 200 to body
private fun serve(route: (String) -> Pair<Int, String>): HttpServer {
val server = HttpServer.create(InetSocketAddress("127.0.0.1", 0), 0)
server.createContext("/") { exchange ->
val (status, body) = route(exchange.requestURI.path)
val bytes = body.toByteArray()
exchange.responseHeaders.add("Content-Type", "application/json")
exchange.sendResponseHeaders(status, bytes.size.toLong())
exchange.responseBody.use { it.write(bytes) }
}
server.start()
return server
}
}
@@ -0,0 +1,463 @@
package win.taptap.musicmanager.discovery
import java.time.Instant
import kotlin.test.Test
import kotlin.test.assertEquals
import kotlin.test.assertFalse
import kotlin.test.assertTrue
import win.taptap.musicmanager.db.MoodRecord
import win.taptap.musicmanager.db.SongRecord
import win.taptap.musicmanager.source.MusicSource
class DiscoveryServiceTest {
@Test
fun `parses flat playlist output and splits artist from title`() {
val json = """
{"id":"search","entries":[
{"id":"a","title":"Alpha - Focus One (Official Video)","duration":123.0,
"channel":"Alpha","thumbnails":[{"url":"http://t/1.jpg"},{"url":"http://t/2.jpg"}]},
{"id":"b","title":"Some Song","duration":61,"channel":"Some Artist - Topic"},
{"id":"c","title":"Loud | Official Audio","duration":90,"channel":"Gamma"}
]}
""".trimIndent()
val parsed = parseYtSearchOutput(json)
assertEquals(3, parsed.size)
assertEquals(
TrackCandidate(MusicSource.YOUTUBE, "a", "Focus One", "Alpha", null, 123, "http://t/2.jpg"),
parsed[0],
)
assertEquals(
TrackCandidate(MusicSource.YOUTUBE, "b", "Some Song", "Some Artist", null, 61, null),
parsed[1],
)
assertEquals(
TrackCandidate(MusicSource.YOUTUBE, "c", "Loud", "Gamma", null, 90, null),
parsed[2],
)
}
@Test
fun `parses resolved YouTube Music songs and drops browse entries`() {
val json = """
{"id":"daft punk","entries":[
{"id":"wU26xVT_vBU","title":"One More Time (Radio Edit)","artist":"Daft Punk",
"artists":["Daft Punk"],"album":"One More Time","duration":320,
"thumbnails":[{"url":"http://t/1.jpg"},{"url":"http://t/2.jpg"}]},
{"id":"4D7u5KF7SP8","title":"Get Lucky (feat. Pharrell Williams and Nile Rodgers)",
"artists":["Daft Punk","Pharrell Williams","Nile Rodgers"],
"album":"Random Access Memories","duration":370},
{"id":"UCRr1xG_2WIDs18a6cIiCxeA","ie_key":"YoutubeTab","title":"Daft Punk - Topic"},
{"id":"MPREb_uWQeJO6Z53u","ie_key":"YoutubeTab","title":"Album - One More Time"},
{"id":"khnokW3Mw24","title":"Instant Crush (feat. Julian Casablancas)","artist":"Daft Punk",
"album":"NA","duration":338}
]}
""".trimIndent()
val parsed = parseYtSearchOutput(json)
assertEquals(3, parsed.size)
assertEquals(
TrackCandidate(
source = MusicSource.YOUTUBE,
externalId = "wU26xVT_vBU",
title = "One More Time (Radio Edit)",
artist = "Daft Punk",
album = "One More Time",
duration = 320,
thumbnail = "http://t/2.jpg",
),
parsed[0],
)
assertEquals("Get Lucky (feat. Pharrell Williams and Nile Rodgers)", parsed[1].title)
assertEquals("Daft Punk, Pharrell Williams, Nile Rodgers", parsed[1].artist)
assertEquals(null, parsed[2].album)
}
@Test
fun `parses Deezer track results with album covers`() {
val json = """
{"data":[
{"id":67238732,"title":"Instant Crush (feat. Julian Casablancas)","duration":337,
"readable":true,"artist":{"name":"Daft Punk"},
"album":{"title":"Random Access Memories","cover_big":"http://c/500.jpg"}},
{"id":1,"title":"Unreadable","duration":200,"readable":false,
"artist":{"name":"X"},"album":{"title":"Y"}},
{"id":2,"title":"Too Long","duration":900,"readable":true,
"artist":{"name":"X"},"album":{"title":"Y"}},
{"id":3,"title":"Too Short","duration":10,"readable":true,
"artist":{"name":"X"},"album":{"title":"Y"}}
]}
""".trimIndent()
val parsed = parseDeezerSearch(json)
assertEquals(1, parsed.size)
assertEquals(
TrackCandidate(
source = MusicSource.DEEZER,
externalId = "67238732",
title = "Instant Crush (feat. Julian Casablancas)",
artist = "Daft Punk",
album = "Random Access Memories",
duration = 337,
thumbnail = "http://c/500.jpg",
),
parsed.single(),
)
}
@Test
fun `resolved Deezer URLs keep long tracks so the API can reject them`() {
val json = """
{"id":1,"title":"Long Set","duration":900,"readable":true,
"artist":{"name":"DJ"},"album":{"title":"Long"}}
""".trimIndent()
assertEquals("Long Set", parseDeezerTrack(json)?.title)
}
@Test
fun `parses a single resolved Deezer track`() {
val json = """
{"id":66609426,"title":"Get Lucky (Radio Edit)","duration":248,"readable":true,
"artist":{"name":"Daft Punk"},"album":{"title":"Get Lucky","cover_xl":"http://c/xl.jpg"}}
""".trimIndent()
assertEquals(
TrackCandidate(
source = MusicSource.DEEZER,
externalId = "66609426",
title = "Get Lucky (Radio Edit)",
artist = "Daft Punk",
album = "Get Lucky",
duration = 248,
thumbnail = "http://c/xl.jpg",
),
parseDeezerTrack(json),
)
}
@Test
fun `parses a single resolved video`() {
val json = """
{"id":"dQw4w9WgXcQ","title":"Rick Astley - Never Gonna Give You Up (Official Video)",
"channel":"Rick Astley","duration":213,"thumbnail":"http://t/x.jpg"}
""".trimIndent()
assertEquals(
TrackCandidate(
source = MusicSource.YOUTUBE,
externalId = "dQw4w9WgXcQ",
title = "Never Gonna Give You Up",
artist = "Rick Astley",
album = null,
duration = 213,
thumbnail = "http://t/x.jpg",
),
parseYtVideo(json),
)
}
@Test
fun `malformed search output yields no candidates`() {
assertTrue(parseYtSearchOutput("").isEmpty())
assertTrue(parseYtSearchOutput("not json").isEmpty())
assertTrue(parseYtSearchOutput("""{"entries": "nope"}""").isEmpty())
assertEquals(null, parseYtVideo("not json"))
assertTrue(parseDeezerSearch("not json").isEmpty())
assertEquals(null, parseDeezerTrack("not json"))
}
@Test
fun `music length keeps tracks between 30 seconds and 10 minutes`() {
assertTrue(isMusicLength(null))
assertTrue(isMusicLength(30))
assertTrue(isMusicLength(200))
assertTrue(isMusicLength(600))
assertFalse(isMusicLength(29))
assertFalse(isMusicLength(601))
assertFalse(isMusicLength(21 * 60))
}
@Test
fun `match keys ignore brackets punctuation and featuring credits`() {
assertEquals(matchKey("Focus One"), matchKey("Focus One (Official Video)"))
assertEquals(matchKey("Focus One"), matchKey("Focus One [Remastered]"))
assertEquals(matchKey("Alpha"), matchKey("Alpha feat. Beta"))
assertTrue(artistMatches("Alpha - Topic", "Alpha"))
assertTrue(artistMatches("Alpha & Friends", "Alpha"))
assertFalse(artistMatches("Beta", "Alpha"))
assertFalse(artistMatches("A", "ABBA"))
}
@Test
fun `freshness follows the mood refresh window`() {
val now = Instant.parse("2026-11-15T10:00:00Z")
assertFalse(isFresh(null, 24, now))
assertTrue(isFresh("2026-11-15T01:00:00Z", 24, now))
assertFalse(isFresh("2026-11-13T01:00:00Z", 24, now))
}
@Test
fun `queries combine text collection artists and genres`() {
val mood = mood(query = "late night", genres = listOf("Ambient", "Jazz"))
val queries = buildQueries(mood, listOf("Alpha", "Beta", "Alpha"))
assertEquals(listOf("late night", "Alpha", "Beta", "Ambient mix"), queries)
}
@Test
fun `a mood without a hint genres or songs has no query`() {
assertTrue(buildQueries(mood(), emptyList()).isEmpty())
}
@Test
fun `collection artists score higher`() {
val candidate = TrackCandidate(
source = MusicSource.DEEZER,
externalId = "1",
title = "Focus Two",
artist = "Beta",
duration = 200,
)
val plain = scoreCandidate(candidate, mapOf("alpha" to 1.0), emptySet(), "focus")
val seedArtist = scoreCandidate(candidate, mapOf("beta" to 1.0), emptySet(), "focus")
assertTrue(seedArtist > plain)
}
@Test
fun `an artist with more songs in the mood ranks higher`() {
val candidate = TrackCandidate(
source = MusicSource.DEEZER,
externalId = "1",
title = "Focus Two",
artist = "Beta",
duration = 200,
)
val once = scoreCandidate(candidate, mapOf("beta" to 1.0), emptySet(), "")
val often = scoreCandidate(candidate, mapOf("beta" to 12.0), emptySet(), "")
assertTrue(often > once)
}
@Test
fun `album deep cuts outrank tracks from unrelated albums`() {
val candidate = TrackCandidate(
source = MusicSource.DEEZER,
externalId = "1",
title = "Contact",
artist = "Daft Punk",
album = "Random Access Memories",
duration = 382,
)
val unrelated = scoreCandidate(candidate, emptyMap(), setOf("homework"), "")
val sameAlbum = scoreCandidate(
candidate,
emptyMap(),
setOf(matchKey("Random Access Memories")),
"",
)
assertEquals(1.0, sameAlbum - unrelated)
}
@Test
fun `merge keeps one copy of a recording and prefers the stronger affinity`() {
val album = TrackCandidate(
source = MusicSource.DEEZER,
externalId = "album-copy",
title = "Get Lucky (Radio Edit)",
artist = "Daft Punk",
duration = 248,
)
val radio = album.copy(externalId = "radio-copy", affinity = 2.0)
val cover = TrackCandidate(
source = MusicSource.DEEZER,
externalId = "cover",
title = "Get Lucky",
artist = "Daft Punk Tribute",
duration = 240,
)
val merged = mergeCandidates(listOf(album, cover, radio))
assertEquals(setOf("radio-copy", "cover"), merged.keys)
assertEquals(2.0, merged.getValue("radio-copy").affinity)
assertEquals(
recordingKey("Daft Punk", "Get Lucky"),
recordingKey("Daft Punk", "Get Lucky (Radio Edit)"),
)
}
@Test
fun `ranking spreads picks across artists instead of repeating one`() {
val lead = (1..12).map {
TrackCandidate(
source = MusicSource.DEEZER,
externalId = "lead-$it",
title = "Lead $it",
artist = "Alpha",
duration = 200,
)
}
val related = (1..6).map {
TrackCandidate(
source = MusicSource.DEEZER,
externalId = "related-$it",
title = "Related $it",
artist = "Beta",
duration = 200,
affinity = 1.5,
)
}
val scored = (lead + related).map {
it to scoreCandidate(it, mapOf("alpha" to 1.0), emptySet(), "")
}
val ranked = rankCandidates(scored, limit = 8)
assertEquals(8, ranked.size)
assertTrue(ranked.any { it.artist == "Beta" }, "A related artist must take a slot")
assertTrue(
ranked.count { it.artist == "Alpha" } <= 5,
"The lead artist must not fill every slot",
)
}
@Test
fun `catalog affinity raises a related artist hit`() {
val candidate = TrackCandidate(
source = MusicSource.DEEZER,
externalId = "1",
title = "Robot Rock",
artist = "Daft Punk Tribute",
duration = 300,
)
val plain = scoreCandidate(candidate, emptyMap(), emptySet(), "")
val related = scoreCandidate(
candidate.copy(affinity = 1.5),
emptyMap(),
emptySet(),
"",
)
assertEquals(1.5, related - plain)
}
@Test
fun `queries prefer collection artists over genres`() {
val mood = mood(query = "late night", genres = listOf("Ambient", "Jazz"))
val queries = buildQueries(mood, listOf("Alpha", "Beta", "Gamma", "Delta"))
assertEquals(listOf("late night", "Alpha", "Beta", "Gamma"), queries)
}
@Test
fun `collection artists are ordered by how many songs the mood has`() {
val collection = listOf(
song("Alpha"),
song("Beta"),
song("Alpha"),
song("Beta"),
song("Alpha"),
)
assertEquals(
listOf(SeedArtist("Alpha", 3), SeedArtist("Beta", 2)),
artistsByFrequency(collection),
)
}
@Test
fun `radio seeds use the newest song per artist of the same catalog`() {
val collection = listOf(
song("Alpha", externalId = "yt-1", source = MusicSource.YOUTUBE),
song("Beta", externalId = "yt-2", source = MusicSource.YOUTUBE),
song("Alpha", externalId = "yt-3", source = MusicSource.YOUTUBE),
song("Gamma", externalId = "dz-1", source = MusicSource.DEEZER),
)
assertEquals(
listOf("yt-3", "yt-2"),
radioSeedIds(collection, MusicSource.YOUTUBE, limit = 2),
)
assertEquals(
listOf("dz-1"),
radioSeedIds(collection, MusicSource.DEEZER, limit = 2),
)
}
@Test
fun `albums are collected once per title and artist`() {
val collection = listOf(
song("Daft Punk", album = "Random Access Memories"),
song("Daft Punk", album = "Random Access Memories"),
song("Daft Punk", album = "Homework"),
song("Other", album = null),
)
assertEquals(
listOf(
SeedAlbum("Random Access Memories", "Daft Punk"),
SeedAlbum("Homework", "Daft Punk"),
),
albumsOf(collection),
)
}
private fun song(
artist: String,
album: String? = "Album",
source: String = MusicSource.DEEZER,
externalId: String = "1",
) = SongRecord(
id = "sg-$artist-$album-$source-$externalId",
source = source,
externalId = externalId,
title = "Title",
artist = artist,
album = album,
duration = 180,
bitRate = 128,
size = 1,
suffix = "mp3",
contentType = "audio/mpeg",
relativePath = "deezer-1/Title.mp3",
fileMtime = 0,
thumbnail = null,
createdAt = "2026-01-01T00:00:00Z",
updatedAt = "2026-01-01T00:00:00Z",
)
private fun mood(
query: String? = null,
genres: List<String> = emptyList(),
) = MoodRecord(
id = "mo-1",
owner = "alice",
name = "Focus",
description = null,
query = query,
genres = genres,
source = MusicSource.DEEZER,
collectionPercent = 50,
recommendationCount = 10,
refreshHours = 24,
lastDiscoveryAt = null,
createdAt = "2026-01-01T00:00:00Z",
changedAt = "2026-01-01T00:00:00Z",
lastError = null,
songCount = 0,
recommendationReady = 0,
recommendationDownloading = 0,
recommendationFailed = 0,
)
}
@@ -0,0 +1,452 @@
package win.taptap.musicmanager.download
import java.nio.file.AccessDeniedException
import java.nio.file.Files
import java.nio.file.Path
import java.time.Instant
import kotlin.test.Test
import kotlin.test.assertEquals
import kotlin.test.assertNotNull
import kotlin.test.assertNull
import kotlin.test.assertTrue
import org.junit.jupiter.api.io.TempDir
import win.taptap.musicmanager.TestAudio
import win.taptap.musicmanager.config.AppConfig
import win.taptap.musicmanager.config.UserAccount
import win.taptap.musicmanager.config.UsersConfig
import win.taptap.musicmanager.db.ConfigRepository
import win.taptap.musicmanager.db.DatabaseFactory
import win.taptap.musicmanager.db.DatabaseHandle
import win.taptap.musicmanager.db.DownloadRepository
import win.taptap.musicmanager.db.MoodRecord
import win.taptap.musicmanager.db.MoodRepository
import win.taptap.musicmanager.db.RecommendationUpsert
import win.taptap.musicmanager.db.SongRecord
import win.taptap.musicmanager.db.SongRepository
import win.taptap.musicmanager.db.SongUpsert
import win.taptap.musicmanager.media.MediaGcService
import win.taptap.musicmanager.media.MediaStore
import win.taptap.musicmanager.scanner.CoverArtService
import win.taptap.musicmanager.scanner.Mp3TagExtractor
import win.taptap.musicmanager.scanner.SongIndexer
import win.taptap.musicmanager.source.MusicSource
class DownloadServiceTest {
@TempDir
lateinit var tempDir: Path
@Test
fun `a manual download lands in the media store and attaches to the mood`() {
TestAudio.assumeFfmpeg()
withService { service, mediaStore, _ ->
val mood = mood(recommendationCount = 0)
DownloadRepository.enqueue(
MusicSource.YOUTUBE,
"vid-manual",
mood.id,
DownloadRepository.KIND_SONG,
)
assertTrue(service.processOne())
val job = assertNotNull(
DownloadRepository.find(mood.id, "vid-manual", DownloadRepository.KIND_SONG),
)
assertEquals("done", job.state)
val songId = assertNotNull(job.songId)
assertEquals(listOf(songId), MoodRepository.songIds(mood.id))
val song = assertNotNull(SongRepository.byId(songId))
assertEquals(MusicSource.YOUTUBE, song.source)
assertEquals("vid-manual", song.externalId)
assertEquals("Song vid-manual", song.title)
assertEquals("Artist", song.artist)
assertEquals("youtube-vid-manual/Song vid-manual.mp3", song.relativePath)
assertTrue(Files.isRegularFile(assertNotNull(mediaStore.file(song))))
}
}
@Test
fun `a Deezer job uses the Deezer downloader and stores its tags`() {
var called = false
val youtube = AudioDownloader { _, _ -> error("YouTube downloader must not run") }
val deezer = AudioDownloader { externalId, dir ->
called = true
assertEquals("67238732", externalId)
mp3("deezer-$externalId", dir, title = "Instant Crush", artist = "Daft Punk")
}
withService(youtube = youtube, deezer = deezer) { service, mediaStore, _ ->
val mood = mood(recommendationCount = 0, source = MusicSource.DEEZER)
DownloadRepository.enqueue(
MusicSource.DEEZER,
"67238732",
mood.id,
DownloadRepository.KIND_SONG,
)
assertTrue(service.processOne())
assertTrue(called)
val song = assertNotNull(SongRepository.byExternalId(MusicSource.DEEZER, "67238732"))
assertEquals("Daft Punk", song.artist)
assertEquals("deezer-67238732/Instant Crush.mp3", song.relativePath)
assertTrue(Files.isRegularFile(assertNotNull(mediaStore.file(song))))
}
}
@Test
fun `an already downloaded song attaches without another download`() {
withService(
youtube = AudioDownloader { _, _ -> error("downloader must not run") },
) { service, mediaStore, _ ->
val mood = mood(recommendationCount = 0)
val song = insertSong(mediaStore, MusicSource.YOUTUBE, "vid-existing")
DownloadRepository.enqueue(
MusicSource.YOUTUBE,
"vid-existing",
mood.id,
DownloadRepository.KIND_SONG,
)
assertTrue(service.processOne())
assertEquals(listOf(song.id), MoodRepository.songIds(mood.id))
}
}
@Test
fun `the same video is downloaded once for two moods`() {
TestAudio.assumeFfmpeg()
var downloads = 0
withService(
youtube = AudioDownloader { externalId, dir -> downloads++; mp3(externalId, dir) },
) { service, _, _ ->
val first = mood(recommendationCount = 0, name = "First")
val second = mood(recommendationCount = 0, name = "Second")
DownloadRepository.enqueue(
MusicSource.YOUTUBE,
"vid-shared",
first.id,
DownloadRepository.KIND_SONG,
)
DownloadRepository.enqueue(
MusicSource.YOUTUBE,
"vid-shared",
second.id,
DownloadRepository.KIND_SONG,
)
assertTrue(service.processOne())
assertTrue(service.processOne())
assertEquals(1, downloads)
assertEquals(1, SongRepository.count())
assertEquals(MoodRepository.songIds(first.id), MoodRepository.songIds(second.id))
}
}
@Test
fun `a recommendation download becomes ready but is not a mood song`() {
TestAudio.assumeFfmpeg()
withService { service, _, _ ->
val mood = mood(recommendationCount = 1)
storeRecommendation(mood.id, "vid-rec")
val promoted = MoodRepository
.promoteCandidates(mood.id, MusicSource.YOUTUBE, 1)
.single()
DownloadRepository.enqueue(
promoted.source,
promoted.externalId,
mood.id,
DownloadRepository.KIND_RECOMMENDATION,
)
assertTrue(service.processOne())
val recommendation = assertNotNull(
MoodRepository.recommendation(mood.id, "vid-rec"),
)
assertEquals("ready", recommendation.state)
assertNotNull(recommendation.songId)
assertTrue(MoodRepository.songIds(mood.id).isEmpty())
}
}
@Test
fun `adding while downloading attaches when the job finishes`() {
TestAudio.assumeFfmpeg()
withService { service, _, _ ->
val mood = mood(recommendationCount = 1)
storeRecommendation(mood.id, "vid-rec")
MoodRepository.promoteCandidates(mood.id, MusicSource.YOUTUBE, 1)
DownloadRepository.enqueue(
MusicSource.YOUTUBE,
"vid-rec",
mood.id,
DownloadRepository.KIND_RECOMMENDATION,
)
MoodRepository.markAdded(mood.id, "vid-rec")
assertTrue(service.processOne())
val recommendation = assertNotNull(MoodRepository.recommendation(mood.id, "vid-rec"))
assertEquals("added", recommendation.state)
assertEquals(listOf(assertNotNull(recommendation.songId)), MoodRepository.songIds(mood.id))
}
}
@Test
fun `rejecting while downloading leaves no mood song and collects the file`() {
TestAudio.assumeFfmpeg()
withService { service, mediaStore, _ ->
val mood = mood(recommendationCount = 1)
storeRecommendation(mood.id, "vid-rec")
MoodRepository.promoteCandidates(mood.id, MusicSource.YOUTUBE, 1)
DownloadRepository.enqueue(
MusicSource.YOUTUBE,
"vid-rec",
mood.id,
DownloadRepository.KIND_RECOMMENDATION,
)
MoodRepository.markRejected(mood.id, "vid-rec")
assertTrue(service.processOne())
assertTrue(MoodRepository.songIds(mood.id).isEmpty())
assertNull(SongRepository.byExternalId(MusicSource.YOUTUBE, "vid-rec"))
assertTrue(Files.notExists(mediaStore.root.resolve("youtube-vid-rec")))
}
}
@Test
fun `failures retry three times and leave the recommendation failed`() {
withService(youtube = AudioDownloader { _, _ -> error("network down") }) { service, _, _ ->
val mood = mood(recommendationCount = 1)
storeRecommendation(mood.id, "vid-broken")
MoodRepository.promoteCandidates(mood.id, MusicSource.YOUTUBE, 1)
DownloadRepository.enqueue(
MusicSource.YOUTUBE,
"vid-broken",
mood.id,
DownloadRepository.KIND_RECOMMENDATION,
)
assertTrue(service.processOne())
assertTrue(service.processOne())
assertTrue(service.processOne())
val job = assertNotNull(
DownloadRepository.find(
mood.id,
"vid-broken",
DownloadRepository.KIND_RECOMMENDATION,
),
)
assertEquals("failed", job.state)
assertEquals(3, job.attempts)
assertTrue(job.error?.contains("network down") == true)
val recommendation = assertNotNull(MoodRepository.recommendation(mood.id, "vid-broken"))
assertEquals("failed", recommendation.state)
assertTrue(recommendation.error?.contains("network down") == true)
}
}
@Test
fun `permission errors fail without burning retries`() {
withService(
youtube = AudioDownloader { _, _ -> throw AccessDeniedException("/data/media") },
) { service, _, _ ->
val mood = mood(recommendationCount = 0)
DownloadRepository.enqueue(
MusicSource.YOUTUBE,
"vid-denied",
mood.id,
DownloadRepository.KIND_SONG,
)
assertTrue(service.processOne())
val job = assertNotNull(
DownloadRepository.find(mood.id, "vid-denied", DownloadRepository.KIND_SONG),
)
assertEquals("failed", job.state)
assertEquals(1, job.attempts)
assertTrue(job.error?.contains("AccessDeniedException") == true)
}
}
@Test
fun `an unconfigured Deezer account fails permanently`() {
val deezer = DeezerDownloader(arl = null)
withService(deezer = deezer) { service, _, _ ->
val mood = mood(recommendationCount = 0, source = MusicSource.DEEZER)
DownloadRepository.enqueue(
MusicSource.DEEZER,
"67238732",
mood.id,
DownloadRepository.KIND_SONG,
)
assertTrue(service.processOne())
val job = assertNotNull(
DownloadRepository.find(mood.id, "67238732", DownloadRepository.KIND_SONG),
)
assertEquals("failed", job.state)
assertEquals(1, job.attempts)
assertTrue(job.error?.contains("Deezer is not configured") == true)
}
}
@Test
fun `manual adds outrank recommendation downloads`() {
withDatabase { _, _ ->
val mood = mood(recommendationCount = 1)
DownloadRepository.enqueue(
MusicSource.YOUTUBE,
"vid-rec",
mood.id,
DownloadRepository.KIND_RECOMMENDATION,
)
DownloadRepository.enqueue(
MusicSource.YOUTUBE,
"vid-manual",
mood.id,
DownloadRepository.KIND_SONG,
)
val claimed = assertNotNull(DownloadRepository.claimNext())
assertEquals("vid-manual", claimed.externalId)
assertEquals(DownloadRepository.KIND_SONG, claimed.kind)
}
}
private fun mood(
recommendationCount: Int,
name: String = "Focus",
source: String = MusicSource.YOUTUBE,
): MoodRecord = MoodRepository.create(
owner = "alice",
name = name,
description = null,
query = null,
genres = emptyList(),
source = source,
collectionPercent = 50,
recommendationCount = recommendationCount,
refreshHours = 24,
)
private fun storeRecommendation(moodId: String, externalId: String) {
MoodRepository.storeCandidates(
moodId,
MusicSource.YOUTUBE,
listOf(
RecommendationUpsert(
source = MusicSource.YOUTUBE,
externalId = externalId,
title = "Recommended",
artist = "Artist",
album = null,
duration = 180,
thumbnail = null,
),
),
Instant.now().toString(),
)
}
private fun withService(
youtube: AudioDownloader = AudioDownloader { externalId, dir -> mp3(externalId, dir) },
deezer: AudioDownloader = AudioDownloader { externalId, dir -> mp3("deezer-$externalId", dir) },
block: (DownloadService, MediaStore, MediaGcService) -> Unit,
) {
withDatabase { handle, store ->
val coverArt = CoverArtService(tempDir.resolve("data"))
val indexer = SongIndexer(store, coverArt)
val gc = MediaGcService(store, coverArt)
val service = DownloadService(
downloaders = SourceDownloaders(
mapOf(
MusicSource.YOUTUBE to youtube,
MusicSource.DEEZER to deezer,
),
),
mediaStore = store,
indexer = indexer,
gc = gc,
tagExtractor = Mp3TagExtractor(),
tempDir = tempDir.resolve("downloads"),
)
try {
block(service, store, gc)
} finally {
handle.close()
}
}
}
private fun withDatabase(block: (DatabaseHandle, MediaStore) -> Unit) {
val handle = DatabaseFactory.connect(appConfig())
try {
ConfigRepository.sync(UsersConfig(listOf(UserAccount("alice", "pw", admin = true))))
val store = MediaStore(Files.createDirectories(tempDir.resolve("media")))
block(handle, store)
} finally {
handle.close()
}
}
private fun mp3(
externalId: String,
dir: Path,
title: String = "Song $externalId",
artist: String = "Artist",
): Path {
val file = dir.resolve("$externalId.mp3")
TestAudio.createMp3(
file,
metadata = mapOf(
"title" to title,
"artist" to artist,
"album" to "Album",
),
)
return file
}
private fun insertSong(
mediaStore: MediaStore,
source: String,
externalId: String,
): SongRecord {
val file = mediaStore.target(source, externalId, "Existing")
Files.createDirectories(file.parent)
Files.write(file, "fake-mp3".toByteArray())
return SongRepository.upsert(
SongUpsert(
source = source,
externalId = externalId,
title = "Existing",
artist = "Artist",
album = null,
duration = 180,
bitRate = 128,
size = Files.size(file),
relativePath = mediaStore.root.relativize(file).joinToString("/") { it.toString() },
fileMtime = Files.getLastModifiedTime(file).toMillis(),
thumbnail = null,
),
)
}
private fun appConfig(): AppConfig = AppConfig(
port = 8080,
dataDir = tempDir.resolve("data"),
usersFile = tempDir.resolve("users.yaml"),
ffmpegPath = "ffmpeg",
downloadDir = tempDir.resolve("downloads"),
)
}
@@ -0,0 +1,53 @@
package win.taptap.musicmanager.media
import kotlin.test.Test
import kotlin.test.assertEquals
import kotlin.test.assertNull
class LoudnessAnalyzerTest {
@Test
fun `parses the loudnorm summary`() {
val output = """
[Parsed_loudnorm_0 @ 0x1]
{
"input_i" : "-9.42",
"input_tp" : "-0.21",
"input_lra" : "6.10",
"output_i" : "-16.00"
}
""".trimIndent()
val measurement = LoudnessAnalyzer.parse(output)
assertEquals(-9.42, measurement?.lufs)
assertEquals(-0.21, measurement?.truePeakDb)
}
@Test
fun `gain targets minus sixteen lufs`() {
val analyzer = LoudnessAnalyzer(targetLufs = -16.0, truePeakDb = -1.5)
assertEquals(-6.6, analyzer.gainFor(LoudnessAnalyzer.Measurement(-9.4, -3.0)))
}
@Test
fun `gain is limited by the true peak ceiling`() {
val analyzer = LoudnessAnalyzer(targetLufs = -16.0, truePeakDb = -1.5)
// The loudness target would need +6 dB, but the peak only allows +1.5.
assertEquals(1.5, analyzer.gainFor(LoudnessAnalyzer.Measurement(-22.0, -3.0)))
}
@Test
fun `silence is left alone`() {
val analyzer = LoudnessAnalyzer()
assertEquals(0.0, analyzer.gainFor(LoudnessAnalyzer.Measurement(-70.0, -70.0)))
}
@Test
fun `missing summary is ignored`() {
assertNull(LoudnessAnalyzer.parse("ffmpeg: no such file"))
}
}
@@ -0,0 +1,140 @@
package win.taptap.musicmanager.radio
import kotlin.random.Random
import kotlin.test.Test
import kotlin.test.assertEquals
import kotlin.test.assertTrue
import win.taptap.musicmanager.db.AffinityRecord
import win.taptap.musicmanager.db.SongRecord
class RadioServiceTest {
private val service = RadioService()
@Test
fun `recently played and already heard songs are penalized`() {
val song = song("s-1")
assertEquals(
0.0,
service.scoreCandidate(song, affinity = 0.0, recent = false, jitter = 0.0),
)
assertEquals(
-6.0,
service.scoreCandidate(song, affinity = 0.0, recent = true, jitter = 0.0),
)
assertEquals(
-20.0,
service.scoreCandidate(song, affinity = 0.0, recent = false, jitter = 0.0, heard = true),
)
}
@Test
fun `skips outweigh completions`() {
assertEquals(1.0, AffinityRecord(completions = 1, skips = 0).score)
assertEquals(-2.0, AffinityRecord(completions = 1, skips = 1).score)
assertEquals(-5.0, AffinityRecord(completions = 1, skips = 2).score)
}
@Test
fun `higher scores are picked first`() {
val scored = listOf(
song("low", artist = "A") to 1.0,
song("high", artist = "B") to 9.0,
)
val picked = service.pickMixed(scored, emptyList(), limit = 2, collectionPercent = 100)
assertEquals("high", picked.first().id)
}
@Test
fun `picks keep artists and albums apart when possible`() {
val scored = listOf(
song("s1", artist = "A", album = "a1") to 0.0,
song("s2", artist = "B", album = "a2") to 0.0,
song("s3", artist = "C", album = "a3") to 0.0,
song("s4", artist = "D", album = "a4") to 0.0,
song("s5", artist = "A", album = "a5") to 0.0,
song("s6", artist = "B", album = "a6") to 0.0,
)
val picked = service.pickMixed(
scored,
emptyList(),
limit = 6,
collectionPercent = 100,
artistSpacing = 2,
albumSpacing = 3,
)
assertEquals(6, picked.size)
picked.indices.forEach { index ->
val artists = picked.subList(index, minOf(index + 3, picked.size)).map { it.artist }
assertEquals(artists.size, artists.distinct().size, "artist spacing at $index")
}
}
@Test
fun `mood mix draws from collection and recommendations in proportion`() {
val collection = (1..20).map { song("col-$it", artist = "Col $it") }
val recommendations = (1..20).map { song("rec-$it", artist = "Rec $it") }
val collectionScored = collection.mapIndexed { index, song -> song to (10.0 - index * 0.1) }
val recommendationScored = recommendations.mapIndexed { index, song -> song to (9.0 - index * 0.1) }
val allCollection = RadioService(Random(1)).pickMixed(
collectionScored, recommendationScored, limit = 10, collectionPercent = 100,
)
assertTrue(allCollection.all { it.id.startsWith("col-") })
val allRecommendations = RadioService(Random(1)).pickMixed(
collectionScored, recommendationScored, limit = 10, collectionPercent = 0,
)
assertTrue(allRecommendations.all { it.id.startsWith("rec-") })
val mixed = RadioService(Random(7)).pickMixed(
collectionScored, recommendationScored, limit = 20, collectionPercent = 50,
)
assertEquals(20, mixed.size)
assertEquals(20, mixed.map { it.id }.distinct().size)
assertTrue(mixed.any { it.id.startsWith("col-") })
assertTrue(mixed.any { it.id.startsWith("rec-") })
}
@Test
fun `mood mix falls back when one pool is empty`() {
val onlyCollection = listOf(song("only-collection"))
val picked = RadioService(Random(1)).pickMixed(
collectionScored = onlyCollection.map { it to 1.0 },
recommendationScored = emptyList(),
limit = 5,
collectionPercent = 0,
)
assertEquals(listOf("only-collection"), picked.map { it.id })
}
private fun song(
id: String,
artist: String = "Artist $id",
album: String? = "Album $id",
) = SongRecord(
id = id,
source = "youtube",
externalId = "video-$id",
title = "Title $id",
artist = artist,
album = album,
duration = 180,
bitRate = 320,
size = 1_000,
suffix = "mp3",
contentType = "audio/mpeg",
relativePath = "video-$id/$id.mp3",
fileMtime = 0,
thumbnail = null,
createdAt = "2026-01-01T00:00:00Z",
updatedAt = "2026-01-01T00:00:00Z",
)
}
@@ -2,32 +2,19 @@ package win.taptap.musicmanager.scanner
import java.nio.file.Files
import java.nio.file.Path
import java.nio.file.attribute.FileTime
import kotlin.test.Test
import kotlin.test.assertEquals
import kotlin.test.assertNotNull
import kotlin.test.assertNull
import kotlin.test.assertTrue
import org.jetbrains.exposed.sql.insert
import org.jetbrains.exposed.sql.selectAll
import org.jetbrains.exposed.sql.transactions.transaction
import org.junit.jupiter.api.io.TempDir
import win.taptap.musicmanager.TestAudio
import win.taptap.musicmanager.config.AppConfig
import win.taptap.musicmanager.config.FolderType
import win.taptap.musicmanager.config.LibraryConfig
import win.taptap.musicmanager.config.MusicFolder
import win.taptap.musicmanager.config.UserAccount
import win.taptap.musicmanager.config.UsersConfig
import win.taptap.musicmanager.db.AlbumsTable
import win.taptap.musicmanager.db.ArtistsTable
import win.taptap.musicmanager.db.ConfigRepository
import win.taptap.musicmanager.db.DatabaseFactory
import win.taptap.musicmanager.db.DatabaseHandle
import win.taptap.musicmanager.db.MediaRepository
import win.taptap.musicmanager.db.SongGenresTable
import win.taptap.musicmanager.db.SongsTable
import win.taptap.musicmanager.db.UserSongStateTable
import win.taptap.musicmanager.db.SongRepository
import win.taptap.musicmanager.media.MediaStore
import win.taptap.musicmanager.source.MusicSource
import win.taptap.musicmanager.util.IdGenerator
class MediaScannerTest {
@@ -35,175 +22,101 @@ class MediaScannerTest {
@TempDir
lateinit var tempDir: Path
private lateinit var music: Path
private lateinit var longform: Path
private lateinit var media: Path
@Test
fun `scans, caches covers and keeps user state across metadata updates`() {
fun `indexes the media root by video id and caches covers`() {
TestAudio.assumeFfmpeg()
val db = DatabaseFactory.connect(appConfig())
try {
val library = library()
ConfigRepository.sync(library, users())
val albumDir = Files.createDirectories(music.resolve("Album Artist/Album"))
TestAudio.writePng(albumDir.resolve("cover.jpg"))
val one = albumDir.resolve("01 - One.mp3")
val two = albumDir.resolve("02 - Two.mp3")
TestAudio.createMp3(one, metadata = albumMetadata("One", "1/2"))
TestAudio.createMp3(two, metadata = albumMetadata("Two", "2/2"))
val book = longform.resolve("Author/Book/01 - Chapter One.mp3")
withScanner { scanner, coverArt ->
val one = media.resolve("vid-one/One.mp3")
val two = media.resolve("vid-two/Two.mp3")
val cover = media.resolve("cover.png")
TestAudio.writePng(cover)
TestAudio.createMp3(
book,
one,
metadata = mapOf(
"title" to "Chapter One",
"artist" to "Author",
"album_artist" to "Author",
"album" to "Book",
"title" to "One",
"artist" to "Alpha",
"album" to "Album",
"genre" to "Rock",
),
cover = cover,
)
Files.writeString(music.resolve("notes.txt"), "not audio")
val scanner = MediaScanner(library, CoverArtService(tempDir.resolve("data")))
TestAudio.createMp3(two, metadata = mapOf("title" to "Two", "artist" to "Beta"))
Files.writeString(media.resolve("notes.txt"), "not audio")
Files.writeString(media.resolve("loose.mp3"), "not in a video folder")
val first = scanner.scanBlocking()
assertNull(first.error)
assertEquals(3L, first.count)
assertEquals(3L, MediaRepository.countSongs())
assertEquals(2L, first.count)
assertEquals(2L, SongRepository.count())
val albumId = IdGenerator.albumId("Album Artist", "Album")
val songId = IdGenerator.songId(1, "Album Artist/Album/01 - One.mp3")
val song = assertNotNull(SongRepository.byExternalId(MusicSource.YOUTUBE, "vid-one"))
assertEquals(IdGenerator.songId(MusicSource.YOUTUBE, "vid-one"), song.id)
assertEquals("One", song.title)
assertEquals("Alpha", song.artist)
assertEquals("Album", song.album)
assertEquals("vid-one/One.mp3", song.relativePath)
assertEquals(128, song.bitRate)
assertEquals("mp3", song.suffix)
assertEquals("audio/mpeg", song.contentType)
assertEquals(Files.size(one), song.size)
transaction(db.database) {
assertEquals(3L, SongsTable.selectAll().count())
assertEquals(2L, AlbumsTable.selectAll().count())
assertEquals(2L, ArtistsTable.selectAll().count())
assertEquals(2L, SongGenresTable.selectAll().count())
assertTrue(Files.isRegularFile(coverArt.coverFile(song.id)))
val album = AlbumsTable.selectAll()
.where { AlbumsTable.id eq albumId }
.single()
assertEquals(albumId, album[AlbumsTable.coverArt])
val song = SongsTable.selectAll()
.where { SongsTable.id eq songId }
.single()
assertEquals(128, song[SongsTable.bitRate])
assertEquals("mp3", song[SongsTable.suffix])
assertEquals("audio/mpeg", song[SongsTable.contentType])
assertEquals(Files.size(one), song[SongsTable.size])
UserSongStateTable.insert {
it[userName] = "alice"
it[this.songId] = songId
it[playCount] = 7
}
}
assertTrue(Files.isRegularFile(tempDir.resolve("data/covers/$albumId.jpg")))
// Touching the file forces re-extraction; the play count must survive.
Files.setLastModifiedTime(one, FileTime.fromMillis(System.currentTimeMillis() + 2_000))
// Unchanged files are skipped on a second scan.
val second = scanner.scanBlocking()
assertEquals(3L, second.count)
assertEquals(3L, MediaRepository.countSongs())
assertEquals(2L, second.count)
assertEquals(2L, SongRepository.count())
transaction(db.database) {
val state = UserSongStateTable.selectAll()
.where { UserSongStateTable.songId eq songId }
.single()
assertEquals(7, state[UserSongStateTable.playCount])
}
// Removing the whole album and the long-form book cleans up orphans.
// Deleting a file removes its row.
Files.delete(two)
Files.delete(one)
Files.delete(book)
val third = scanner.scanBlocking()
assertNull(third.error)
assertEquals(0L, third.count) // notes.txt is ignored, the three MP3s are gone
assertEquals(0L, MediaRepository.countSongs())
assertEquals(1L, third.count)
assertNull(SongRepository.byExternalId(MusicSource.YOUTUBE, "vid-two"))
transaction(db.database) {
assertEquals(0L, SongsTable.selectAll().count())
assertEquals(0L, AlbumsTable.selectAll().count())
assertEquals(0L, ArtistsTable.selectAll().count())
}
} finally {
db.close()
// Re-adding it indexes it again.
TestAudio.createMp3(two, metadata = mapOf("title" to "Two", "artist" to "Beta"))
val fourth = scanner.scanBlocking()
assertEquals(2L, fourth.count)
assertNotNull(SongRepository.byExternalId(MusicSource.YOUTUBE, "vid-two"))
}
}
@Test
fun `unchanged files are skipped on a second scan`() {
fun `a missing media root does not delete its songs`() {
TestAudio.assumeFfmpeg()
withScanner { scanner, _ ->
val file = media.resolve("vid-one/One.mp3")
TestAudio.createMp3(file)
assertEquals(1L, scanner.scanBlocking().count)
media.toFile().deleteRecursively()
val broken = scanner.scanBlocking()
assertNotNull(broken.error)
assertEquals(1L, SongRepository.count())
}
}
private fun withScanner(block: (MediaScanner, CoverArtService) -> Unit) {
val db = DatabaseFactory.connect(appConfig())
try {
val library = library()
ConfigRepository.sync(library, users())
val file = music.resolve("Artist/Album/song.mp3")
TestAudio.createMp3(file)
TestAudio.writePng(file.parent.resolve("cover.jpg"))
val scanner = MediaScanner(library, CoverArtService(tempDir.resolve("data")))
scanner.scanBlocking()
val albumId = IdGenerator.albumId("Artist", "Album")
transaction(db.database) {
val album = AlbumsTable.selectAll()
.where { AlbumsTable.id eq albumId }
.single()
assertEquals(albumId, album[AlbumsTable.coverArt])
}
assertTrue(Files.isRegularFile(tempDir.resolve("data/covers/$albumId.jpg")))
val before = transaction(db.database) {
SongsTable.selectAll().single()[SongsTable.fileMtime]
}
val mtimeBefore = Files.getLastModifiedTime(file).toMillis()
assertEquals(mtimeBefore, before)
val second = scanner.scanBlocking()
assertEquals(1L, second.count)
assertEquals(1L, MediaRepository.countSongs())
val store = MediaStore(Files.createDirectories(tempDir.resolve("media")))
media = store.root
val coverArt = CoverArtService(tempDir.resolve("data"))
val scanner = MediaScanner(store, SongIndexer(store, coverArt))
block(scanner, coverArt)
} finally {
db.close()
}
}
private fun albumMetadata(title: String, track: String): Map<String, String> = mapOf(
"title" to title,
"artist" to "Album Artist",
"album_artist" to "Album Artist",
"album" to "Album",
"track" to track,
"genre" to "Rock",
)
private fun appConfig(): AppConfig = AppConfig(
port = 8080,
dataDir = tempDir.resolve("data"),
libraryConfigPath = tempDir.resolve("library.yaml"),
usersFile = tempDir.resolve("users.yaml"),
ffmpegPath = "ffmpeg",
)
private fun library(): LibraryConfig {
music = Files.createDirectories(tempDir.resolve("music"))
longform = Files.createDirectories(tempDir.resolve("longform"))
return LibraryConfig(
listOf(
MusicFolder(1, "Music", FolderType.MUSIC, music),
MusicFolder(10, "Long-form", FolderType.LONGFORM, longform),
),
)
}
private fun users(): UsersConfig = UsersConfig(
listOf(UserAccount("alice", "pw", admin = true, folders = setOf(1))),
)
}
@@ -9,7 +9,6 @@ import kotlin.test.assertNull
import kotlin.test.assertTrue
import org.junit.jupiter.api.io.TempDir
import win.taptap.musicmanager.TestAudio
import win.taptap.musicmanager.util.IdGenerator
class Mp3TagExtractorTest {
@@ -17,95 +16,84 @@ class Mp3TagExtractorTest {
lateinit var tempDir: Path
@Test
fun `reads ID3 tags and separates album artist from track artists`() {
fun `reads ID3 tags from a downloaded MP3`() {
TestAudio.assumeFfmpeg()
val root = Files.createDirectories(tempDir.resolve("music"))
val file = root.resolve("Album Artist/Album/02 - Song.mp3")
val file = tempDir.resolve("video-id/02 - Song.mp3")
TestAudio.createMp3(
file,
metadata = mapOf(
"title" to "Test Song",
"artist" to "Track Artist",
"album_artist" to "Album Artist",
"album" to "Test Album",
"track" to "2/10",
"disc" to "1/2",
"date" to "2021-05-01",
"genre" to "Rock",
),
)
val track = assertNotNull(Mp3TagExtractor().extract(1, root, file))
val song = assertNotNull(Mp3TagExtractor().extract(file))
assertEquals("Test Song", track.title)
assertEquals("Album Artist/Album/02 - Song.mp3", track.relativePath)
assertEquals("Test Album", track.albumTitle)
assertEquals("Track Artist", track.displayArtist)
assertEquals(listOf("Track Artist"), track.artists.map { it.name })
assertEquals(listOf("Album Artist"), track.albumArtists.map { it.name })
assertEquals("Album Artist", track.albumArtists.first().name)
assertEquals(2, track.trackNumber)
assertEquals(1, track.discNumber)
assertEquals(2021, track.year)
assertEquals(listOf("Rock"), track.genres)
assertTrue(track.durationSeconds in 1..4)
assertTrue(track.bitRate > 0)
assertEquals(Files.size(file), track.size)
assertEquals("mp3", track.suffix)
assertEquals("audio/mpeg", track.contentType)
assertEquals(
IdGenerator.songId(1, "Album Artist/Album/02 - Song.mp3"),
track.songId,
)
assertEquals(
IdGenerator.albumId("Album Artist", "Test Album"),
track.albumId,
)
assertEquals("Test Song", song.title)
assertEquals("Track Artist", song.artist)
assertEquals("Test Album", song.album)
assertTrue(song.durationSeconds in 1..4)
assertTrue(song.bitRate > 0)
assertEquals(Files.size(file), song.size)
assertEquals("mp3", song.suffix)
assertEquals("audio/mpeg", song.contentType)
}
@Test
fun `falls back to the folder structure when tags are missing`() {
fun `cleans the raw YouTube title prefix and upload noise`() {
TestAudio.assumeFfmpeg()
val root = Files.createDirectories(tempDir.resolve("music"))
val file = root.resolve("Artist/Album/song.mp3")
TestAudio.createMp3(file)
val track = assertNotNull(Mp3TagExtractor().extract(1, root, file))
assertEquals("song", track.title)
assertEquals("Album", track.albumTitle)
assertEquals("Artist", track.displayArtist)
assertEquals(listOf("Artist"), track.albumArtists.map { it.name })
}
@Test
fun `reads embedded artwork and sidecar lyrics`() {
TestAudio.assumeFfmpeg()
val root = Files.createDirectories(tempDir.resolve("music"))
val cover = root.resolve("cover.png")
TestAudio.writePng(cover)
val file = root.resolve("Artist/Album/song.mp3")
val file = tempDir.resolve("video-id/rick.mp3")
TestAudio.createMp3(
file,
metadata = mapOf("title" to "Covered"),
cover = cover,
metadata = mapOf(
"title" to "Rick Astley - Never Gonna Give You Up (Official Video) (4K Remaster)",
"artist" to "Rick Astley - Topic",
),
)
Files.writeString(file.resolveSibling("song.lrc"), "[00:00.00]Hello")
val track = assertNotNull(Mp3TagExtractor().extract(1, root, file))
val song = assertNotNull(Mp3TagExtractor().extract(file))
assertNotNull(track.coverBytes)
assertTrue(track.coverBytes!!.isNotEmpty())
assertEquals("[00:00.00]Hello", track.lyrics)
assertEquals("Never Gonna Give You Up", song.title)
assertEquals("Rick Astley", song.artist)
}
@Test
fun `falls back to the file name and a generic artist`() {
TestAudio.assumeFfmpeg()
val file = tempDir.resolve("video-id/My Track.mp3")
TestAudio.createMp3(file)
val song = assertNotNull(Mp3TagExtractor().extract(file))
assertEquals("My Track", song.title)
assertEquals("Unknown artist", song.artist)
assertNull(song.album)
}
@Test
fun `reads embedded artwork`() {
TestAudio.assumeFfmpeg()
val cover = tempDir.resolve("cover.png")
TestAudio.writePng(cover)
val file = tempDir.resolve("video-id/covered.mp3")
TestAudio.createMp3(file, metadata = mapOf("title" to "Covered"), cover = cover)
val song = assertNotNull(Mp3TagExtractor().extract(file))
assertNotNull(song.coverBytes)
assertTrue(song.coverBytes!!.isNotEmpty())
}
@Test
fun `ignores non-mp3 files`() {
val root = Files.createDirectories(tempDir.resolve("music"))
val file = root.resolve("song.flac")
val file = tempDir.resolve("video-id/song.flac")
Files.createDirectories(file.parent)
Files.writeString(file, "not audio")
assertNull(Mp3TagExtractor().extract(1, root, file))
assertNull(Mp3TagExtractor().extract(file))
}
}
@@ -15,42 +15,29 @@ class IdGeneratorTest {
}
@Test
fun `IDs are deterministic, prefixed and 24 hex characters`() {
val first = IdGenerator.artistId("The Beatles")
val second = IdGenerator.artistId("the beatles")
fun `song IDs are deterministic, prefixed and 24 hex characters`() {
val first = IdGenerator.songId("youtube", "dQw4w9WgXcQ")
val second = IdGenerator.songId("youtube", " dQw4w9WgXcQ ")
assertEquals(first, second)
assertTrue(first.startsWith("ar-"))
assertTrue(first.startsWith("sg-"))
assertEquals(27, first.length)
val album = IdGenerator.albumId("The Beatles", "Abbey Road")
assertTrue(album.startsWith("al-"))
assertEquals(27, album.length)
val song = IdGenerator.songId(1, "The Beatles/Abbey Road/01 - Come Together.mp3")
assertTrue(song.startsWith("sg-"))
assertEquals(27, song.length)
}
@Test
fun `path separators do not change the song ID`() {
val unix = IdGenerator.songId(1, "Artist/Album/song.mp3")
val windows = IdGenerator.songId(1, "Artist\\Album\\song.mp3")
val prefixed = IdGenerator.songId(1, "./Artist/Album/song.mp3")
assertEquals(unix, windows)
assertEquals(unix, prefixed)
fun `different videos produce different IDs`() {
assertNotEquals(
IdGenerator.songId("youtube", "dQw4w9WgXcQ"),
IdGenerator.songId("youtube", "dQw4w9WgXcR"),
)
}
@Test
fun `folder and album changes produce different IDs`() {
assertNotEquals(
IdGenerator.songId(1, "Artist/Album/song.mp3"),
IdGenerator.songId(2, "Artist/Album/song.mp3"),
)
assertNotEquals(
IdGenerator.albumId("Artist", "Album"),
IdGenerator.albumId("Artist", "Other Album"),
)
fun `mood IDs are deterministic for the same inputs`() {
val id = IdGenerator.moodId("Alice", "Focus", "2026-01-01T00:00:00Z")
assertTrue(id.startsWith("mo-"))
assertEquals(id, IdGenerator.moodId("alice", " Focus ", "2026-01-01T00:00:00Z"))
assertNotEquals(id, IdGenerator.moodId("Alice", "Focus", "2026-01-02T00:00:00Z"))
}
}