feat: bootstrap MusicManager with scanner and Subsonic auth
- Gradle/Kotlin/Ktor project on JVM 21 with ShadowJar - SQLite WAL, Flyway migrations and config-driven users/folders - deterministic IDs, MP3 tag extraction, cover cache and delta scanner - OpenSubsonic JSON envelope with token/password authentication - system endpoints: ping, license, extensions, scan status
This commit is contained in:
1 parent
961341b449
commit
e194badefb
45 files changed
+4079
-126
No files matched your search
@@ -1,152 +1,310 @@
|
||||
# Implementation Plan: OpenSubsonic Music Server (Kotlin / Ktor)
|
||||
# MusicManager — Self-hosted OpenSubsonic Server (Kotlin / Ktor)
|
||||
|
||||
## 1. Projektziel & Kontext
|
||||
Entwickle einen leichtgewichtigen, selbstgehosteten **OpenSubsonic-kompatiblen Musik-Server** in **Kotlin**, der Plex für Musik-Streaming ablöst und speziell auf die Zusammenarbeit mit dem Android-Client **Symfonium** optimiert ist.
|
||||
## 1. Goal & Context
|
||||
|
||||
### Kernanforderungen (MVP)
|
||||
1. **Library-Indexing:** Einlesen eines lokalen Musik-Verzeichnisses (primär MP3, zusätzlich FLAC/OGG/M4A/OPUS), Auslesen der Metadaten (Tag-First mit Ordnerstruktur `Artist/Album/Track` als Fallback) und Persistierung in SQLite.
|
||||
2. **Symfonium-Sync:** Vollständige Unterstützung der OpenSubsonic-Browsing-Endpunkte, damit Symfonium den gesamten Katalog sauber in seine lokale Offline-First-Datenbank synchronisiert.
|
||||
3. **Streaming & Offline-Downloads:** Auslieferung der Audiodateien inkl. HTTP `Range`-Requests (`206 Partial Content`) sowie exakter Byte-Größen, damit Seeking und Symfoniums Offline-Download-Manager zuverlässig funktionieren.
|
||||
4. **Bidirektionale Interaktion:** Scrobbling (Play-Counts), Favoriten (`star`/`unstar`), Bewertungen (`setRating`), Playlists und Startseiten-Feeds (`getAlbumList2`, `getRandomSongs`).
|
||||
5. **Erweiterbarkeit:** Saubere Trennung zwischen Media-Scanner, Datenbank-Layer und HTTP-API, sodass später ein automatisierter Music-Downloader und eine Discovery-/Recommendation-Engine (z. B. über ListenBrainz/Last.fm oder Staging-Playlists) angedockt werden können.
|
||||
Build a lightweight, self-hosted **OpenSubsonic-compatible music server** in **Kotlin** that replaces Plex for audio streaming and is optimised for the Android client **Symfonium**.
|
||||
|
||||
### Scope (MVP)
|
||||
|
||||
1. **Library indexing:** index MP3 files only for now (other extensions are skipped and logged). Tags come first, with the folder layout `Artist/Album/Track` as fallback. Data is persisted in SQLite.
|
||||
2. **Symfonium sync:** implement the OpenSubsonic browsing endpoints so Symfonium can sync the full catalog into its offline-first database.
|
||||
3. **Streaming & offline downloads:** serve audio files with HTTP `Range` (`206 Partial Content`), exact byte sizes and correct metadata so seeking and Symfonium's offline download manager work reliably.
|
||||
4. **Two content types, never mixed:** *music* and *long-form* (audiobooks, podcasts, lectures) live in **separate music folders**. Clients see and select them independently.
|
||||
5. **Multi-user:** users are defined outside the database (config + HashiCorp Vault). Music roots are assigned per user, long-form is shared. Stars, ratings, play counts, bookmarks and playlists are per user.
|
||||
6. **Transcoding:** optional on-the-fly MP3 transcoding with `maxBitRate` and `timeOffset` support.
|
||||
7. **Extensibility:** clean separation between scanner, database layer and HTTP API so a downloader and a discovery/recommendation engine can be added later.
|
||||
|
||||
### Explicit non-goals (MVP)
|
||||
|
||||
- Audio formats other than MP3 (FLAC/OGG/M4A/Opus are future work).
|
||||
- XML responses (JSON only).
|
||||
- Podcast RSS ingestion, chapters or audiobook-specific metadata.
|
||||
- Upload, sharing, social features.
|
||||
- User-management API (users come from config/Vault).
|
||||
- TLS inside the app (terminated by the k3s ingress).
|
||||
- Automatic filesystem watching (scans are manual or on boot when the DB is empty).
|
||||
- Backups (documented manual procedure only).
|
||||
|
||||
### Roles
|
||||
|
||||
- **Admin:** may trigger scans.
|
||||
- **User:** regular library access.
|
||||
- All users share the same long-form roots; each user has their own music roots.
|
||||
- Per-user state: play counts, last played, stars, ratings, bookmarks, playlists.
|
||||
|
||||
---
|
||||
|
||||
## 2. Tech-Stack & Dependencies
|
||||
## 2. Tech Stack & Dependencies
|
||||
|
||||
* **Sprache & Runtime:** Kotlin 2.x auf JVM 21 (`eclipse-temurin:21-jre-alpine`)
|
||||
* **Build-Tool:** Gradle (Kotlin DSL `build.gradle.kts`) mit `ShadowJar` Plugin (Fat-JAR)
|
||||
* **Web-Framework:** **Ktor Server (Netty)**
|
||||
* `ktor-server-core`, `ktor-server-netty`
|
||||
* `ktor-server-content-negotiation` + `ktor-serialization-kotlinx-json`
|
||||
* `ktor-server-partial-content` (Pflicht für HTTP `Range` / `206 Partial Content`)
|
||||
* `ktor-server-status-pages`, `ktor-server-call-logging`, `ktor-server-auto-head-response`
|
||||
* **Datenbank:** **SQLite** (im `WAL`-Mode) + **JetBrains Exposed** (`exposed-core`, `exposed-dao`, `exposed-jdbc`, `exposed-java-time`) + `sqlite-jdbc` + `HikariCP`
|
||||
* **Audio-Metadaten & Cover:**
|
||||
* `net.jthink:jaudiotagger` (Auslesen von ID3v1/v2, FLAC, Ogg, M4A, Dauer, Bitrate, Cover, Lyrics)
|
||||
* `net.coobird:thumbnailator` (On-the-fly Resizing von Album-Covern für `getCoverArt?size=...`)
|
||||
* **Logging:** `logback-classic` + `kotlin-logging-jvm`
|
||||
- **Language & runtime:** Kotlin 2.x on JVM 21 (`eclipse-temurin:21-jre-alpine`)
|
||||
- **Build:** Gradle Kotlin DSL (`build.gradle.kts`), ShadowJar fat JAR
|
||||
- **Web:** Ktor Server (Netty)
|
||||
- `ktor-server-core`, `ktor-server-netty`
|
||||
- `ktor-server-content-negotiation` + `ktor-serialization-kotlinx-json`
|
||||
- `ktor-server-partial-content` (HTTP `Range` / `206`)
|
||||
- `ktor-server-status-pages`, `ktor-server-call-logging`, `ktor-server-auto-head-response`
|
||||
- **Database:** SQLite in `WAL` mode + JetBrains Exposed (`exposed-core`, `exposed-dao`, `exposed-jdbc`, `exposed-java-time`) + `sqlite-jdbc` + HikariCP
|
||||
- **Migrations:** Flyway (`flyway-core`) with versioned SQL under `src/main/resources/db/migration`
|
||||
- **Audio metadata:** `net.jthink:jaudiotagger` (MP3/ID3 only for the MVP)
|
||||
- **Covers:** `net.coobird:thumbnailator` for on-the-fly resizing
|
||||
- **Transcoding:** `ffmpeg` binary in the runtime image (libmp3lame), invoked as a subprocess
|
||||
- **Logging:** `logback-classic` + `kotlin-logging-jvm`
|
||||
- **Config:** YAML (`library.yaml`, `users.yaml`) with env-var overrides for paths and secrets
|
||||
- **Deployment:** multi-stage Dockerfile, Helm chart for k3s, Gitea Actions for image builds
|
||||
|
||||
---
|
||||
|
||||
## 3. Projektstruktur
|
||||
## 3. Project Structure
|
||||
|
||||
```text
|
||||
src/main/kotlin/com/musicserver/
|
||||
├── Application.kt # Ktor Einstiegspunkt, Plugins, Graceful Shutdown
|
||||
src/main/kotlin/win/taptap/musicmanager/
|
||||
├── Application.kt # Ktor entry point, plugins, graceful shutdown
|
||||
├── config/
|
||||
│ └── AppConfig.kt # Env-Vars (MUSIC_DIR, DATA_DIR, SUBSONIC_USER, SUBSONIC_PASS, PORT)
|
||||
│ ├── AppConfig.kt # Port, data dir, ffmpeg path, env overrides
|
||||
│ ├── LibraryConfig.kt # Music folders and their type/path
|
||||
│ └── UsersConfig.kt # Users, admins and folder assignments
|
||||
├── db/
|
||||
│ ├── DatabaseFactory.kt # SQLite Init, PRAGMA journal_mode=WAL, Schema-Migration
|
||||
│ ├── Tables.kt # Exposed Table-Definitionen
|
||||
│ └── Repositories.kt # CRUD- & Query-Logik für Artists, Albums, Songs, Playlists
|
||||
│ ├── DatabaseFactory.kt # SQLite init, PRAGMA journal_mode=WAL, Flyway
|
||||
│ ├── Tables.kt # Exposed table definitions
|
||||
│ ├── Repositories.kt # CRUD & queries for artists, albums, songs, playlists
|
||||
│ └── migrations/ # Flyway SQL (V1__init.sql, ...)
|
||||
├── scanner/
|
||||
│ ├── MediaScanner.kt # Rekursiver Verzeichnis-Scan, Delta-Updates, Orphan-Cleanup
|
||||
│ ├── TagExtractor.kt # jaudiotagger Wrapper (Tags, Fallback auf Pfad, Dauer, Bitrate)
|
||||
│ └── CoverArtService.kt # Cover-Extraktion aus Tags/Ordnern, Disk-Cache & Resizing
|
||||
│ ├── MediaScanner.kt # Recursive scan, delta updates, orphan cleanup
|
||||
│ ├── Mp3TagExtractor.kt # jaudiotagger wrapper (tags, path fallback, duration, bitrate)
|
||||
│ └── CoverArtService.kt # Cover extraction from tags/folders, disk cache, resizing
|
||||
├── subsonic/
|
||||
│ ├── SubsonicAuth.kt # Ktor Plugin/Interceptor für Token-Auth (u, t, s) & Plaintext (p)
|
||||
│ ├── SubsonicResponse.kt # Standardisierter JSON-Envelope ("subsonic-response") & Fehlercodes
|
||||
│ ├── models/ # kotlinx.serialization DTOs für OpenSubsonic
|
||||
│ ├── SubsonicAuth.kt # Token (u, t, s) and password (p, enc:) auth
|
||||
│ ├── SubsonicResponse.kt # JSON envelope & error codes
|
||||
│ ├── models/ # kotlinx.serialization DTOs
|
||||
│ │ ├── SystemModels.kt
|
||||
│ │ ├── BrowsingModels.kt
|
||||
│ │ ├── MediaModels.kt
|
||||
│ │ └── PlaylistModels.kt
|
||||
│ └── routes/ # Routing unter /rest/{endpoint} und /rest/{endpoint}.view
|
||||
│ ├── SystemRoutes.kt # ping, getLicense, getOpenSubsonicExtensions, startScan, getScanStatus
|
||||
│ ├── BrowsingRoutes.kt # getMusicFolders, getIndexes, getArtists, getArtist, getAlbum, getSong, search3, getAlbumList2, getRandomSongs
|
||||
│ ├── MediaRoutes.kt # stream, download, getCoverArt, getLyricsBySongId
|
||||
│ ├── InteractionRoutes.kt # scrobble, star, unstar, setRating
|
||||
│ └── PlaylistRoutes.kt # getPlaylists, getPlaylist, createPlaylist, updatePlaylist, deletePlaylist
|
||||
│ └── routes/ # /rest/{endpoint} and /rest/{endpoint}.view
|
||||
│ ├── SystemRoutes.kt # ping, getLicense, extensions, startScan, getScanStatus
|
||||
│ ├── BrowsingRoutes.kt # indexes, artists, album, song, search3, lists, genres
|
||||
│ ├── MediaRoutes.kt # stream, download, getCoverArt, lyrics
|
||||
│ ├── InteractionRoutes.kt # scrobble, star/unstar, setRating, getStarred2
|
||||
│ └── PlaylistRoutes.kt # playlists and playlist songs
|
||||
├── transcoding/
|
||||
│ └── FfmpegTranscoder.kt # MP3 transcoding with maxBitRate and timeOffset
|
||||
└── util/
|
||||
└── IdGenerator.kt # Deterministische SHA-256 ID-Generierung
|
||||
└── IdGenerator.kt # Deterministic SHA-256 IDs
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. Kritische Architektur-Regeln (Symfonium-Kompatibilität)
|
||||
## 4. Architecture Rules
|
||||
|
||||
1. **Deterministische IDs (Extrem wichtig!):**
|
||||
* IDs dürfen sich bei einem erneuten Library-Scan **niemals** ändern, sonst verliert Symfonium den Offline-Download-Status und Playlist-Zuordnungen.
|
||||
* **Artist-ID:** `"ar-" + sha256(normalizedAlbumArtist).take(16)`
|
||||
* **Album-ID:** `"al-" + sha256(normalizedAlbumArtist + "|" + normalizedAlbumTitle).take(16)`
|
||||
* **Song-ID:** `"sg-" + sha256(relativeFilePathFromMusicDir).take(16)`
|
||||
* **Cover-ID:** Identisch zur `Album-ID` (`"al-..."`) bzw. `Artist-ID` (`"ar-..."`).
|
||||
2. **Album-Gruppierung über `AlbumArtist`:**
|
||||
* Alben müssen primär über den `ALBUM_ARTIST`-Tag gruppiert werden (Fallback: `ARTIST`-Tag, Fallback: Elternordner). Andernfalls zerfallen Alben mit Featuring-Artists in einzelne Fragmente.
|
||||
3. **Exakte Datei-Metadaten für Offline-Downloads:**
|
||||
* Jedes Song-Objekt muss zwingend die exakte Dateigröße in Bytes (`size: Long`), die Spieldauer in Sekunden (`duration: Int`), `bitRate: Int`, `suffix: String` (z. B. `"mp3"`) und `contentType: String` (z. B. `"audio/mpeg"`) liefern. Symfonium prüft nach jedem Offline-Download, ob die heruntergeladenen Bytes exakt mit `size` übereinstimmen.
|
||||
4. **Routing-Konvention & `formPost`:**
|
||||
* Symfonium sendet Requests wahlweise an `/rest/getArtists` oder `/rest/getArtists.view` und nutzt bei langen Parameterlisten (z. B. `updatePlaylist`) `POST` mit `application/x-www-form-urlencoded`.
|
||||
* Registriere alle Endpunkte für **beide** URL-Varianten (`/{method}` und `/{method}.view`) und lies Parameter vereinheitlicht aus `call.request.queryParameters` **und** `call.receiveParameters()` (bei POST).
|
||||
5. **OpenSubsonic JSON-Envelope:**
|
||||
* Jede Antwort (Standard `f=json`) muss in folgendes Root-Objekt verpackt sein:
|
||||
```json
|
||||
{
|
||||
"subsonic-response": {
|
||||
"status": "ok",
|
||||
"version": "1.16.1",
|
||||
"type": "subsonic-kt",
|
||||
"serverVersion": "0.1.0",
|
||||
"openSubsonic": true
|
||||
}
|
||||
}
|
||||
```
|
||||
### 4.1 Deterministic IDs (critical)
|
||||
|
||||
IDs must never change on a re-scan; otherwise Symfonium loses offline download state and playlist assignments.
|
||||
|
||||
- IDs use **24 hex characters** (96 bit).
|
||||
- **Artist ID:** `"ar-" + sha256(normalizedAlbumArtist).take(24)`
|
||||
- **Album ID:** `"al-" + sha256(normalizedAlbumArtist + "|" + normalizedAlbumTitle).take(24)`
|
||||
- **Song ID:** `"sg-" + sha256(folderId + ":" + relativePathFromFolderRoot).take(24)`
|
||||
- **Cover ID:** same as the album ID (`"al-..."`). Artist artwork is not served in the MVP.
|
||||
|
||||
**Normalization is frozen** and must never change without a migration:
|
||||
|
||||
1. Unicode NFC
|
||||
2. Trim and collapse internal whitespace
|
||||
3. Lowercase with `Locale.ROOT`
|
||||
|
||||
Song IDs are path-based: renaming or moving a file creates a new ID. This is accepted.
|
||||
|
||||
### 4.2 Album grouping via album artist
|
||||
|
||||
Albums are grouped by the `ALBUM_ARTIST` tag. Fallbacks: `ARTIST` tag, then parent folder name. This prevents albums with featuring artists from fragmenting.
|
||||
|
||||
- `albums` have one or more artists (`album_artists`), position 0 is the primary album artist.
|
||||
- Songs additionally keep a display artist string and their full artist list (`song_artists`).
|
||||
- `song.artist_id` in API responses means the **album artist**; track credits are exposed via `displayArtist` / `artists[]`.
|
||||
|
||||
### 4.3 User-scoped library visibility
|
||||
|
||||
- The catalog is global; users see only folders assigned to them.
|
||||
- Every browsing and media query filters songs by the requesting user's visible folder IDs.
|
||||
- Because album IDs are deterministic and roots can contain the same album for different users, an album may aggregate songs from several folders. Therefore:
|
||||
- `getAlbum(id)` returns only songs from folders visible to the caller.
|
||||
- An album/artist is returned only if it has at least one visible song.
|
||||
- Album-level names, years and covers are global; per-user state is never global.
|
||||
- `getMusicFolders` returns the caller's assigned music folders plus the shared long-form folders.
|
||||
- Songs store `folder_id`; `music_folders` is synced from config on startup and referenced by foreign key.
|
||||
- Long-form content uses the same artist/album/song model; separation is via folder type, not a separate API.
|
||||
|
||||
### 4.4 Exact file metadata for offline downloads
|
||||
|
||||
Every song object must include:
|
||||
|
||||
- `size: Long` — exact byte size
|
||||
- `duration: Int` — seconds
|
||||
- `bitRate: Int` — kbps
|
||||
- `suffix: String` — `"mp3"`
|
||||
- `contentType: String` — `"audio/mpeg"`
|
||||
|
||||
Symfonium verifies downloaded bytes against `size` after each offline download.
|
||||
|
||||
### 4.5 Routing conventions & `formPost`
|
||||
|
||||
- Register every endpoint for **both** `/{method}` and `/{method}.view`.
|
||||
- Read parameters from `call.request.queryParameters` **and** `call.receiveParameters()` (POST `application/x-www-form-urlencoded`).
|
||||
- Declare the `formPost` OpenSubsonic extension.
|
||||
- Repeatable parameters (`songId`, `songIdToAdd`, `songIndexToRemove`) must be read as lists.
|
||||
|
||||
### 4.6 JSON envelope
|
||||
|
||||
Every response, including errors, uses `status: "ok"` or `status: "failed"` and HTTP 200:
|
||||
|
||||
```json
|
||||
{
|
||||
"subsonic-response": {
|
||||
"status": "ok",
|
||||
"version": "1.16.1",
|
||||
"type": "musicmanager",
|
||||
"serverVersion": "0.1.0",
|
||||
"openSubsonic": true
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Error codes: `0` generic, `10` missing parameter, `20` incompatible client, `30` incompatible server, `40` wrong credentials, `50` not authorized, `70` not found.
|
||||
|
||||
### 4.7 Transcoding rules
|
||||
|
||||
- `stream` accepts `maxBitRate` (kbps), `format` and `timeOffset` (seconds).
|
||||
- Direct play when no transcode is requested or when the source bitrate is already `<= maxBitRate`.
|
||||
- Transcoding target is always **MP3** (`libmp3lame`). A requested `format` other than `mp3` falls back to MP3.
|
||||
- `timeOffset` is implemented with `ffmpeg -ss`, declared as the `transcodeOffset` OpenSubsonic extension.
|
||||
- Transcoded responses are chunked and do **not** support byte `Range`. Seeking uses `timeOffset`.
|
||||
- `download` always returns the original file, never a transcode.
|
||||
- Direct-play responses support `Range` / `206` via the `PartialContent` plugin.
|
||||
|
||||
---
|
||||
|
||||
## 5. Datenbank-Schema (SQLite)
|
||||
## 5. Database Schema (SQLite)
|
||||
|
||||
Identity and folder tables are synced from config at startup; passwords are never stored in SQLite.
|
||||
|
||||
```sql
|
||||
CREATE TABLE users (
|
||||
name TEXT PRIMARY KEY,
|
||||
is_admin BOOLEAN NOT NULL DEFAULT 0,
|
||||
updated_at TEXT NOT NULL
|
||||
);
|
||||
|
||||
CREATE TABLE music_folders (
|
||||
id INTEGER PRIMARY KEY,
|
||||
name TEXT NOT NULL,
|
||||
type TEXT NOT NULL CHECK (type IN ('music', 'longform')),
|
||||
root_path TEXT NOT NULL
|
||||
);
|
||||
|
||||
CREATE TABLE artists (
|
||||
id TEXT PRIMARY KEY,
|
||||
name TEXT NOT NULL,
|
||||
sort_name TEXT NOT NULL,
|
||||
cover_art TEXT,
|
||||
starred_at TEXT
|
||||
search_text TEXT NOT NULL
|
||||
);
|
||||
|
||||
CREATE TABLE albums (
|
||||
id TEXT PRIMARY KEY,
|
||||
artist_id TEXT NOT NULL REFERENCES artists(id) ON DELETE CASCADE,
|
||||
name TEXT NOT NULL,
|
||||
year INTEGER,
|
||||
genre TEXT,
|
||||
cover_art TEXT,
|
||||
created_at TEXT NOT NULL,
|
||||
starred_at TEXT,
|
||||
user_rating INTEGER
|
||||
search_text TEXT NOT NULL
|
||||
);
|
||||
|
||||
CREATE TABLE album_artists (
|
||||
album_id TEXT NOT NULL REFERENCES albums(id) ON DELETE CASCADE,
|
||||
artist_id TEXT NOT NULL REFERENCES artists(id) ON DELETE CASCADE,
|
||||
position INTEGER NOT NULL,
|
||||
PRIMARY KEY (album_id, position)
|
||||
);
|
||||
|
||||
CREATE TABLE songs (
|
||||
id TEXT PRIMARY KEY,
|
||||
album_id TEXT NOT NULL REFERENCES albums(id) ON DELETE CASCADE,
|
||||
artist_id TEXT NOT NULL REFERENCES artists(id) ON DELETE CASCADE,
|
||||
folder_id INTEGER NOT NULL REFERENCES music_folders(id),
|
||||
title TEXT NOT NULL,
|
||||
artist_name TEXT NOT NULL,
|
||||
display_artist TEXT NOT NULL,
|
||||
album_name TEXT NOT NULL,
|
||||
track_number INTEGER,
|
||||
disc_number INTEGER DEFAULT 1,
|
||||
year INTEGER,
|
||||
genre TEXT,
|
||||
duration INTEGER NOT NULL,
|
||||
bit_rate INTEGER NOT NULL,
|
||||
size INTEGER NOT NULL,
|
||||
suffix TEXT NOT NULL,
|
||||
content_type TEXT NOT NULL,
|
||||
relative_path TEXT NOT NULL UNIQUE,
|
||||
relative_path TEXT NOT NULL,
|
||||
file_mtime INTEGER NOT NULL,
|
||||
created_at TEXT NOT NULL,
|
||||
search_text TEXT NOT NULL,
|
||||
UNIQUE (folder_id, relative_path)
|
||||
);
|
||||
|
||||
CREATE TABLE song_artists (
|
||||
song_id TEXT NOT NULL REFERENCES songs(id) ON DELETE CASCADE,
|
||||
artist_id TEXT NOT NULL REFERENCES artists(id) ON DELETE CASCADE,
|
||||
position INTEGER NOT NULL,
|
||||
PRIMARY KEY (song_id, position)
|
||||
);
|
||||
|
||||
-- Multi-value tags. Text keys keep the schema simple; uniqueness is enforced.
|
||||
CREATE TABLE song_genres (
|
||||
song_id TEXT NOT NULL REFERENCES songs(id) ON DELETE CASCADE,
|
||||
genre TEXT NOT NULL,
|
||||
PRIMARY KEY (song_id, genre)
|
||||
);
|
||||
|
||||
CREATE TABLE album_genres (
|
||||
album_id TEXT NOT NULL REFERENCES albums(id) ON DELETE CASCADE,
|
||||
genre TEXT NOT NULL,
|
||||
PRIMARY KEY (album_id, genre)
|
||||
);
|
||||
|
||||
CREATE TABLE artist_genres (
|
||||
artist_id TEXT NOT NULL REFERENCES artists(id) ON DELETE CASCADE,
|
||||
genre TEXT NOT NULL,
|
||||
PRIMARY KEY (artist_id, genre)
|
||||
);
|
||||
|
||||
-- Per-user state
|
||||
CREATE TABLE user_song_state (
|
||||
user_name TEXT NOT NULL REFERENCES users(name) ON DELETE CASCADE,
|
||||
song_id TEXT NOT NULL REFERENCES songs(id) ON DELETE CASCADE,
|
||||
play_count INTEGER NOT NULL DEFAULT 0,
|
||||
last_played_at TEXT,
|
||||
starred_at TEXT,
|
||||
user_rating INTEGER
|
||||
rating INTEGER,
|
||||
PRIMARY KEY (user_name, song_id)
|
||||
);
|
||||
|
||||
CREATE TABLE user_stars (
|
||||
user_name TEXT NOT NULL REFERENCES users(name) ON DELETE CASCADE,
|
||||
item_type TEXT NOT NULL CHECK (item_type IN ('album', 'artist')),
|
||||
item_id TEXT NOT NULL,
|
||||
starred_at TEXT NOT NULL,
|
||||
PRIMARY KEY (user_name, item_type, item_id)
|
||||
);
|
||||
|
||||
CREATE TABLE bookmarks (
|
||||
user_name TEXT NOT NULL REFERENCES users(name) ON DELETE CASCADE,
|
||||
song_id TEXT NOT NULL REFERENCES songs(id) ON DELETE CASCADE,
|
||||
position_ms INTEGER NOT NULL,
|
||||
updated_at TEXT NOT NULL,
|
||||
PRIMARY KEY (user_name, song_id)
|
||||
);
|
||||
|
||||
CREATE TABLE playlists (
|
||||
id TEXT PRIMARY KEY,
|
||||
owner TEXT NOT NULL REFERENCES users(name) ON DELETE CASCADE,
|
||||
name TEXT NOT NULL,
|
||||
comment TEXT,
|
||||
owner TEXT NOT NULL,
|
||||
is_public BOOLEAN NOT NULL DEFAULT 1,
|
||||
created_at TEXT NOT NULL,
|
||||
changed_at TEXT NOT NULL
|
||||
@@ -160,55 +318,241 @@ CREATE TABLE playlist_songs (
|
||||
);
|
||||
```
|
||||
|
||||
---
|
||||
Required indexes:
|
||||
|
||||
## 6. Spezifikation der MVP-Endpunkte
|
||||
- `albums(search_text)`, `artists(search_text)`, `songs(search_text)`
|
||||
- `songs(album_id)`, `songs(folder_id)`, `songs(created_at)`
|
||||
- `album_artists(artist_id)`, `song_artists(artist_id)`
|
||||
- `user_song_state(user_name, starred_at)`, `user_stars(user_name, item_type)`
|
||||
- `playlist_songs(song_id)`, `playlists(owner)`
|
||||
|
||||
### Authentifizierung (`SubsonicAuth`)
|
||||
* Prüfe bei jedem `/rest/*`-Call (außer optional Healthcheck):
|
||||
* `u`: Muss mit `SUBSONIC_USER` übereinstimmen.
|
||||
* Entweder Token-Auth: `t` und `s` vorhanden -> prüfe `t.equals(md5(SUBSONIC_PASS + s), ignoreCase = true)`.
|
||||
* Oder Passwort-Auth: `p` vorhanden -> falls `p.startsWith("enc:")`, Hex-dekodieren, sonst direkt mit `SUBSONIC_PASS` vergleichen.
|
||||
* Bei Fehler: HTTP 200 mit Subsonic-Error-JSON (`code = 40`, `message = "Wrong username or password"`).
|
||||
|
||||
### Endpunkt-Liste
|
||||
1. **System & Scan:**
|
||||
* `ping`: Leeres `subsonic-response` mit `status: "ok"`.
|
||||
* `getLicense`: `{"license": {"valid": true}}`.
|
||||
* `getOpenSubsonicExtensions`: Deklariert `[{"name": "formPost", "versions": [1]}, {"name": "songLyrics", "versions": [1]}]`.
|
||||
* `startScan` / `getScanStatus`: Startet asynchronen Coroutine-Scan und gibt `{"scanStatus": {"scanning": Boolean, "count": Long}}` zurück.
|
||||
2. **Browsing & Suche:**
|
||||
* `getMusicFolders`: Liefert `[{"id": 1, "name": "Music"}]`.
|
||||
* `getIndexes`: Alphabetisch gruppierte Artists (`index: [{name: "A", artist: [...]}]`).
|
||||
* `getArtists`: Identische Alphabet-Struktur für ID3-basiertes Browsing (`artists: {ignoredArticles: "The El La Los Las Le Les", index: [...]}`).
|
||||
* `getArtist(id)`: Artist-Details inkl. Liste aller zugehörigen `album`-Objekte.
|
||||
* `getAlbum(id)`: Album-Details inkl. aller `song`-Objekte (sortiert nach `discNumber`, `trackNumber`).
|
||||
* `getSong(id)`: Einzelnes Song-Objekt.
|
||||
* `search3(query, artistCount, artistOffset, albumCount, albumOffset, songCount, songOffset)`: SQL-`LIKE`-Suche über Artists, Alben und Songs (`query=""` darf leere oder paginierte Ergebnisse liefern).
|
||||
3. **Home-Feeds:**
|
||||
* `getAlbumList2(type, size, offset, fromYear, toYear, genre)`: Unterstützt `type` = `newest`, `recent`, `frequent`, `random`, `alphabeticalByName`, `alphabeticalByArtist`, `starred`, `byYear`, `byGenre`.
|
||||
* `getRandomSongs(size, genre, fromYear, toYear)`: Zufällige Songs aus der DB.
|
||||
4. **Media & Artwork:**
|
||||
* `stream(id, maxBitRate, format)`: Liefert die Datei via `call.respondFile(file)` aus (Direct Play; `PartialContent`-Plugin übernimmt `Range`-Requests). Setzt `Content-Type` passend zur Datei.
|
||||
* `download(id)`: Wie `stream`, zusätzlich mit `Content-Disposition: attachment; filename="..."`.
|
||||
* `getCoverArt(id, size)`: Lädt das gecachte Cover aus `DATA_DIR/covers/{id}.jpg`. Falls `size` (Int) angegeben ist, skaliert `Thumbnailator` das Bild proportional auf `size x size` und cached die skalierte Variante (`{id}_{size}.jpg`).
|
||||
* `getLyricsBySongId(id)`: Liest eingebettete Lyrics (`FieldKey.LYRICS`) oder eine gleichnamige `.lrc`-Datei neben der Audiodatei und gibt sie im OpenSubsonic-`lyricsList`-Format zurück.
|
||||
5. **Interaktion & Playlists:**
|
||||
* `scrobble(id, time, submission)`: Bei `submission=true` (Default) `play_count + 1` und `last_played_at` aktualisieren.
|
||||
* `star(id, albumId, artistId)` / `unstar(id, albumId, artistId)`: Setzt bzw. löscht `starred_at`.
|
||||
* `setRating(id, rating)`: Setzt `user_rating` (1–5; bei `0` auf `null`).
|
||||
* `getPlaylists`, `getPlaylist(id)`, `createPlaylist(playlistId, name, songId)`, `updatePlaylist(playlistId, name, comment, public, songIdToAdd, songIndexToRemove)`, `deletePlaylist(playlistId)`.
|
||||
All timestamps are ISO 8601 in UTC.
|
||||
|
||||
---
|
||||
|
||||
## 7. Schrittweise Implementierungs-Reihenfolge
|
||||
## 6. Configuration
|
||||
|
||||
Setze das Projekt inkrementell in folgenden Schritten um und verifiziere jeden Schritt über `./gradlew test` bzw. `curl`:
|
||||
### 6.1 `library.yaml` (ConfigMap)
|
||||
|
||||
1. **Setup & DB:** Erstelle `build.gradle.kts`, `AppConfig.kt`, `DatabaseFactory.kt` (inkl. SQLite WAL-Pragma) und `Tables.kt`.
|
||||
2. **Scanner & Cover-Cache:** Implementiere `IdGenerator.kt`, `TagExtractor.kt` (`jaudiotagger`), `CoverArtService.kt` (`Thumbnailator`) und `MediaScanner.kt` (inkl. Entfernen gelöschter Dateien aus der DB). Führe beim Serverstart automatisch einen initialen Scan aus, falls die DB leer ist.
|
||||
3. **Ktor Core & Auth:** Konfiguriere Ktor mit `PartialContent`, `ContentNegotiation` (`ignoreUnknownKeys = true`, `encodeDefaults = true`), `SubsonicAuth` und dem `SubsonicResponse`-Wrapper.
|
||||
4. **Endpoints (Browsing + Media):** Implementiere System-, Browsing- und Media-Routes (`stream`, `download`, `getCoverArt`).
|
||||
5. **Endpoints (Sync + Interaktion + Playlists):** Implementiere `getAlbumList2`, `search3`, `scrobble`, `star`/`unstar`, `setRating` und alle Playlist-Endpunkte.
|
||||
6. **Containerisierung:** Erstelle ein Multi-Stage `Dockerfile` (Gradle Build -> `eclipse-temurin:21-jre-alpine` Runtime mit `-XX:+UseSerialGC -Xmx256m`) sowie eine `docker-compose.yml` mit Volumes für `/music:ro` und `/data`.
|
||||
7. **Automatisierte Tests:** Schreibe Ktor-Integrationstests (`ktor-server-test-host`) für Auth-Hashing, deterministische IDs, `Range`-Header (`206 Partial Content`) beim Streaming und den kompletten Browsing-Flow (`getArtists` -> `getArtist` -> `getAlbum` -> `stream`).
|
||||
```yaml
|
||||
library:
|
||||
folders:
|
||||
- id: 1
|
||||
name: "Alice — Music"
|
||||
type: music
|
||||
path: /mnt/media/music/alice
|
||||
- id: 2
|
||||
name: "Bob — Music"
|
||||
type: music
|
||||
path: /mnt/media/music/bob
|
||||
- id: 10
|
||||
name: "Audiobooks & Podcasts"
|
||||
type: longform
|
||||
path: /mnt/media/longform
|
||||
```
|
||||
|
||||
### 6.2 `users.yaml` (Vault-managed Secret)
|
||||
|
||||
Mounted from a HashiCorp Vault secret and read at startup. Passwords exist only in that secret and in memory — never in SQLite, logs or the Helm chart.
|
||||
|
||||
```yaml
|
||||
users:
|
||||
- name: alice
|
||||
password: "<from-vault>"
|
||||
admin: true
|
||||
folders: [1]
|
||||
- name: bob
|
||||
password: "<from-vault>"
|
||||
admin: false
|
||||
folders: [2]
|
||||
```
|
||||
|
||||
- Long-form folders are shared automatically; users only list their music folders.
|
||||
- Env override: `USERS_FILE` (default `./config/users.yaml`; in the container `/etc/musicmanager/users.yaml`).
|
||||
- Env override: `LIBRARY_CONFIG` (default `./config/library.yaml`; in the container `/etc/musicmanager/library.yaml`).
|
||||
- `users` and `music_folders` tables are reconciled on startup; removed users/folders are deleted with their per-user state or orphaned songs.
|
||||
- If `users.yaml` is missing or empty, the server refuses to start (fail closed).
|
||||
|
||||
### 6.3 Environment variables
|
||||
|
||||
| Variable | Default | Purpose |
|
||||
| --- | --- | --- |
|
||||
| `PORT` | `8080` | HTTP listen port |
|
||||
| `DATA_DIR` | `./data` | SQLite DB and cover cache (`/data` in Kubernetes) |
|
||||
| `LIBRARY_CONFIG` | `./config/library.yaml` | Library folder definitions |
|
||||
| `USERS_FILE` | `./config/users.yaml` | Vault-managed users file |
|
||||
| `FFMPEG_PATH` | `ffmpeg` | ffmpeg binary used for transcoding |
|
||||
|
||||
### 6.4 Local development
|
||||
|
||||
```bash
|
||||
cp config/library.example.yaml config/library.yaml
|
||||
cp config/users.example.yaml config/users.yaml
|
||||
mkdir -p media/music media/longform
|
||||
# edit the paths in config/library.yaml to point at ./media/*
|
||||
./gradlew run
|
||||
```
|
||||
|
||||
The database and covers are written to `./data`. `config/users.yaml` is git-ignored.
|
||||
|
||||
---
|
||||
|
||||
## 7. MVP Endpoint Specification
|
||||
|
||||
All endpoints live under `/rest/{endpoint}` and `/rest/{endpoint}.view`.
|
||||
|
||||
### 7.1 Authentication (`SubsonicAuth`)
|
||||
|
||||
For every `/rest/*` call (except an optional health check):
|
||||
|
||||
- `u` must match a configured user.
|
||||
- Token auth: `t` and `s` present → verify `t.equals(md5(password + s), ignoreCase = true)`.
|
||||
- Password auth: `p` present → if `p` starts with `enc:`, hex-decode it; compare to the password.
|
||||
- Comparisons use constant-time equality.
|
||||
- On failure: HTTP 200 with a Subsonic error (`code = 40`).
|
||||
- The authenticated user is attached to the call and used for all per-user filtering and state.
|
||||
|
||||
### 7.2 System & scan
|
||||
|
||||
- `ping`: empty `subsonic-response` with `status: "ok"`.
|
||||
- `getLicense`: `{"license": {"valid": true}}`.
|
||||
- `getOpenSubsonicExtensions`: `[{formPost, [1]}, {songLyrics, [1]}, {transcodeOffset, [1]}]`.
|
||||
- `startScan`: admin only; starts an async coroutine scan.
|
||||
- `getScanStatus`: `{"scanStatus": {"scanning": Boolean, "count": Long}}`.
|
||||
|
||||
### 7.3 Browsing & search
|
||||
|
||||
- `getMusicFolders`: the caller's music folders plus shared long-form folders.
|
||||
- `getIndexes`: alphabetically grouped artists visible to the caller (`index: [{name, artist[]}]`), including `lastModified` and `ignoredArticles`.
|
||||
- `getArtists`: ID3-based alphabet structure (`artists: {ignoredArticles, index[]}`), filtered by visibility.
|
||||
- `getArtist(id)`: artist details plus all visible albums.
|
||||
- `getAlbum(id)`: album details plus visible songs, sorted by `discNumber`, `trackNumber`.
|
||||
- `getSong(id)`: single song, only if visible to the caller.
|
||||
- `search3(query, …Count, …Offset)`: case-insensitive substring search over normalized `search_text` using `LIKE` with escaped wildcards. Empty `query` returns paginated results; `query="*"` returns everything visible.
|
||||
- `getGenres`: distinct genres from visible songs.
|
||||
- `getSongsByGenre(genre, count, offset)`: paginated visible songs in a genre.
|
||||
- `getStarred2`: the caller's starred artists, albums and songs for the home screen.
|
||||
|
||||
### 7.4 Home feeds
|
||||
|
||||
- `getAlbumList2(type, size, offset, fromYear, toYear, genre)`: `newest`, `recent`, `frequent`, `random`, `alphabeticalByName`, `alphabeticalByArtist`, `starred`, `byYear`, `byGenre`. All results are visibility- and user-scoped.
|
||||
- `getRandomSongs(size, genre, fromYear, toYear)`: random visible songs.
|
||||
|
||||
### 7.5 Media & artwork
|
||||
|
||||
- `stream(id, maxBitRate, format, timeOffset)`:
|
||||
- direct play via `call.respondFile` (PartialContent handles `Range`);
|
||||
- otherwise stream ffmpeg MP3 output with chunked transfer;
|
||||
- set `Content-Type` from the actual response.
|
||||
- `download(id)`: original file with `Content-Disposition: attachment; filename*=UTF-8''…` (RFC 5987 for non-ASCII names).
|
||||
- `getCoverArt(id, size)`: serve `DATA_DIR/covers/{albumId}.jpg`; if `size` is given, resize proportionally with Thumbnailator and cache as `{albumId}_{size}.jpg`. Validate `size` (`1..2048`).
|
||||
- `getLyricsBySongId(id)`: read embedded ID3 lyrics or a sibling `.lrc` file; return the OpenSubsonic `lyricsList` format.
|
||||
|
||||
### 7.6 Interaction, bookmarks & playlists
|
||||
|
||||
- `scrobble(id, time, submission)`: per user; `submission=true` (default) increments `play_count` and sets `last_played_at`.
|
||||
- `star(id, albumId, artistId)` / `unstar(...)`: per user.
|
||||
- `setRating(id, rating)`: per user, 1–5; `0` clears.
|
||||
- `getBookmarks` / `createBookmark(id, position)` / `deleteBookmark(id)`: per user; `position` is milliseconds.
|
||||
- `getPlaylists`: the caller's playlists plus all public ones.
|
||||
- `getPlaylist(id)`: if public or owned by the caller.
|
||||
- `createPlaylist(playlistId, name, songId)`: owner is the caller.
|
||||
- `updatePlaylist(playlistId, name, comment, public, songIdToAdd, songIndexToRemove)`: owner only; indices are applied in order.
|
||||
- `deletePlaylist(playlistId)`: owner only.
|
||||
|
||||
No `createUser` / `updateUser` / `deleteUser` endpoints: config/Vault is the source of truth.
|
||||
|
||||
---
|
||||
|
||||
## 8. Scanner
|
||||
|
||||
- Scan all configured folder roots recursively; accept `.mp3` (case-insensitive) only, log every skipped file.
|
||||
- Tag-first: title, artist, album artist, album, track/disc, year, genre(s), duration, bitrate, embedded cover, lyrics.
|
||||
- Fallback: parent folder as album, grandparent as artist; filename as title.
|
||||
- Delta scan by `(file_mtime, size)`; unchanged files are skipped.
|
||||
- Orphan cleanup: delete songs (and their state) whose files disappeared.
|
||||
- Initial scan on startup only when the `songs` table is empty; otherwise rely on `startScan`.
|
||||
- One scan at a time; a second `startScan` returns the current status.
|
||||
- All writes in a single transaction per batch; streaming reads continue under WAL.
|
||||
- Album/artist rows are recomputed after each scan (aggregated genre lists, sort names, cover references).
|
||||
|
||||
Cover extraction order: embedded APIC frame → `cover.jpg`/`folder.jpg`/`front.jpg` next to the file → album directory → none.
|
||||
|
||||
---
|
||||
|
||||
## 9. Deployment (k3s + Helm)
|
||||
|
||||
### 9.1 Image
|
||||
|
||||
- Multi-stage Dockerfile: Gradle build → `eclipse-temurin:21-jre-alpine` with `ffmpeg` installed.
|
||||
- Runtime flags: `-XX:+UseSerialGC -Xmx256m` (adjustable via Helm).
|
||||
- Non-root user (UID 10001), read-only root filesystem, `/tmp` as `emptyDir`, `/data` as PVC.
|
||||
- Image: `gitea.tap-tap.win/tap-tap/audiomanager:<tag>` (registry paths must be lowercase).
|
||||
- Gitea Actions builds and pushes on tags; `imagePullSecret` is provided by the cluster.
|
||||
|
||||
### 9.2 Helm chart
|
||||
|
||||
`charts/musicmanager` with values for:
|
||||
|
||||
- image repository/tag, `imagePullSecrets`
|
||||
- `library` config (folders) rendered into a ConfigMap
|
||||
- `users.existingSecret` + key (Vault-managed; never put passwords in values)
|
||||
- ingress host `subsonic.tap-tap.win`, ingress class, TLS via cluster issuer
|
||||
- `/data` PVC (size, storage class)
|
||||
- `hostPath` mounts: `/mnt/media/music` and `/mnt/media/longform`, read-only
|
||||
- no node pinning: every schedulable node must expose the same mounts (documented)
|
||||
- resource requests/limits, security context, liveness/readiness probes (`/health`)
|
||||
- optional `nodeSelector`/`tolerations` for future use
|
||||
|
||||
The server fails its readiness probe if a configured root path is missing or unreadable.
|
||||
|
||||
### 9.3 Operations
|
||||
|
||||
- Backups: documented manual copy of `DATA_DIR` (SQLite DB + covers) while the server is stopped, or via `sqlite3 .backup`.
|
||||
- Logs to stdout (JSON optional), picked up by the cluster.
|
||||
- Graceful shutdown: stop accepting requests, finish in-flight streams, close the DB pool.
|
||||
|
||||
---
|
||||
|
||||
## 10. Testing
|
||||
|
||||
- No audio files are committed to the repository. A local script generates tiny silent MP3 fixtures (and `.lrc` files) under `build/test-media`; integration tests skip if fixtures are absent.
|
||||
- Ktor integration tests (`ktor-server-test-host`):
|
||||
- token and password auth, `enc:` decoding, wrong credentials → code 40
|
||||
- deterministic IDs and normalization
|
||||
- user visibility: user A cannot see or stream user B's music; both see long-form
|
||||
- `Range` request → `206 Partial Content` with exact bytes
|
||||
- exact `size` / `duration` / `bitRate` / `suffix` / `contentType` fields
|
||||
- browsing flow: `getArtists` → `getArtist` → `getAlbum` → `stream`
|
||||
- `search3` with Unicode and wildcards
|
||||
- scrobble/star/rating/bookmark isolation between users
|
||||
- playlist ownership and public visibility
|
||||
- transcoding: `maxBitRate` produces MP3, `timeOffset` seeks (skipped when `ffmpeg` is unavailable)
|
||||
|
||||
---
|
||||
|
||||
## 11. Implementation Order
|
||||
|
||||
Verify each step with `./gradlew test` and/or `curl`:
|
||||
|
||||
- [x] 1. **Setup & DB:** `build.gradle.kts`, `AppConfig.kt`, `LibraryConfig.kt`, `UsersConfig.kt`, `DatabaseFactory.kt` (WAL + Flyway), `Tables.kt`, `V1__init.sql`.
|
||||
- [x] 2. **Scanner & covers:** `IdGenerator.kt`, `Mp3TagExtractor.kt`, `CoverArtService.kt`, `MediaScanner.kt` (delta + orphan cleanup + startup scan when empty).
|
||||
- [x] 3. **Ktor core & auth:** `PartialContent`, content negotiation (`ignoreUnknownKeys`, `encodeDefaults`), `SubsonicAuth`, response wrapper, health endpoint.
|
||||
- [ ] 4. **Browsing & media:** system, browsing, genre and media routes (`stream`, `download`, `getCoverArt`).
|
||||
- [ ] 5. **Transcoding:** `FfmpegTranscoder.kt`, `maxBitRate`/`format`/`timeOffset`, extension declaration.
|
||||
- [ ] 6. **Interaction & playlists:** `getAlbumList2`, `getRandomSongs`, `getStarred2`, `search3`, `scrobble`, `star`/`unstar`, `setRating`, bookmarks, playlists.
|
||||
- [ ] 7. **Packaging:** multi-stage Dockerfile (with ffmpeg), Helm chart, Gitea Actions workflow.
|
||||
- [ ] 8. **Tests:** integration suite as described above, plus a `curl` smoke script.
|
||||
|
||||
---
|
||||
|
||||
## 12. Open Items / Assumptions
|
||||
|
||||
1. **Gitea registry path:** image assumed to be `gitea.tap-tap.win/tap-tap/audiomanager` (lowercased repo `TapTap/AudioManager`).
|
||||
2. **Vault layout:** the chart expects a secret with a `users.yaml` key. Exact Vault path/agent template is deployment-side and not part of the app.
|
||||
3. **Per-user music roots:** Vault/config must contain one folder entry per user root (e.g. `/mnt/media/music/<user>`); there is no implicit path convention.
|
||||
4. **hostPath without pinning:** requires `/mnt/media/music` and `/mnt/media/longform` to exist on every node that can schedule the pod. Add a `nodeSelector` later if that changes.
|
||||
5. **Long-form modelling:** audiobooks/podcasts are albums/tracks in long-form folders; no RSS ingestion or chapter model.
|
||||
6. **No XML:** clients that insist on XML are unsupported.
|
||||
7. **No user-management API:** user changes are applied on restart after the Vault secret/config is updated.
|
||||
8. **Backups** are manual and documented; no backup endpoint.
|
||||
Reference in new issue
Block a user