feat: add Deezer as the default download source
CI / chart (push) Successful in 3s
CI / test (push) Successful in 2m23s
CI / chart-push (push) Successful in 5s
CI / image (push) Successful in 3m50s

- V9 adds source/external_id to songs, moods, recommendations and jobs
- Deezer public catalog for search, related album covers and metadata
- streamrip downloader configured with DEEZER_ARL, 128/320 MP3
- per-mood source selector; switching clears and refills recommendations
- keep YouTube Music as a full second catalog and downloader
- media layout <source>-<externalId>, legacy YouTube folders still read
- Docker/streamrip install, chart ARL secret, docs and dual-source smoke test
This commit is contained in:
TapTap committed 2026-11-17 07:32:34 +00:00
1 parent 81a47f4cd2
commit 141698282d
43 files changed
+2072 -605

No files matched your search

+65 -44
View File
@@ -13,10 +13,10 @@ Repos:
- Users have **moods only**. There is no mounted music directory, no file
upload and no longform/podcast content.
- All content enters the server by downloading from YouTube Music through
`yt-dlp` (regular YouTube is only a fallback host).
- Search is a remote YouTube Music search. Results are shown before
anything is added.
- Content enters the server by downloading from **Deezer** (default) or
**YouTube Music** through `streamrip` / `yt-dlp`. Each mood picks one source.
- Search is a remote catalog search against the mood's source. Results are
shown before anything is added.
- A mood contains:
- **Songs** — the user's collection. Added from search, from a recommendation,
or by attaching an already-downloaded song.
@@ -40,7 +40,7 @@ Repos:
It never replaces a ready recommendation the user has not acted on.
- Removing a song only detaches it from the mood. Files are deleted only when
no mood references them.
- Songs are stored once server-wide, keyed by YouTube video id and
- Songs are stored once server-wide, keyed by `(source, externalId)` and
deduplicated. Moods are strictly per-user: a user never sees another user's
moods or collection, but the same file can serve several users.
- No per-song owner, no storage quota.
@@ -75,7 +75,8 @@ Repos:
13. Chart uses a PVC for `/data` (media at `/data/media`); music hostPath,
`library.yaml`, user folder lists, `DOWNLOADS_ENABLED` and
`DOWNLOAD_FOLDER_ID` disappear.
14. One song per YouTube video id; the same track from two videos is two songs.
14. One song per `(source, externalId)`; the same recording from two catalogs is
two songs. Deezer track ids and YouTube video ids do not overlap.
Manual downloads outrank automatic recommendation downloads. Detach only;
GC deletes unreferenced files. A removed playing track finishes and drops
from future queues.
@@ -107,7 +108,8 @@ Current files and responsibilities, and what happens to them:
| `discovery/DiscoveryService.kt` | seeds + genres/query -> candidates, review | same inputs from collection; store `new` candidates; no `matchedSongId` library matching |
| `discovery/YtSearch.kt` | yt-dlp flat search | YouTube Music songs search with resolved metadata; regular `ytsearch` fallback; add URL resolve |
| `download/DownloadService.kt` | accept -> import folder -> scan -> seed | index directly into media store; attach song/recommendation; GC on failure |
| `download/YtDlpDownloader.kt` | MP3 + embedded metadata | keep; target `<videoId>/<title>.mp3` |
| `download/YtDlpDownloader.kt` | MP3 + embedded metadata | keep; target `<source>-<externalId>/<title>.mp3` |
| `download/DeezerDownloader.kt` | - | streamrip with `DEEZER_ARL`; 128/320 MP3 |
| `scanner/MediaScanner.kt` | walks folder config | walks `MEDIA_DIR`; derive video id from path |
| `scanner/Mp3TagExtractor.kt` | tags/cover/lyrics | keep tags+cover; drop album/artist IDs and lyrics use |
| `scanner/CoverArtService.kt` | album-keyed covers | song-keyed covers |
@@ -134,14 +136,15 @@ cases (auth, ownership, streaming, transcription, permissions).
`/data`, `DATA_DIR=/data`, `MEDIA_DIR=/data/media`).
- Download temp/work dir: `<data>/downloads` or `<media>/.tmp`; must be on the
same filesystem as the final path for atomic moves.
- Layout: `<media>/<videoId>/<sanitized-title>.mp3` plus
- Layout: `<media>/<source>-<externalId>/<sanitized-title>.mp3` plus
`<media>/cover-cache/...` or a song-keyed cover derived from embedded art.
The first path segment is the YouTube video id, which makes startup scans
The first path segment carries the source and external id, which makes startup scans
deterministic and independent of tags.
- Song id: `sg-` + SHA-256 prefix of `"yt:<videoId>"`.
- Song id: `sg-` + SHA-256 prefix of `"yt:<videoId>"` for YouTube (stable legacy
ids) or `"dz:<trackId>"` for Deezer.
- MP3 only (`yt-dlp -x --audio-format mp3 --audio-quality 0`), embedded
metadata and thumbnail.
- The scanner walks `MEDIA_DIR`, extracts `videoId` from the top-level folder,
- The scanner walks `MEDIA_DIR`, extracts `(source, externalId)` from the top-level folder,
reads tags/cover, and upserts the song. The downloader can call the same
indexer for one file to avoid waiting for a full scan.
- Orphan cleanup keeps the old safety guard: never wipe when the media root is
@@ -154,7 +157,8 @@ Keep `users` and `api_tokens`. Drop and recreate everything else.
```sql
songs(
id TEXT PRIMARY KEY,
video_id TEXT NOT NULL UNIQUE,
source TEXT NOT NULL DEFAULT 'youtube',
external_id TEXT NOT NULL UNIQUE,
title TEXT NOT NULL,
artist TEXT NOT NULL,
album TEXT,
@@ -198,7 +202,8 @@ mood_songs(
mood_recommendations(
mood_id TEXT NOT NULL REFERENCES moods(id) ON DELETE CASCADE,
video_id TEXT NOT NULL,
source TEXT NOT NULL DEFAULT 'youtube',
external_id TEXT NOT NULL,
title TEXT NOT NULL,
artist TEXT NOT NULL,
album TEXT,
@@ -211,12 +216,13 @@ mood_recommendations(
error TEXT,
first_seen TEXT NOT NULL,
last_seen TEXT NOT NULL,
PRIMARY KEY (mood_id, video_id)
PRIMARY KEY (mood_id, external_id)
)
download_jobs(
id TEXT PRIMARY KEY,
video_id TEXT NOT NULL,
source TEXT NOT NULL DEFAULT 'youtube',
external_id TEXT NOT NULL,
mood_id TEXT NOT NULL REFERENCES moods(id) ON DELETE CASCADE,
kind TEXT NOT NULL CHECK (kind IN ('song', 'recommendation')),
state TEXT NOT NULL DEFAULT 'queued'
@@ -279,7 +285,7 @@ States: `new` (hidden candidate pool), `downloading`, `ready`, `failed`,
- `download_jobs` are global; `kind=song` (manual add) has higher priority than
`kind=recommendation`. Claim by `priority DESC, created_at ASC`.
- Before downloading, check `songs.video_id`:
- Before downloading, check `songs(source, external_id)`:
- if the song exists and the file is present, attach immediately;
- otherwise download.
- A per-video download is effectively deduplicated by the existence check; two
@@ -287,9 +293,9 @@ States: `new` (hidden candidate pool), `downloading`, `ready`, `failed`,
completes instantly.
- Worker steps:
1. Claim a job.
2. Resolve target `<media>/<videoId>/<title>.mp3`.
2. Resolve target `<media>/<source>-<externalId>/<title>.mp3`.
3. If the song exists, finish.
4. Download to temp with yt-dlp; move atomically into the target dir; index
4. Download to temp with yt-dlp/streamrip; move atomically; index
the file (tags/cover) and upsert the song by video id.
5. Apply the job outcome (song attach or recommendation state).
6. On permanent failure, remove the partial file and set `failed`.
@@ -323,7 +329,7 @@ A song is referenced by:
`downloading|ready|added|failed` (`added` also has `mood_songs`);
- an open playback session item (avoid deleting a track currently streaming).
When the reference count is zero, delete the song row, the `<videoId>` media
When the reference count is zero, delete the song row, the media
directory and the cover. Run GC:
- after removing a song;
- after rejecting a recommendation;
@@ -333,15 +339,15 @@ directory and the cover. Run GC:
### 4.7 Search and URL adds
- `GET /api/v1/search?q=...&limit=...` -> YouTube Music song results with
`videoId,title,artist,album,duration,thumbnail,songId?,downloaded,inMood`.
- Hide results with `duration > 600` seconds (and probably `< 30`).
- `POST /api/v1/moods/{id}/songs` accepts either `songId` (existing download)
or download metadata, or `url`. A URL is resolved with yt-dlp
(`--dump-single-json --no-playlist`) before enqueueing.
- Search requires auth; it is a server-side yt-dlp call that resolves every
Music result, so use a 60s client timeout and return 502 with the yt-dlp
error on failure.
- `GET /api/v1/search?q=...&limit=...&source=...&moodId=...` -> catalog results
with `source,externalId,title,artist,album,duration,thumbnail,songId?,downloaded,inMood`.
- `source` falls back to the mood's source, then `DEFAULT_SOURCE`.
- Hide results with `duration > 600` seconds and `< 30`.
- `POST /api/v1/moods/{id}/songs` accepts `songId` (existing download),
`source`+`externalId`, or `url`. Deezer URLs use the public API; YouTube URLs
use yt-dlp `--dump-single-json --no-playlist`.
- Search requires auth; Deezer is a fast public API call, while YouTube Music
resolves every hit (use a 60s client timeout). Return 502 on failure.
### 4.8 API surface
@@ -357,9 +363,9 @@ DELETE /api/v1/moods/{id}
GET /api/v1/moods/{id}/preview
GET /api/v1/moods/{id}/recommendations
POST /api/v1/moods/{id}/recommendations/refresh
POST /api/v1/moods/{id}/recommendations/{videoId}/add
POST /api/v1/moods/{id}/recommendations/{videoId}/reject
POST /api/v1/moods/{id}/recommendations/{videoId}/retry
POST /api/v1/moods/{id}/recommendations/{externalId}/add
POST /api/v1/moods/{id}/recommendations/{externalId}/reject
POST /api/v1/moods/{id}/recommendations/{externalId}/retry
POST /api/v1/moods/{id}/songs
DELETE /api/v1/moods/{id}/songs/{songId}
POST /api/v1/moods/{id}/sessions
@@ -378,7 +384,7 @@ DTO sketch:
```text
UserDto(name, admin)
SongDto(id, videoId, title, artist, album, duration, bitRate, size, suffix,
SongDto(id, source, externalId, title, artist, album, duration, bitRate, size, suffix,
thumbnail)
MoodDto(id, name, description, query, genres, collectionPercent,
recommendationPercent, recommendationCount, refreshHours,
@@ -386,9 +392,9 @@ MoodDto(id, name, description, query, genres, collectionPercent,
recommendationReady, recommendationDownloading, recommendationFailed)
MoodSongDto(song, origin, addedAt)
MoodDetailDto(mood, songs, recommendations, downloads)
RecommendationDto(videoId, title, artist, album, duration, thumbnail, state,
RecommendationDto(source, externalId, title, artist, album, duration, thumbnail, state,
song?, download?, firstSeen, lastSeen)
SearchResultDto(videoId, title, artist, album, duration, thumbnail, songId?,
SearchResultDto(source, externalId, title, artist, album, duration, thumbnail, songId?,
downloaded)
```
@@ -485,7 +491,7 @@ Offline playback uses device downloads; online mood playback uses sessions.
- keep users Secret, probes, resources; add persistence values;
- README deployment section rewritten.
- Nix shell: no library config; `DATA_DIR=./data`.
- Docker: no extra packages; `yt-dlp` and `ffmpeg` already present.
- Docker: `yt-dlp`, `ffmpeg`, Python 3 and `streamrip` (provides `rip`).
- Smoke test: rewrite for mood creation, search stub or seeded songs, add,
recommendations, playback. A fake `yt-dlp` script can return deterministic
search JSON and a tiny MP3 fixture for downloads.
@@ -603,7 +609,7 @@ Client:
duplicate collection row.
- The migration is destructive by design; document it prominently.
## Implementation status (2026-11-15)
## Implementation status (2026-11-17)
Phases 1-8 are implemented in both repositories.
@@ -614,9 +620,10 @@ Phases 1-8 are implemented in both repositories.
playback tables and `song_affinity`. Users and device tokens survive.
The `mood_recommendations` table has an additional `rank` column so
promotion follows the discovery score order.
- `SongRepository` dedupes by video id and is the only song table access.
- `MediaStore` owns `<MEDIA_DIR>/<videoId>/<title>.mp3`; `SongIndexer` is shared
by the scanner and the downloader.
- `SongRepository` dedupes by `(source, externalId)` and is the only song table access.
- `MediaStore` owns `<MEDIA_DIR>/<source>-<externalId>/<title>.mp3` (legacy
YouTube folders without a prefix are still read); `SongIndexer` is shared by
the scanner and the downloader.
- `MoodRepository` implements the song collection, hidden `new` candidates,
active slots and the add/reject/retry lifecycle.
- `DiscoveryService` + `TopUpService` implement collection-based discovery and
@@ -631,7 +638,9 @@ Phases 1-8 are implemented in both repositories.
covers use `/songs/{id}/cover`. `/moods/{id}/songs` returns `202` with
`downloading: true` when a new video was queued.
- `AppConfig` is `port`, `dataDir`, `usersFile`, `ffmpegPath`, `ytDlpPath`,
`mediaDir`, `downloadDir`. `LibraryConfig` and the folder APIs are deleted.
`streamripPath`, `defaultSource`, `deezerArl`, `deezerQuality`,
`deezerApiBase`, `mediaDir`, `downloadDir`. `LibraryConfig` and the folder
APIs are deleted.
- `YtDlpSearch` searches `music.youtube.com` with the Songs filter and resolves
each hit, so results carry the canonical artist, album and duration.
`YtDlpDownloader` requests the Music watch URL first and falls back to
@@ -641,19 +650,29 @@ Phases 1-8 are implemented in both repositories.
(hourly), media GC (hourly), startup scan.
- Chart 0.3.0 uses one `/data` PVC, `MEDIA_DIR=/data/media`, no hostPath, no
library ConfigMap and no init container.
- Tests: 75 server tests, 48-step smoke test with a fake `yt-dlp`.
- V9 adds `source` to songs, moods, recommendations and download jobs and
renames `video_id` to `external_id`; existing rows are YouTube.
- `MusicCatalogs` dispatches search/resolve by source (`DeezerSearch` +
`YtDlpSearch`). `SourceDownloaders` dispatches downloads (`DeezerDownloader`
via streamrip + `YtDlpDownloader`).
- Deezer is the default for new moods; every mood stores its own source and
can switch it, which clears the active recommendation pool and refills it.
- Chart 0.4.0 adds `DEFAULT_SOURCE`, `DEEZER_*` and an optional ARL secret.
- Tests: 81 server tests, 64-step smoke test with fake `yt-dlp`, `rip` and a
fake Deezer API.
### Client
- Models, `ApiClient`, `AppCache` (version 2) and `AppStore` match the new API.
- Models, `ApiClient`, `AppCache` (version 3) and `AppStore` match the new API.
- Screens are `Moods | MoodDetail | Search | Downloads | Settings`.
`LibraryScreen`, longform UI, seed management and global shuffle are gone.
- Search supports a target-mood picker, downloaded badges and paste-a-URL.
- Search supports a target-mood picker, a **Deezer / YouTube Music** source
selector, downloaded badges and paste-a-URL.
- Mood detail shows songs, add music, recommendation states (Add / Not
interested / Retry), next batch and the new options.
- Playback, sleep timer, queue editing, Android media session and device
downloads are unchanged apart from song-keyed covers.
- Tests: 16 client tests; desktop, Android debug and Wasm compile.
- Tests: 17 client tests; desktop, Android debug and Wasm compile.
### Known follow-ups
@@ -662,3 +681,5 @@ Phases 1-8 are implemented in both repositories.
- Automatic downloads are bounded by `recommendationCount` but can still consume
noticeable disk/bandwidth for unplayed moods.
- yt-dlp bot checks remain an operational risk; failed rows keep Retry.
- A Deezer Free ARL is limited to 128 kbps and ARL downloads are outside the
public API terms. A Premium ARL enables 320 kbps MP3 (`DEEZER_QUALITY=1`).