feat: add Deezer as the default download source
- V9 adds source/external_id to songs, moods, recommendations and jobs - Deezer public catalog for search, related album covers and metadata - streamrip downloader configured with DEEZER_ARL, 128/320 MP3 - per-mood source selector; switching clears and refills recommendations - keep YouTube Music as a full second catalog and downloader - media layout <source>-<externalId>, legacy YouTube folders still read - Docker/streamrip install, chart ARL secret, docs and dual-source smoke test
This commit is contained in:
1 parent
81a47f4cd2
commit
141698282d
43 files changed
+2072
-605
No files matched your search
+65
-44
@@ -13,10 +13,10 @@ Repos:
|
||||
|
||||
- Users have **moods only**. There is no mounted music directory, no file
|
||||
upload and no longform/podcast content.
|
||||
- All content enters the server by downloading from YouTube Music through
|
||||
`yt-dlp` (regular YouTube is only a fallback host).
|
||||
- Search is a remote YouTube Music search. Results are shown before
|
||||
anything is added.
|
||||
- Content enters the server by downloading from **Deezer** (default) or
|
||||
**YouTube Music** through `streamrip` / `yt-dlp`. Each mood picks one source.
|
||||
- Search is a remote catalog search against the mood's source. Results are
|
||||
shown before anything is added.
|
||||
- A mood contains:
|
||||
- **Songs** — the user's collection. Added from search, from a recommendation,
|
||||
or by attaching an already-downloaded song.
|
||||
@@ -40,7 +40,7 @@ Repos:
|
||||
It never replaces a ready recommendation the user has not acted on.
|
||||
- Removing a song only detaches it from the mood. Files are deleted only when
|
||||
no mood references them.
|
||||
- Songs are stored once server-wide, keyed by YouTube video id and
|
||||
- Songs are stored once server-wide, keyed by `(source, externalId)` and
|
||||
deduplicated. Moods are strictly per-user: a user never sees another user's
|
||||
moods or collection, but the same file can serve several users.
|
||||
- No per-song owner, no storage quota.
|
||||
@@ -75,7 +75,8 @@ Repos:
|
||||
13. Chart uses a PVC for `/data` (media at `/data/media`); music hostPath,
|
||||
`library.yaml`, user folder lists, `DOWNLOADS_ENABLED` and
|
||||
`DOWNLOAD_FOLDER_ID` disappear.
|
||||
14. One song per YouTube video id; the same track from two videos is two songs.
|
||||
14. One song per `(source, externalId)`; the same recording from two catalogs is
|
||||
two songs. Deezer track ids and YouTube video ids do not overlap.
|
||||
Manual downloads outrank automatic recommendation downloads. Detach only;
|
||||
GC deletes unreferenced files. A removed playing track finishes and drops
|
||||
from future queues.
|
||||
@@ -107,7 +108,8 @@ Current files and responsibilities, and what happens to them:
|
||||
| `discovery/DiscoveryService.kt` | seeds + genres/query -> candidates, review | same inputs from collection; store `new` candidates; no `matchedSongId` library matching |
|
||||
| `discovery/YtSearch.kt` | yt-dlp flat search | YouTube Music songs search with resolved metadata; regular `ytsearch` fallback; add URL resolve |
|
||||
| `download/DownloadService.kt` | accept -> import folder -> scan -> seed | index directly into media store; attach song/recommendation; GC on failure |
|
||||
| `download/YtDlpDownloader.kt` | MP3 + embedded metadata | keep; target `<videoId>/<title>.mp3` |
|
||||
| `download/YtDlpDownloader.kt` | MP3 + embedded metadata | keep; target `<source>-<externalId>/<title>.mp3` |
|
||||
| `download/DeezerDownloader.kt` | - | streamrip with `DEEZER_ARL`; 128/320 MP3 |
|
||||
| `scanner/MediaScanner.kt` | walks folder config | walks `MEDIA_DIR`; derive video id from path |
|
||||
| `scanner/Mp3TagExtractor.kt` | tags/cover/lyrics | keep tags+cover; drop album/artist IDs and lyrics use |
|
||||
| `scanner/CoverArtService.kt` | album-keyed covers | song-keyed covers |
|
||||
@@ -134,14 +136,15 @@ cases (auth, ownership, streaming, transcription, permissions).
|
||||
`/data`, `DATA_DIR=/data`, `MEDIA_DIR=/data/media`).
|
||||
- Download temp/work dir: `<data>/downloads` or `<media>/.tmp`; must be on the
|
||||
same filesystem as the final path for atomic moves.
|
||||
- Layout: `<media>/<videoId>/<sanitized-title>.mp3` plus
|
||||
- Layout: `<media>/<source>-<externalId>/<sanitized-title>.mp3` plus
|
||||
`<media>/cover-cache/...` or a song-keyed cover derived from embedded art.
|
||||
The first path segment is the YouTube video id, which makes startup scans
|
||||
The first path segment carries the source and external id, which makes startup scans
|
||||
deterministic and independent of tags.
|
||||
- Song id: `sg-` + SHA-256 prefix of `"yt:<videoId>"`.
|
||||
- Song id: `sg-` + SHA-256 prefix of `"yt:<videoId>"` for YouTube (stable legacy
|
||||
ids) or `"dz:<trackId>"` for Deezer.
|
||||
- MP3 only (`yt-dlp -x --audio-format mp3 --audio-quality 0`), embedded
|
||||
metadata and thumbnail.
|
||||
- The scanner walks `MEDIA_DIR`, extracts `videoId` from the top-level folder,
|
||||
- The scanner walks `MEDIA_DIR`, extracts `(source, externalId)` from the top-level folder,
|
||||
reads tags/cover, and upserts the song. The downloader can call the same
|
||||
indexer for one file to avoid waiting for a full scan.
|
||||
- Orphan cleanup keeps the old safety guard: never wipe when the media root is
|
||||
@@ -154,7 +157,8 @@ Keep `users` and `api_tokens`. Drop and recreate everything else.
|
||||
```sql
|
||||
songs(
|
||||
id TEXT PRIMARY KEY,
|
||||
video_id TEXT NOT NULL UNIQUE,
|
||||
source TEXT NOT NULL DEFAULT 'youtube',
|
||||
external_id TEXT NOT NULL UNIQUE,
|
||||
title TEXT NOT NULL,
|
||||
artist TEXT NOT NULL,
|
||||
album TEXT,
|
||||
@@ -198,7 +202,8 @@ mood_songs(
|
||||
|
||||
mood_recommendations(
|
||||
mood_id TEXT NOT NULL REFERENCES moods(id) ON DELETE CASCADE,
|
||||
video_id TEXT NOT NULL,
|
||||
source TEXT NOT NULL DEFAULT 'youtube',
|
||||
external_id TEXT NOT NULL,
|
||||
title TEXT NOT NULL,
|
||||
artist TEXT NOT NULL,
|
||||
album TEXT,
|
||||
@@ -211,12 +216,13 @@ mood_recommendations(
|
||||
error TEXT,
|
||||
first_seen TEXT NOT NULL,
|
||||
last_seen TEXT NOT NULL,
|
||||
PRIMARY KEY (mood_id, video_id)
|
||||
PRIMARY KEY (mood_id, external_id)
|
||||
)
|
||||
|
||||
download_jobs(
|
||||
id TEXT PRIMARY KEY,
|
||||
video_id TEXT NOT NULL,
|
||||
source TEXT NOT NULL DEFAULT 'youtube',
|
||||
external_id TEXT NOT NULL,
|
||||
mood_id TEXT NOT NULL REFERENCES moods(id) ON DELETE CASCADE,
|
||||
kind TEXT NOT NULL CHECK (kind IN ('song', 'recommendation')),
|
||||
state TEXT NOT NULL DEFAULT 'queued'
|
||||
@@ -279,7 +285,7 @@ States: `new` (hidden candidate pool), `downloading`, `ready`, `failed`,
|
||||
|
||||
- `download_jobs` are global; `kind=song` (manual add) has higher priority than
|
||||
`kind=recommendation`. Claim by `priority DESC, created_at ASC`.
|
||||
- Before downloading, check `songs.video_id`:
|
||||
- Before downloading, check `songs(source, external_id)`:
|
||||
- if the song exists and the file is present, attach immediately;
|
||||
- otherwise download.
|
||||
- A per-video download is effectively deduplicated by the existence check; two
|
||||
@@ -287,9 +293,9 @@ States: `new` (hidden candidate pool), `downloading`, `ready`, `failed`,
|
||||
completes instantly.
|
||||
- Worker steps:
|
||||
1. Claim a job.
|
||||
2. Resolve target `<media>/<videoId>/<title>.mp3`.
|
||||
2. Resolve target `<media>/<source>-<externalId>/<title>.mp3`.
|
||||
3. If the song exists, finish.
|
||||
4. Download to temp with yt-dlp; move atomically into the target dir; index
|
||||
4. Download to temp with yt-dlp/streamrip; move atomically; index
|
||||
the file (tags/cover) and upsert the song by video id.
|
||||
5. Apply the job outcome (song attach or recommendation state).
|
||||
6. On permanent failure, remove the partial file and set `failed`.
|
||||
@@ -323,7 +329,7 @@ A song is referenced by:
|
||||
`downloading|ready|added|failed` (`added` also has `mood_songs`);
|
||||
- an open playback session item (avoid deleting a track currently streaming).
|
||||
|
||||
When the reference count is zero, delete the song row, the `<videoId>` media
|
||||
When the reference count is zero, delete the song row, the media
|
||||
directory and the cover. Run GC:
|
||||
- after removing a song;
|
||||
- after rejecting a recommendation;
|
||||
@@ -333,15 +339,15 @@ directory and the cover. Run GC:
|
||||
|
||||
### 4.7 Search and URL adds
|
||||
|
||||
- `GET /api/v1/search?q=...&limit=...` -> YouTube Music song results with
|
||||
`videoId,title,artist,album,duration,thumbnail,songId?,downloaded,inMood`.
|
||||
- Hide results with `duration > 600` seconds (and probably `< 30`).
|
||||
- `POST /api/v1/moods/{id}/songs` accepts either `songId` (existing download)
|
||||
or download metadata, or `url`. A URL is resolved with yt-dlp
|
||||
(`--dump-single-json --no-playlist`) before enqueueing.
|
||||
- Search requires auth; it is a server-side yt-dlp call that resolves every
|
||||
Music result, so use a 60s client timeout and return 502 with the yt-dlp
|
||||
error on failure.
|
||||
- `GET /api/v1/search?q=...&limit=...&source=...&moodId=...` -> catalog results
|
||||
with `source,externalId,title,artist,album,duration,thumbnail,songId?,downloaded,inMood`.
|
||||
- `source` falls back to the mood's source, then `DEFAULT_SOURCE`.
|
||||
- Hide results with `duration > 600` seconds and `< 30`.
|
||||
- `POST /api/v1/moods/{id}/songs` accepts `songId` (existing download),
|
||||
`source`+`externalId`, or `url`. Deezer URLs use the public API; YouTube URLs
|
||||
use yt-dlp `--dump-single-json --no-playlist`.
|
||||
- Search requires auth; Deezer is a fast public API call, while YouTube Music
|
||||
resolves every hit (use a 60s client timeout). Return 502 on failure.
|
||||
|
||||
### 4.8 API surface
|
||||
|
||||
@@ -357,9 +363,9 @@ DELETE /api/v1/moods/{id}
|
||||
GET /api/v1/moods/{id}/preview
|
||||
GET /api/v1/moods/{id}/recommendations
|
||||
POST /api/v1/moods/{id}/recommendations/refresh
|
||||
POST /api/v1/moods/{id}/recommendations/{videoId}/add
|
||||
POST /api/v1/moods/{id}/recommendations/{videoId}/reject
|
||||
POST /api/v1/moods/{id}/recommendations/{videoId}/retry
|
||||
POST /api/v1/moods/{id}/recommendations/{externalId}/add
|
||||
POST /api/v1/moods/{id}/recommendations/{externalId}/reject
|
||||
POST /api/v1/moods/{id}/recommendations/{externalId}/retry
|
||||
POST /api/v1/moods/{id}/songs
|
||||
DELETE /api/v1/moods/{id}/songs/{songId}
|
||||
POST /api/v1/moods/{id}/sessions
|
||||
@@ -378,7 +384,7 @@ DTO sketch:
|
||||
|
||||
```text
|
||||
UserDto(name, admin)
|
||||
SongDto(id, videoId, title, artist, album, duration, bitRate, size, suffix,
|
||||
SongDto(id, source, externalId, title, artist, album, duration, bitRate, size, suffix,
|
||||
thumbnail)
|
||||
MoodDto(id, name, description, query, genres, collectionPercent,
|
||||
recommendationPercent, recommendationCount, refreshHours,
|
||||
@@ -386,9 +392,9 @@ MoodDto(id, name, description, query, genres, collectionPercent,
|
||||
recommendationReady, recommendationDownloading, recommendationFailed)
|
||||
MoodSongDto(song, origin, addedAt)
|
||||
MoodDetailDto(mood, songs, recommendations, downloads)
|
||||
RecommendationDto(videoId, title, artist, album, duration, thumbnail, state,
|
||||
RecommendationDto(source, externalId, title, artist, album, duration, thumbnail, state,
|
||||
song?, download?, firstSeen, lastSeen)
|
||||
SearchResultDto(videoId, title, artist, album, duration, thumbnail, songId?,
|
||||
SearchResultDto(source, externalId, title, artist, album, duration, thumbnail, songId?,
|
||||
downloaded)
|
||||
```
|
||||
|
||||
@@ -485,7 +491,7 @@ Offline playback uses device downloads; online mood playback uses sessions.
|
||||
- keep users Secret, probes, resources; add persistence values;
|
||||
- README deployment section rewritten.
|
||||
- Nix shell: no library config; `DATA_DIR=./data`.
|
||||
- Docker: no extra packages; `yt-dlp` and `ffmpeg` already present.
|
||||
- Docker: `yt-dlp`, `ffmpeg`, Python 3 and `streamrip` (provides `rip`).
|
||||
- Smoke test: rewrite for mood creation, search stub or seeded songs, add,
|
||||
recommendations, playback. A fake `yt-dlp` script can return deterministic
|
||||
search JSON and a tiny MP3 fixture for downloads.
|
||||
@@ -603,7 +609,7 @@ Client:
|
||||
duplicate collection row.
|
||||
- The migration is destructive by design; document it prominently.
|
||||
|
||||
## Implementation status (2026-11-15)
|
||||
## Implementation status (2026-11-17)
|
||||
|
||||
Phases 1-8 are implemented in both repositories.
|
||||
|
||||
@@ -614,9 +620,10 @@ Phases 1-8 are implemented in both repositories.
|
||||
playback tables and `song_affinity`. Users and device tokens survive.
|
||||
The `mood_recommendations` table has an additional `rank` column so
|
||||
promotion follows the discovery score order.
|
||||
- `SongRepository` dedupes by video id and is the only song table access.
|
||||
- `MediaStore` owns `<MEDIA_DIR>/<videoId>/<title>.mp3`; `SongIndexer` is shared
|
||||
by the scanner and the downloader.
|
||||
- `SongRepository` dedupes by `(source, externalId)` and is the only song table access.
|
||||
- `MediaStore` owns `<MEDIA_DIR>/<source>-<externalId>/<title>.mp3` (legacy
|
||||
YouTube folders without a prefix are still read); `SongIndexer` is shared by
|
||||
the scanner and the downloader.
|
||||
- `MoodRepository` implements the song collection, hidden `new` candidates,
|
||||
active slots and the add/reject/retry lifecycle.
|
||||
- `DiscoveryService` + `TopUpService` implement collection-based discovery and
|
||||
@@ -631,7 +638,9 @@ Phases 1-8 are implemented in both repositories.
|
||||
covers use `/songs/{id}/cover`. `/moods/{id}/songs` returns `202` with
|
||||
`downloading: true` when a new video was queued.
|
||||
- `AppConfig` is `port`, `dataDir`, `usersFile`, `ffmpegPath`, `ytDlpPath`,
|
||||
`mediaDir`, `downloadDir`. `LibraryConfig` and the folder APIs are deleted.
|
||||
`streamripPath`, `defaultSource`, `deezerArl`, `deezerQuality`,
|
||||
`deezerApiBase`, `mediaDir`, `downloadDir`. `LibraryConfig` and the folder
|
||||
APIs are deleted.
|
||||
- `YtDlpSearch` searches `music.youtube.com` with the Songs filter and resolves
|
||||
each hit, so results carry the canonical artist, album and duration.
|
||||
`YtDlpDownloader` requests the Music watch URL first and falls back to
|
||||
@@ -641,19 +650,29 @@ Phases 1-8 are implemented in both repositories.
|
||||
(hourly), media GC (hourly), startup scan.
|
||||
- Chart 0.3.0 uses one `/data` PVC, `MEDIA_DIR=/data/media`, no hostPath, no
|
||||
library ConfigMap and no init container.
|
||||
- Tests: 75 server tests, 48-step smoke test with a fake `yt-dlp`.
|
||||
- V9 adds `source` to songs, moods, recommendations and download jobs and
|
||||
renames `video_id` to `external_id`; existing rows are YouTube.
|
||||
- `MusicCatalogs` dispatches search/resolve by source (`DeezerSearch` +
|
||||
`YtDlpSearch`). `SourceDownloaders` dispatches downloads (`DeezerDownloader`
|
||||
via streamrip + `YtDlpDownloader`).
|
||||
- Deezer is the default for new moods; every mood stores its own source and
|
||||
can switch it, which clears the active recommendation pool and refills it.
|
||||
- Chart 0.4.0 adds `DEFAULT_SOURCE`, `DEEZER_*` and an optional ARL secret.
|
||||
- Tests: 81 server tests, 64-step smoke test with fake `yt-dlp`, `rip` and a
|
||||
fake Deezer API.
|
||||
|
||||
### Client
|
||||
|
||||
- Models, `ApiClient`, `AppCache` (version 2) and `AppStore` match the new API.
|
||||
- Models, `ApiClient`, `AppCache` (version 3) and `AppStore` match the new API.
|
||||
- Screens are `Moods | MoodDetail | Search | Downloads | Settings`.
|
||||
`LibraryScreen`, longform UI, seed management and global shuffle are gone.
|
||||
- Search supports a target-mood picker, downloaded badges and paste-a-URL.
|
||||
- Search supports a target-mood picker, a **Deezer / YouTube Music** source
|
||||
selector, downloaded badges and paste-a-URL.
|
||||
- Mood detail shows songs, add music, recommendation states (Add / Not
|
||||
interested / Retry), next batch and the new options.
|
||||
- Playback, sleep timer, queue editing, Android media session and device
|
||||
downloads are unchanged apart from song-keyed covers.
|
||||
- Tests: 16 client tests; desktop, Android debug and Wasm compile.
|
||||
- Tests: 17 client tests; desktop, Android debug and Wasm compile.
|
||||
|
||||
### Known follow-ups
|
||||
|
||||
@@ -662,3 +681,5 @@ Phases 1-8 are implemented in both repositories.
|
||||
- Automatic downloads are bounded by `recommendationCount` but can still consume
|
||||
noticeable disk/bandwidth for unplayed moods.
|
||||
- yt-dlp bot checks remain an operational risk; failed rows keep Retry.
|
||||
- A Deezer Free ARL is limited to 128 kbps and ARL downloads are outside the
|
||||
public API terms. A Premium ARL enables 320 kbps MP3 (`DEEZER_QUALITY=1`).
|
||||
Reference in new issue
Block a user