feat: base discovery on the whole mood collection
CI / chart (push) Successful in 3s
CI / test (push) Successful in 1m56s
CI / chart-push (push) Successful in 5s
CI / image (push) Successful in 3m34s

- seed from every artist (weighted by song count) and every album, not one track
- walk Deezer's graph: artist top tracks, related artists, album deep cuts
- score by artist frequency, collection album, catalog affinity and hint text
- penalise repeated artists so a 30-slot mood spans artists, not one
- refresh the pool when a song is added instead of waiting for refreshHours
- skip discovery for a mood with no hint, genres or songs
- extend tests and the smoke test with the artist graph
This commit is contained in:
TapTap committed 2026-11-17 10:37:47 +00:00
1 parent a78d359d0c
commit 62bcefad20
13 files changed
+911 -72

No files matched your search

+5
View File
@@ -36,6 +36,11 @@ keeps playing without a mounted music library.
- **Recommendations are active slots.** A mood keeps `recommendationCount`
(0–30, default 10) recommendations in `downloading`, `ready` or `failed`
state. Hidden `new` candidates refill the slots automatically.
- **Discovery uses the whole mood.** Every artist (weighted by how many songs
the mood has), every album, the hint and the genres feed one pass. On Deezer
the server also walks Deezer's artist graph: top tracks of the collection
artists, their related artists, and deep cuts from the collection's albums.
Adding a song makes the pool refresh instead of waiting for `refreshHours`.
- **Add / Not interested are permanent.** Adding promotes a recommendation into
the collection. Rejecting is remembered per mood. Both free a slot and the
server refills it.
+69 -16
View File
@@ -254,19 +254,32 @@ V5/V6/V7 before V8 can wipe it.
States: `new` (hidden candidate pool), `downloading`, `ready`, `failed`,
`added`, `rejected`.
- Discovery stores up to a bounded pool of `new` candidates (e.g. 60) from the
collection's artists/titles + `query` + `genres` + mood name. Exclude video
ids already in `mood_songs`, `added` or `rejected`.
- Discovery reads the **whole collection** (not one seed track): every artist
with its song count, every album, `query`, `genres` and the mood name.
- Discovery has two candidate sources, merged and ranked together:
1. **Text queries**: `query`, then up to 3 collection artists ordered by how
many songs the mood has, then up to 2 genres, capped at 4 queries.
2. **Catalog similarity graph** (`MusicCatalog.discover`): up to 5 collection
artists -> their top tracks; up to 2 of those artists' related artists ->
their top tracks; up to 2 collection albums -> album deep cuts. Deezer
implements this; YouTube Music has no such API and returns nothing.
- Scoring: exact collection artist `3.0 + 0.5` per extra song (capped `+1.5`),
partial artist `1.5`, album already in the collection `+1.0`, catalog
affinity (related artist `1.5`, album deep cut `1.0`, artist top `0.5`),
mood words in the title `+1.0`, over 8 minutes `-1.0`.
- Stores up to a bounded pool of `new` candidates (60). Excludes external ids
already in `mood_songs`, `added` or `rejected`.
- `topUp(mood, force)`:
1. Count active = `downloading + ready + failed`.
2. If `active >= recommendationCount`, stop.
3. Promote the best `new` candidates to `downloading` and enqueue
`download_jobs` until the count is met.
4. If more slots remain and (`force || discoveryDue(mood)`), run discovery,
then promote again.
- `force = true` on user add/reject/retry and on the explicit refresh action.
Coalesce per-mood discovery with an in-flight guard. Background passes use
`force = false` and respect `refreshHours`.
2. Promote the best `new` candidates to `downloading` and enqueue
`download_jobs` until `recommendationCount` is met.
3. Run discovery when slots are missing, or when a song was added after the
last pass (the pool is based on the collection), and
(`force || discoveryDue(mood)`).
4. Promote again after the pass.
- `force = true` on user add/reject/retry, on a finished manual download and on
the explicit refresh action. Background passes use `force = false` and run
hourly.
- `recommendationCount = 0` stops discovery and downloads for the mood.
- On download success:
- `downloading` -> `ready` with `song_id`.
@@ -281,6 +294,43 @@ States: `new` (hidden candidate pool), `downloading`, `ready`, `failed`,
- Retry: `failed` -> `downloading`, reset the job, top up.
- `new` candidates are not shown in the client.
### How this compares to the big services
The commercial engines are trained on behaviour this server does not have: no
account-level listening history, no cross-user co-occurrence, no audio models.
Still, their published architecture explains which parts of the problem can be
replicated from a mood collection alone.
- **Spotify** combines content-based features (audio features such as
danceability, energy and valence, plus metadata and text/visual embeddings)
with collaborative filtering that treats playlist co-occurrence as the
similarity signal ("two tracks are similar if people curate them together").
Retrieval produces candidates, a ranking stage orders them, and each surface
has its own reward (skips, saves, session length).
- **YouTube Music** runs retrieval -> ranking -> filtering. Candidates come from
a deep model over watch history, and a 2024 transformer encodes the *sequence*
of user actions (intent, how much of a track was played, artist, language)
before blending with the ranker. Audio-text embeddings (MuLan) describe the
content itself.
- **Deezer Flow / Flow Moods** (RecSys 2022) mixes collaborative filtering,
audio analysis and professional curator mood annotations to build
mood-specific radios; the 2026 Flow Tuner adds genre toggles, explicit
dislikes, long-term habits and context such as time of day.
What this server reproduces without telemetry:
| Signal | Here | There |
|---|---|---|
| Taste profile | whole mood collection, artists weighted by song count + albums | per-user history and playlists |
| Similarity | catalog graph (related artists, artist top tracks, album deep cuts) + text queries | learned embeddings + co-occurrence |
| Feedback | add / not interested (permanent) | skips, saves, play counts, survey models |
| Context | mood hint, genres, refresh window | session, time of day, device, activity |
| Ranking | weighted score + per-artist diversity penalty | deep rankers with hundreds of features |
Not reproducible here: cross-user collaborative filtering, audio analysis and
sequence models. The add/reject lifecycle and the collection itself are the
only behavioural signals, which is why discovery leans on the catalog graph.
### 4.4 Download pipeline
- `download_jobs` are global; `kind=song` (manual add) has higher priority than
@@ -626,9 +676,12 @@ Phases 1-8 are implemented in both repositories.
the scanner and the downloader.
- `MoodRepository` implements the song collection, hidden `new` candidates,
active slots and the add/reject/retry lifecycle.
- `DiscoveryService` + `TopUpService` implement collection-based discovery and
slot refills. `TopUpService` coalesces work behind a mutex and can be triggered
fire-and-forget from API handlers.
- `DiscoveryService` + `TopUpService` implement whole-collection discovery and
slot refills: text queries plus `MusicCatalog.discover` (Deezer related
artists, artist top tracks, album deep cuts; YouTube Music text only).
`TopUpService` coalesces work behind a mutex and can be triggered
fire-and-forget from API handlers and finished downloads. A song added after
the last pass marks discovery due again, so the pool follows the collection.
- `DownloadService` is a single global worker with `song` jobs outranking
`recommendation` jobs. It indexes immediately after moving the file.
- `MediaGcService` implements refcounted deletion (moods, active recommendations,
@@ -658,8 +711,8 @@ Phases 1-8 are implemented in both repositories.
- Deezer is the default for new moods; every mood stores its own source and
can switch it, which clears the active recommendation pool and refills it.
- Chart 0.4.0 adds `DEFAULT_SOURCE`, `DEEZER_*` and an optional ARL secret.
- Tests: 83 server tests, 64-step smoke test with fake `yt-dlp`, `rip` and a
fake Deezer API.
- Tests: 94 server tests, 66-step smoke test with fake `yt-dlp`, `rip` and a
fake Deezer API (including the artist graph).
### Client
+70 -2
View File
@@ -114,6 +114,37 @@ TRACKS = {
"artist": {"name": "DJ"}, "album": {"title": "Long"}},
}
# Only resolvable by URL, so adding it cannot race a recommendation download.
EXTRA_TRACKS = {
"55555555": {"id": 55555555, "title": "One More Time", "duration": 320,
"readable": True, "artist": {"name": "Daft Punk"},
"album": {"title": "Discovery"}},
}
# Artist graph: the mood holds Daft Punk, so discovery walks to a related act
# and to a deep cut of an album already in the collection.
ARTISTS = {
"27": {"id": 27, "name": "Daft Punk"},
"99": {"id": 99, "name": "Daft Punk Tribute"},
}
TOP_TRACKS = {
"27": [TRACKS["77777777"]],
"99": [{"id": 88888888, "title": "Robot Rock", "duration": 305,
"readable": True, "artist": {"name": "Daft Punk Tribute"},
"album": {"title": "Tribute"}}],
}
RELATED = {"27": [ARTISTS["99"]]}
ALBUMS = {"5": {"id": 5, "title": "Random Access Memories",
"artist": {"name": "Daft Punk"}}}
ALBUM_TRACKS = {
"5": [{"id": 99999999, "title": "Contact", "duration": 382,
"readable": True, "artist": {"name": "Daft Punk"},
"album": {"title": "Random Access Memories"}}],
}
STATS = {"artist_search": 0, "artist_top": 0, "artist_related": 0,
"album_search": 0, "album_tracks": 0}
class Handler(BaseHTTPRequestHandler):
def log_message(self, *args):
pass
@@ -128,10 +159,28 @@ class Handler(BaseHTTPRequestHandler):
def do_GET(self):
parsed = urllib.parse.urlparse(self.path)
parts = [p for p in parsed.path.split("/") if p]
if parsed.path == "/search/track":
self.reply({"data": list(TRACKS.values())})
elif parsed.path == "/search/artist":
STATS["artist_search"] += 1
self.reply({"data": list(ARTISTS.values())})
elif parsed.path == "/search/album":
STATS["album_search"] += 1
self.reply({"data": list(ALBUMS.values())})
elif parsed.path == "/stats":
self.reply(STATS)
elif len(parts) == 3 and parts[0] == "artist" and parts[2] == "top":
STATS["artist_top"] += 1
self.reply({"data": TOP_TRACKS.get(parts[1], [])})
elif len(parts) == 3 and parts[0] == "artist" and parts[2] == "related":
STATS["artist_related"] += 1
self.reply({"data": RELATED.get(parts[1], [])})
elif len(parts) == 3 and parts[0] == "album" and parts[2] == "tracks":
STATS["album_tracks"] += 1
self.reply({"data": ALBUM_TRACKS.get(parts[1], [])})
elif parsed.path.startswith("/track/"):
track = TRACKS.get(parsed.path.rsplit("/", 1)[-1])
track = TRACKS.get(parts[-1]) or EXTRA_TRACKS.get(parts[-1])
if track is None:
self.reply({"error": {"type": "DataException", "message": "no track"}}, 404)
else:
@@ -160,6 +209,7 @@ case "$id" in
67238732) title="Instant Crush"; artist="Daft Punk"; album="Random Access Memories" ;;
66609426) title="Get Lucky"; artist="Daft Punk"; album="Get Lucky" ;;
77777777) title="Around the World"; artist="Daft Punk"; album="Homework" ;;
55555555) title="One More Time"; artist="Daft Punk"; album="Discovery" ;;
*) title="Deezer Track $id"; artist="Deezer Artist"; album="Deezer Album" ;;
esac
ffmpeg -hide_banner -loglevel error -y \
@@ -333,8 +383,26 @@ expect "add a Deezer track" "202" "POST" "/moods/$DZ_MOOD/songs" \
wait_for 90 "Deezer download attaches to the mood" 'd["mood"]["songCount"] >= 1' "/moods/$DZ_MOOD"
wait_for 90 "Deezer recommendations download" 'd["mood"]["recommendationReady"] >= 1' "/moods/$DZ_MOOD"
expect "add a Deezer URL" "202" "POST" "/moods/$DZ_MOOD/songs" \
'{"url":"https://www.deezer.com/track/66609426"}'
'{"url":"https://www.deezer.com/track/55555555"}'
wait_for 90 "Deezer URL attaches to the mood" 'd["mood"]["songCount"] >= 2' "/moods/$DZ_MOOD"
# The collection changed, so the next top-up re-runs discovery against the
# artist graph instead of waiting for the refresh window.
DZ_STATS=""
for _ in $(seq 1 90); do
DZ_STATS="$(curl -sf "http://127.0.0.1:${DZ_API_PORT}/stats" || true)"
if [[ -n "$DZ_STATS" ]] && printf '%s' "$DZ_STATS" | python3 -c '
import json, sys
d = json.load(sys.stdin)
sys.exit(0 if d["artist_related"] > 0 and d["album_tracks"] > 0 else 1)
'; then
break
fi
sleep 1
done
expect_eq "Deezer discovery walks the artist graph" "true" \
"$(printf '%s' "$DZ_STATS" | python3 -c 'import json,sys;d=json.load(sys.stdin);print(str(d["artist_related"] > 0 and d["artist_top"] > 0).lower())')"
expect_eq "Deezer discovery reads album deep cuts" "true" \
"$(printf '%s' "$DZ_STATS" | python3 -c 'import json,sys;d=json.load(sys.stdin);print(str(d["album_tracks"] > 0).lower())')"
if [[ -f "$MEDIA/deezer-67238732/Instant Crush.mp3" ]]; then
pass "Deezer media path"
else
@@ -105,6 +105,7 @@ fun main() {
indexer = indexer,
gc = gc,
tempDir = config.downloadDir,
onSongAttached = topUpService::trigger,
)
if (config.defaultSource == MusicSource.DEEZER && config.deezerArl == null) {
log.warn("DEFAULT_SOURCE is deezer but DEEZER_ARL is empty: downloads will fail")
@@ -117,6 +117,14 @@ object MoodRepository {
?.let { moodRecord(it) }
}
/** Owner-agnostic lookup for background work that only has a mood id. */
fun byId(id: String): MoodRecord? = transaction {
MoodsTable.selectAll()
.where { MoodsTable.id eq id }
.singleOrNull()
?.let { moodRecord(it) }
}
fun create(
owner: String,
name: String,
@@ -590,7 +598,10 @@ object MoodRepository {
.map { moodRecord(it) }
.filter { mood ->
mood.recommendationCount > 0 &&
isStale(mood.lastDiscoveryAt, mood.refreshHours, now)
(
isStale(mood.lastDiscoveryAt, mood.refreshHours, now) ||
songsAddedSince(mood.id, mood.lastDiscoveryAt)
)
}
.take(limit.coerceAtLeast(1))
}
@@ -613,7 +624,23 @@ object MoodRepository {
}
fun isDiscoveryDue(mood: MoodRecord, now: Instant = Instant.now()): Boolean =
isStale(mood.lastDiscoveryAt, mood.refreshHours, now)
isStale(mood.lastDiscoveryAt, mood.refreshHours, now) ||
songsAddedSince(mood.id, mood.lastDiscoveryAt)
/**
* The collection is the discovery seed, so a song added after the last pass
* makes the hidden candidate pool stale, even when every slot is filled.
* Both timestamps come from [Instant.toString], so the lexical comparison is
* also chronological.
*/
fun songsAddedSince(moodId: String, since: String?): Boolean {
if (since == null) return false
return transaction {
MoodSongsTable.selectAll()
.where { MoodSongsTable.moodId eq moodId }
.any { row -> row[MoodSongsTable.addedAt] > since }
}
}
// ------------------------------------------------------------ internal
@@ -6,6 +6,7 @@ import java.net.http.HttpClient
import java.net.http.HttpRequest
import java.net.http.HttpResponse
import java.time.Duration
import java.util.concurrent.ConcurrentHashMap
import kotlinx.serialization.SerialName
import kotlinx.serialization.Serializable
import kotlinx.serialization.json.Json
@@ -16,8 +17,8 @@ import kotlinx.serialization.json.jsonPrimitive
import win.taptap.musicmanager.source.MusicSource
/**
* Deezer's public catalog API. No user token is needed for search or track
* metadata; the ARL is only used by the downloader.
* Deezer's public catalog API. No user token is needed for search, track
* metadata or the similarity graph; the ARL is only used by the downloader.
*
* Deezer returns canonical tracks with a square album cover, which is also what
* the indexer embeds into the downloaded MP3.
@@ -39,6 +40,10 @@ class DeezerSearch(
.followRedirects(HttpClient.Redirect.NORMAL)
.build()
/** Artist and album lookups are stable for a release, so cache them. */
private val artistIds = ConcurrentHashMap<String, Lookup<Long>>()
private val albumRefs = ConcurrentHashMap<String, Lookup<DeezerAlbumRefDto>>()
override fun search(query: String, limit: Int): List<TrackCandidate> {
val count = limit.coerceIn(1, MAX_RESULTS)
val url = "$apiBase/search/track?q=${encode(query)}&limit=$count&output=json"
@@ -54,6 +59,147 @@ class DeezerSearch(
return parseDeezerTrack(get("$apiBase/track/$trackId"))
}
/**
* Walks Deezer's own graph from the entire collection:
*
* 1. the most represented artists → their top tracks;
* 2. those artists' related artists → their top tracks;
* 3. the collection's albums → their deep cuts.
*
* A mood with 40 songs by 6 artists therefore still yields fresh material
* even after the obvious artist hits are already in the mood. Every request
* is best-effort; a blocked endpoint does not fail the pass.
*/
override fun discover(seeds: DiscoverySeeds, limit: Int): List<TrackCandidate> {
if (seeds.isEmpty()) return emptyList()
val found = LinkedHashMap<String, TrackCandidate>()
val seedArtists = seeds.artists
.take(MAX_SEED_ARTISTS)
.mapNotNull { seed -> artistId(seed.name)?.let { seed to it } }
seedArtists.forEach { (_, id) ->
artistTopTracks(id, SEED_TOP_LIMIT).forEach {
add(found, it, SEED_TOP_AFFINITY)
}
}
val related = LinkedHashSet<Long>()
seedArtists.take(MAX_RELATED_SEEDS).forEach { (_, id) ->
related += relatedArtists(id, MAX_RELATED_PER_SEED)
}
val seedIds = seedArtists.map { it.second }.toSet()
related.asSequence()
.filterNot { it in seedIds }
.take(MAX_RELATED_ARTISTS)
.forEach { id ->
artistTopTracks(id, RELATED_TOP_LIMIT).forEach {
add(found, it, RELATED_AFFINITY)
}
}
seeds.albums
.take(MAX_SEED_ALBUMS)
.mapNotNull(::albumRef)
.forEach { album ->
albumTracks(album, ALBUM_TRACK_LIMIT).forEach {
add(found, it, ALBUM_AFFINITY)
}
}
return found.values.take(limit.coerceAtLeast(1))
}
/** Keeps the highest weight when the same track comes from two edges. */
private fun add(
into: MutableMap<String, TrackCandidate>,
candidate: TrackCandidate,
affinity: Double,
) {
if (candidate.externalId.isEmpty()) return
val existing = into[candidate.externalId]
if (existing == null || existing.affinity < affinity) {
into[candidate.externalId] = candidate.copy(affinity = affinity)
}
}
private fun artistId(name: String): Long? = lookup(
cache = artistIds,
key = matchKey(name),
request = "$apiBase/search/artist?q=${encode(name)}&limit=$ARTIST_LOOKUP_LIMIT&output=json",
) { body ->
val parsed = json.decodeFromString<DeezerArtistListResponse>(body)
parsed.data.firstOrNull { artistMatches(it.name, name) && it.id > 0 }?.id
?: parsed.data.firstOrNull { it.id > 0 }?.id
}
private fun albumRef(seed: SeedAlbum): DeezerAlbumRefDto? {
val titleKey = matchKey(seed.title)
if (titleKey.isEmpty()) return null
val artistKey = matchKey(seed.artist)
return lookup(
cache = albumRefs,
key = "$titleKey|$artistKey",
request = "$apiBase/search/album?q=${encode("${seed.title} ${seed.artist}")}" +
"&limit=$ALBUM_LOOKUP_LIMIT&output=json",
) { body ->
val parsed = json.decodeFromString<DeezerAlbumListResponse>(body)
parsed.data.firstOrNull { album ->
album.id > 0 &&
matchKey(album.title) == titleKey &&
(album.artist == null || artistMatches(album.artist.name, seed.artist))
}
}
}
/**
* Resolves a name to an id once. Misses are cached too, so a mood full of
* obscure artists does not re-search them on every refresh.
*/
private inline fun <T : Any> lookup(
cache: MutableMap<String, Lookup<T>>,
key: String,
request: String,
crossinline parse: (String) -> T?,
): T? {
if (key.isEmpty()) return null
cache[key]?.let { return it.value }
val value = runCatching { parse(get(request)) }.getOrNull()
cache[key] = Lookup(value)
return value
}
private fun relatedArtists(id: Long, limit: Int): List<Long> = runCatching {
val body = get("$apiBase/artist/$id/related?limit=$limit&output=json")
json.decodeFromString<DeezerArtistListResponse>(body)
.data
.map { it.id }
.filter { it > 0 }
}.getOrDefault(emptyList())
private fun artistTopTracks(id: Long, limit: Int): List<TrackCandidate> = runCatching {
val body = get("$apiBase/artist/$id/top?limit=$limit&output=json")
json.decodeFromString<DeezerTracksResponse>(body).data.mapNotNull { it.toCandidate() }
}.getOrDefault(emptyList())
/**
* `/album/{id}/tracks` omits the album object, so inject the album the
* collection already gave us to keep the cover and the album name.
*/
private fun albumTracks(
album: DeezerAlbumRefDto,
limit: Int,
): List<TrackCandidate> = runCatching {
val body = get("$apiBase/album/${album.id}/tracks?limit=$limit&output=json")
json.decodeFromString<DeezerTracksResponse>(body).data
.mapNotNull { it.toCandidate() }
.map { candidate ->
candidate.copy(
album = candidate.album ?: album.title.takeIf { it.isNotBlank() },
thumbnail = candidate.thumbnail ?: album.cover(),
)
}
}.getOrDefault(emptyList())
private fun get(url: String): String {
throttle()
val request = HttpRequest.newBuilder(URI.create(url))
@@ -111,8 +257,31 @@ class DeezerSearch(
}
}
/** `value == null` is a cached miss. */
private data class Lookup<T>(val value: T?)
private companion object {
const val MAX_RESULTS = 25
/** Bounds one discovery pass to roughly twenty throttled requests. */
const val MAX_SEED_ARTISTS = 5
const val MAX_RELATED_SEEDS = 2
const val MAX_RELATED_PER_SEED = 8
const val MAX_RELATED_ARTISTS = 6
const val MAX_SEED_ALBUMS = 2
const val SEED_TOP_LIMIT = 25
const val RELATED_TOP_LIMIT = 10
const val ALBUM_TRACK_LIMIT = 40
const val ARTIST_LOOKUP_LIMIT = 5
const val ALBUM_LOOKUP_LIMIT = 5
/** A hit by an artist already in the mood; the scorer adds the rest. */
const val SEED_TOP_AFFINITY = 0.5
/** A hit by an artist Deezer considers similar to the collection. */
const val RELATED_AFFINITY = 1.5
/** Another track from an album already in the mood. */
const val ALBUM_AFFINITY = 1.0
const val MIN_REQUEST_INTERVAL_NANOS = 120_000_000L
val THROTTLE_LOCK = Any()
@@ -152,6 +321,29 @@ internal fun parseDeezerTrack(json: String): TrackCandidate? {
@Serializable
private data class DeezerSearchResponse(val data: List<DeezerTrackDto> = emptyList())
@Serializable
private data class DeezerTracksResponse(val data: List<DeezerTrackDto> = emptyList())
@Serializable
private data class DeezerArtistListResponse(val data: List<DeezerArtistRefDto> = emptyList())
@Serializable
private data class DeezerArtistRefDto(val id: Long = 0, val name: String = "")
@Serializable
private data class DeezerAlbumListResponse(val data: List<DeezerAlbumRefDto> = emptyList())
@Serializable
private data class DeezerAlbumRefDto(
val id: Long = 0,
val title: String = "",
val artist: DeezerArtistDto? = null,
@SerialName("cover_big") val coverBig: String? = null,
@SerialName("cover_xl") val coverXl: String? = null,
) {
fun cover(): String? = coverBig ?: coverXl
}
@Serializable
private data class DeezerTrackDto(
val id: Long = 0,
@@ -10,6 +10,7 @@ import org.slf4j.LoggerFactory
import win.taptap.musicmanager.db.MoodRecord
import win.taptap.musicmanager.db.MoodRepository
import win.taptap.musicmanager.db.RecommendationUpsert
import win.taptap.musicmanager.db.SongRecord
import win.taptap.musicmanager.util.IdGenerator
data class DiscoveryOutcome(
@@ -39,24 +40,54 @@ class DiscoveryService(
val catalog = catalogs.forSource(mood.source)
val collection = MoodRepository.songRecords(mood.id)
val seedArtists = collection.map { it.artist }.filter { it.isNotBlank() }
val queries = buildQueries(mood, seedArtists)
if (queries.isEmpty()) {
MoodRepository.markDiscovered(mood.id)
return DiscoveryOutcome(0)
val artists = artistsByFrequency(collection)
val albums = albumsOf(collection)
val seeds = DiscoverySeeds(
artists = artists,
albums = albums,
hint = mood.query,
genres = mood.genres,
name = mood.name,
)
val queries = buildQueries(mood, artists.map { it.name })
if (queries.isEmpty() && seeds.isEmpty()) {
// A new mood with no hint, no genres and no songs has nothing to
// base music on. Leave the slots empty instead of downloading
// whatever the mood's name happens to match.
log.debug("Skipping discovery for mood {}: no music signal yet", mood.name)
return DiscoveryOutcome(0, skipped = true)
}
// Queries are independent; running them together keeps a refresh within
// the client's request timeout even when a catalog is slow.
// Text queries and the catalog's similarity graph run together: the
// graph knows about related artists and album deep cuts, which a title
// search cannot express.
val results = coroutineScope {
queries.map { query ->
async(Dispatchers.IO) {
runCatching { catalog.search(query, QUERY_LIMIT) }
val searches = queries.map { query ->
async(Dispatchers.IO) { runCatching { catalog.search(query, QUERY_LIMIT) } }
}
val expansion = async(Dispatchers.IO) {
if (seeds.isEmpty()) {
emptyList()
} else {
runCatching { catalog.discover(seeds, DISCOVERY_LIMIT) }
.onFailure {
log.debug(
"{} expansion failed for mood {}: {}",
catalog.source,
mood.name,
it.message,
)
}
.getOrDefault(emptyList())
}
}.awaitAll()
}
searches.awaitAll() to expansion.await()
}
val failures = results.filter { it.isFailure }
if (failures.size == queries.size) {
val searches = results.first
val discovered = results.second
val failures = searches.filter { it.isFailure }
if (searches.isNotEmpty() && failures.size == searches.size && discovered.isEmpty()) {
val message = failures.firstNotNullOfOrNull { it.exceptionOrNull()?.message }
?: "Search failed"
log.warn("{} discovery failed for mood {}: {}", catalog.source, mood.name, message)
@@ -65,9 +96,11 @@ class DiscoveryService(
}
val candidates = LinkedHashMap<String, TrackCandidate>()
results.mapNotNull { it.getOrNull() }.flatten().forEach { candidate ->
if (isMusicLength(candidate.duration)) {
candidates.putIfAbsent(candidate.externalId, candidate)
(searches.mapNotNull { it.getOrNull() }.flatten() + discovered).forEach { candidate ->
if (!isMusicLength(candidate.duration)) return@forEach
val existing = candidates[candidate.externalId]
if (existing == null || existing.affinity < candidate.affinity) {
candidates[candidate.externalId] = candidate
}
}
@@ -85,28 +118,33 @@ class DiscoveryService(
.toSet()
val moodText = listOfNotNull(mood.query, mood.description, mood.name).joinToString(" ")
val artistWeights = artists
.associate { matchKey(it.name) to it.songs.toDouble() }
.filterKeys { it.isNotEmpty() }
val albumKeys = albums
.map { matchKey(it.title) }
.filter { it.isNotEmpty() }
.toSet()
val upserts = candidates.values
.filterNot { it.externalId in excluded }
.map { candidate ->
candidate to scoreCandidate(candidate, seedArtists, moodText)
}
.sortedWith(
compareByDescending<Pair<TrackCandidate, Double>> { it.second }
.thenBy { it.first.title },
val ranked = rankCandidates(
candidates.values
.filterNot { it.externalId in excluded }
.map { candidate ->
candidate to scoreCandidate(candidate, artistWeights, albumKeys, moodText)
},
CANDIDATE_POOL,
)
val upserts = ranked.map { candidate ->
RecommendationUpsert(
source = mood.source,
externalId = candidate.externalId,
title = candidate.title,
artist = candidate.artist,
album = candidate.album,
duration = candidate.duration,
thumbnail = candidate.thumbnail,
)
.take(CANDIDATE_POOL)
.map { (candidate, _) ->
RecommendationUpsert(
source = mood.source,
externalId = candidate.externalId,
title = candidate.title,
artist = candidate.artist,
album = candidate.album,
duration = candidate.duration,
thumbnail = candidate.thumbnail,
)
}
}
if (upserts.isEmpty()) {
MoodRepository.markDiscovered(mood.id)
@@ -121,12 +159,22 @@ class DiscoveryService(
CANDIDATE_POOL,
)
MoodRepository.markDiscovered(mood.id)
log.info("{} discovery for mood {} stored {} candidates", catalog.source, mood.name, stored)
log.info(
"{} discovery for mood {} stored {} candidates from {} searches and {} expanded hits",
catalog.source,
mood.name,
stored,
queries.size,
discovered.size,
)
return DiscoveryOutcome(stored = stored)
}
private companion object {
const val QUERY_LIMIT = 10
/** The graph expands far more than a single search page does. */
const val DISCOVERY_LIMIT = 200
const val CANDIDATE_POOL = 60
}
}
@@ -144,28 +192,96 @@ internal fun isMusicLength(seconds: Int?): Boolean =
internal fun buildQueries(mood: MoodRecord, seedArtists: List<String>): List<String> {
val queries = LinkedHashSet<String>()
mood.query?.trim()?.takeIf { it.isNotEmpty() }?.let(queries::add)
seedArtists.map { it.trim() }.filter { it.isNotEmpty() }.distinct().take(2).forEach {
// The collection is the strongest signal, so its artists fill the slots
// before genres do. They arrive ordered by how many songs the mood holds.
seedArtists.map { it.trim() }.filter { it.isNotEmpty() }.distinct().take(3).forEach {
queries += it
}
mood.genres.map { it.trim() }.filter { it.isNotEmpty() }.take(2).forEach {
queries += "$it mix"
}
if (queries.isEmpty()) queries += mood.name
return queries.distinctBy { it.lowercase() }.take(4)
}
/** Collection artists, most represented first. */
internal fun artistsByFrequency(collection: List<SongRecord>): List<SeedArtist> =
collection
.map { it.artist.trim() }
.filter { it.isNotEmpty() }
.groupingBy { it }
.eachCount()
.entries
.sortedWith(
compareByDescending<Map.Entry<String, Int>> { it.value }.thenBy { it.key.lowercase() },
)
.map { SeedArtist(it.key, it.value) }
/** Distinct albums of the collection, deduplicated loosely. */
internal fun albumsOf(collection: List<SongRecord>): List<SeedAlbum> =
collection.mapNotNull { song ->
val album = song.album?.trim()?.takeIf { it.isNotEmpty() } ?: return@mapNotNull null
SeedAlbum(album, song.artist)
}.distinctBy { "${matchKey(it.title)}|${matchKey(it.artist)}" }
/**
* Picks the best [limit] candidates while penalising an artist that keeps
* winning: without it, a mood with three songs by one artist would fill every
* slot with that artist. Each additional pick by the same artist costs
* [ARTIST_REPEAT_PENALTY], so a related artist eventually takes a slot.
*/
internal fun rankCandidates(
scored: List<Pair<TrackCandidate, Double>>,
limit: Int,
): List<TrackCandidate> {
val remaining = scored.toMutableList()
val chosen = ArrayList<TrackCandidate>(limit.coerceAtMost(remaining.size))
val perArtist = HashMap<String, Int>()
while (chosen.size < limit && remaining.isNotEmpty()) {
var bestIndex = 0
var bestScore = Double.NEGATIVE_INFINITY
for (index in remaining.indices) {
val candidate = remaining[index].first
val picked = perArtist[matchKey(candidate.artist)] ?: 0
val adjusted = remaining[index].second - ARTIST_REPEAT_PENALTY * picked
if (adjusted > bestScore) {
bestScore = adjusted
bestIndex = index
}
}
val candidate = remaining.removeAt(bestIndex).first
chosen += candidate
val key = matchKey(candidate.artist)
perArtist[key] = (perArtist[key] ?: 0) + 1
}
return chosen
}
/**
* Ranks a candidate against the whole collection:
*
* - an artist already in the mood is the strongest signal, and an artist with
* more songs in the mood ranks higher;
* - another track from an album already in the mood is a strong signal;
* - a catalog-supplied [TrackCandidate.affinity] (related artist, top track)
* is added as-is;
* - the mood's own words in the title are a tie-breaker.
*/
internal fun scoreCandidate(
candidate: TrackCandidate,
seedArtists: List<String>,
artistWeights: Map<String, Double>,
albumKeys: Set<String>,
moodText: String,
): Double {
var score = 0.0
var score = candidate.affinity
val artist = matchKey(candidate.artist)
val seedKeys = seedArtists.map { matchKey(it) }.filter { it.isNotEmpty() }
val exact = artistWeights[artist]
when {
artist.isNotEmpty() && artist in seedKeys -> score += 3.0
artist.isNotEmpty() && seedKeys.any { artist.contains(it) || it.contains(artist) } -> score += 1.5
exact != null -> score += 3.0 + (0.5 * (exact - 1)).coerceAtMost(1.5)
artist.isNotEmpty() && artistWeights.keys.any { artistMatches(artist, it) } -> score += 1.5
}
val album = matchKey(candidate.album.orEmpty())
if (album.isNotEmpty() && album in albumKeys) score += 1.0
val text = matchKey(moodText)
if (text.isNotEmpty() && matchKey(candidate.title).contains(text)) score += 1.0
if ((candidate.duration ?: 0) > 8 * 60) score -= 1.0
@@ -198,3 +314,4 @@ internal fun artistMatches(a: String, b: String): Boolean {
internal const val MIN_TRACK_SECONDS = 30
internal const val MAX_TRACK_SECONDS = 600
internal const val ARTIST_REPEAT_PENALTY = 1.0
@@ -2,7 +2,13 @@ package win.taptap.musicmanager.discovery
import win.taptap.musicmanager.source.MusicSource
/** One source-specific search hit, already split into an artist and a title. */
/**
* One source-specific search hit, already split into an artist and a title.
*
* [affinity] is a catalog-supplied similarity weight: the catalog knows where a
* hit came from (a related artist, an album deep cut), which is information a
* title search cannot express.
*/
data class TrackCandidate(
val source: String,
val externalId: String,
@@ -11,8 +17,29 @@ data class TrackCandidate(
val album: String? = null,
val duration: Int? = null,
val thumbnail: String? = null,
val affinity: Double = 0.0,
)
/** One artist of the mood collection and how many of its songs the mood holds. */
data class SeedArtist(val name: String, val songs: Int)
/** One album of the mood collection. */
data class SeedAlbum(val title: String, val artist: String)
/**
* The whole mood collection, handed to a catalog that can expand it through its
* own similarity graph instead of title searches alone.
*/
data class DiscoverySeeds(
val artists: List<SeedArtist> = emptyList(),
val albums: List<SeedAlbum> = emptyList(),
val hint: String? = null,
val genres: List<String> = emptyList(),
val name: String = "",
) {
fun isEmpty(): Boolean = artists.isEmpty() && albums.isEmpty()
}
/** A music catalog that can be searched and can resolve a pasted URL. */
interface MusicCatalog {
val source: String
@@ -21,6 +48,16 @@ interface MusicCatalog {
/** Resolves a pasted track URL, or returns null when it is not a track. */
fun resolve(url: String): TrackCandidate?
/**
* Expands the whole collection through the catalog's own similarity graph:
* related artists, artist top tracks and album deep cuts. Catalogs without
* such an API keep the default empty list and rely on text queries.
*
* Implementations must be best-effort: one failing sub-request returns what
* the other requests found instead of failing the whole discovery pass.
*/
fun discover(seeds: DiscoverySeeds, limit: Int): List<TrackCandidate> = emptyList()
}
/**
@@ -38,6 +38,19 @@ class TopUpService(
}
}
/**
* Same, for work that only knows the id: a finished download attaches a song
* and changes the collection the next discovery pass is based on.
*/
fun trigger(moodId: String) {
val scope = backgroundScope ?: return
scope.launch(Dispatchers.IO) {
val mood = MoodRepository.byId(moodId) ?: return@launch
runCatching { topUp(mood, force = true) }
.onFailure { log.warn("Recommendation top-up for mood {} failed", mood.name, it) }
}
}
suspend fun topUp(mood: MoodRecord, force: Boolean = false): TopUpResult = mutex.withLock {
var current = MoodRepository.get(mood.id, mood.owner) ?: return@withLock TopUpResult(0)
if (current.recommendationCount <= 0) return@withLock TopUpResult(0)
@@ -46,7 +59,10 @@ class TopUpService(
val stillMissing = current.recommendationCount - MoodRepository.activeCount(current.id)
var discoveryError: String? = null
if (stillMissing > 0 && (force || MoodRepository.isDiscoveryDue(current))) {
val collectionChanged = MoodRepository.songsAddedSince(current.id, current.lastDiscoveryAt)
if ((stillMissing > 0 || collectionChanged) &&
(force || MoodRepository.isDiscoveryDue(current))
) {
val outcome = discovery.refresh(current, force)
discoveryError = outcome.error
current = MoodRepository.get(mood.id, mood.owner) ?: current
@@ -37,6 +37,12 @@ class DownloadService(
private val tagExtractor: Mp3TagExtractor = Mp3TagExtractor(),
private val tempDir: Path,
private val maxAttempts: Int = 3,
/**
* Called after a manual download becomes a permanent mood song. The mood
* collection is the discovery seed, so the recommendation pool has to be
* refreshed instead of waiting for the hourly worker.
*/
private val onSongAttached: (String) -> Unit = {},
) {
private val log = LoggerFactory.getLogger(DownloadService::class.java)
@@ -119,6 +125,8 @@ class DownloadService(
DownloadRepository.KIND_SONG -> {
MoodRepository.addSong(job.moodId, song.id, "search")
DownloadRepository.markDone(job.id, song.id)
runCatching { onSongAttached(job.moodId) }
.onFailure { log.warn("Mood {} top-up failed", job.moodId, it) }
}
else -> {
when (
@@ -179,6 +179,28 @@ class MoodRepositoryTest {
}
}
@Test
fun `adding a song makes discovery due again`() {
withDatabase { mediaStore, _ ->
val mood = createMood(recommendationCount = 2)
assertTrue(MoodRepository.isDiscoveryDue(mood), "A never-seen mood is due")
MoodRepository.markDiscovered(mood.id)
val fresh = assertNotNull(MoodRepository.get(mood.id, "alice"))
assertFalse(MoodRepository.isDiscoveryDue(fresh), "Nothing changed since the pass")
// Timestamps are compared strictly; keep them distinguishable.
Thread.sleep(5)
val song = insertSong(mediaStore, MusicSource.YOUTUBE, "vid-1", "One")
MoodRepository.addSong(mood.id, song.id, "search")
val changed = assertNotNull(MoodRepository.get(mood.id, "alice"))
assertTrue(
MoodRepository.isDiscoveryDue(changed),
"The collection is the discovery seed, so the pool is stale",
)
}
}
@Test
fun `download completion after a rejection does not attach the song`() {
withDatabase { mediaStore, _ ->
@@ -0,0 +1,137 @@
package win.taptap.musicmanager.discovery
import com.sun.net.httpserver.HttpServer
import java.net.InetSocketAddress
import java.util.concurrent.CopyOnWriteArrayList
import kotlin.test.Test
import kotlin.test.assertEquals
import kotlin.test.assertTrue
/**
* Exercises the Deezer similarity walk against a local stand-in API, so the
* request shapes and the affinity weights stay pinned without network access.
*/
class DeezerDiscoverTest {
@Test
fun `expands the collection through related artists and album deep cuts`() {
val requests = CopyOnWriteArrayList<String>()
val server = serve { path ->
requests += path
when (path) {
"/search/artist" ->
ok("""{"data":[{"id":27,"name":"Daft Punk"}]}""")
"/search/album" ->
ok(
"""{"data":[{"id":5,"title":"Random Access Memories",
"artist":{"name":"Daft Punk"}}]}""",
)
"/artist/27/top" ->
ok(
"""{"data":[
{"id":1,"title":"Instant Crush","duration":337,"readable":true,
"artist":{"name":"Daft Punk"},
"album":{"title":"Random Access Memories"}},
{"id":2,"title":"Too Long","duration":900,"readable":true,
"artist":{"name":"Daft Punk"},
"album":{"title":"Random Access Memories"}}]}""",
)
"/artist/27/related" ->
ok("""{"data":[{"id":99,"name":"Daft Punk Tribute"}]}""")
"/artist/99/top" ->
ok(
"""{"data":[
{"id":3,"title":"Robot Rock","duration":300,"readable":true,
"artist":{"name":"Daft Punk Tribute"}}]}""",
)
"/album/5/tracks" ->
ok(
"""{"data":[
{"id":4,"title":"Contact","duration":382,"readable":true,
"artist":{"name":"Daft Punk"},
"album":{"title":"Random Access Memories"}}]}""",
)
else -> 404 to """{"error":{"type":"DataException","message":"not found"}}"""
}
}
try {
val found = catalog(server).discover(seeds(), limit = 50).associateBy { it.externalId }
// The 15-minute track is not music-length and never reaches the pool.
assertEquals(setOf("1", "3", "4"), found.keys)
assertEquals(0.5, found.getValue("1").affinity)
assertEquals(1.5, found.getValue("3").affinity)
assertEquals(1.0, found.getValue("4").affinity)
// `/album/{id}/tracks` has no album object; the search result fills it in.
assertEquals("Random Access Memories", found.getValue("4").album)
assertTrue("/artist/27/related" in requests)
assertTrue("/artist/99/top" in requests)
assertTrue("/album/5/tracks" in requests)
} finally {
server.stop(0)
}
}
@Test
fun `a failing endpoint does not fail the whole expansion`() {
val server = serve { path ->
when (path) {
"/search/artist" -> ok("""{"data":[{"id":27,"name":"Daft Punk"}]}""")
"/artist/27/top" ->
ok(
"""{"data":[
{"id":1,"title":"Instant Crush","duration":337,"readable":true,
"artist":{"name":"Daft Punk"}}]}""",
)
else -> 500 to """{"error":{"type":"DataException","message":"boom"}}"""
}
}
try {
val found = catalog(server).discover(seeds(), limit = 50)
assertEquals(listOf("1"), found.map { it.externalId })
} finally {
server.stop(0)
}
}
@Test
fun `an empty collection makes no requests`() {
val requests = CopyOnWriteArrayList<String>()
val server = serve { path ->
requests += path
404 to """{"error":{"type":"DataException","message":"not found"}}"""
}
try {
assertTrue(catalog(server).discover(DiscoverySeeds(), limit = 50).isEmpty())
assertTrue(requests.isEmpty())
} finally {
server.stop(0)
}
}
private fun seeds() = DiscoverySeeds(
artists = listOf(SeedArtist("Daft Punk", 4)),
albums = listOf(SeedAlbum("Random Access Memories", "Daft Punk")),
)
private fun catalog(server: HttpServer) = DeezerSearch(
apiBase = "http://127.0.0.1:${server.address.port}",
timeoutSeconds = 5,
)
private fun ok(body: String) = 200 to body
private fun serve(route: (String) -> Pair<Int, String>): HttpServer {
val server = HttpServer.create(InetSocketAddress("127.0.0.1", 0), 0)
server.createContext("/") { exchange ->
val (status, body) = route(exchange.requestURI.path)
val bytes = body.toByteArray()
exchange.responseHeaders.add("Content-Type", "application/json")
exchange.sendResponseHeaders(status, bytes.size.toLong())
exchange.responseBody.use { it.write(bytes) }
}
server.start()
return server
}
}
@@ -6,6 +6,7 @@ import kotlin.test.assertEquals
import kotlin.test.assertFalse
import kotlin.test.assertTrue
import win.taptap.musicmanager.db.MoodRecord
import win.taptap.musicmanager.db.SongRecord
import win.taptap.musicmanager.source.MusicSource
class DiscoveryServiceTest {
@@ -209,8 +210,8 @@ class DiscoveryServiceTest {
}
@Test
fun `queries fall back to the mood name`() {
assertEquals(listOf("Focus"), buildQueries(mood(), emptyList()))
fun `a mood without a hint genres or songs has no query`() {
assertTrue(buildQueries(mood(), emptyList()).isEmpty())
}
@Test
@@ -223,12 +224,167 @@ class DiscoveryServiceTest {
duration = 200,
)
val plain = scoreCandidate(candidate, listOf("Alpha"), "focus")
val seedArtist = scoreCandidate(candidate, listOf("Beta"), "focus")
val plain = scoreCandidate(candidate, mapOf("alpha" to 1.0), emptySet(), "focus")
val seedArtist = scoreCandidate(candidate, mapOf("beta" to 1.0), emptySet(), "focus")
assertTrue(seedArtist > plain)
}
@Test
fun `an artist with more songs in the mood ranks higher`() {
val candidate = TrackCandidate(
source = MusicSource.DEEZER,
externalId = "1",
title = "Focus Two",
artist = "Beta",
duration = 200,
)
val once = scoreCandidate(candidate, mapOf("beta" to 1.0), emptySet(), "")
val often = scoreCandidate(candidate, mapOf("beta" to 12.0), emptySet(), "")
assertTrue(often > once)
}
@Test
fun `album deep cuts outrank tracks from unrelated albums`() {
val candidate = TrackCandidate(
source = MusicSource.DEEZER,
externalId = "1",
title = "Contact",
artist = "Daft Punk",
album = "Random Access Memories",
duration = 382,
)
val unrelated = scoreCandidate(candidate, emptyMap(), setOf("homework"), "")
val sameAlbum = scoreCandidate(
candidate,
emptyMap(),
setOf(matchKey("Random Access Memories")),
"",
)
assertEquals(1.0, sameAlbum - unrelated)
}
@Test
fun `ranking spreads picks across artists instead of repeating one`() {
val lead = (1..12).map {
TrackCandidate(
source = MusicSource.DEEZER,
externalId = "lead-$it",
title = "Lead $it",
artist = "Alpha",
duration = 200,
)
}
val related = (1..6).map {
TrackCandidate(
source = MusicSource.DEEZER,
externalId = "related-$it",
title = "Related $it",
artist = "Beta",
duration = 200,
affinity = 1.5,
)
}
val scored = (lead + related).map {
it to scoreCandidate(it, mapOf("alpha" to 1.0), emptySet(), "")
}
val ranked = rankCandidates(scored, limit = 8)
assertEquals(8, ranked.size)
assertTrue(ranked.any { it.artist == "Beta" }, "A related artist must take a slot")
assertTrue(
ranked.count { it.artist == "Alpha" } <= 5,
"The lead artist must not fill every slot",
)
}
@Test
fun `catalog affinity raises a related artist hit`() {
val candidate = TrackCandidate(
source = MusicSource.DEEZER,
externalId = "1",
title = "Robot Rock",
artist = "Daft Punk Tribute",
duration = 300,
)
val plain = scoreCandidate(candidate, emptyMap(), emptySet(), "")
val related = scoreCandidate(
candidate.copy(affinity = 1.5),
emptyMap(),
emptySet(),
"",
)
assertEquals(1.5, related - plain)
}
@Test
fun `queries prefer collection artists over genres`() {
val mood = mood(query = "late night", genres = listOf("Ambient", "Jazz"))
val queries = buildQueries(mood, listOf("Alpha", "Beta", "Gamma", "Delta"))
assertEquals(listOf("late night", "Alpha", "Beta", "Gamma"), queries)
}
@Test
fun `collection artists are ordered by how many songs the mood has`() {
val collection = listOf(
song("Alpha"),
song("Beta"),
song("Alpha"),
song("Beta"),
song("Alpha"),
)
assertEquals(
listOf(SeedArtist("Alpha", 3), SeedArtist("Beta", 2)),
artistsByFrequency(collection),
)
}
@Test
fun `albums are collected once per title and artist`() {
val collection = listOf(
song("Daft Punk", album = "Random Access Memories"),
song("Daft Punk", album = "Random Access Memories"),
song("Daft Punk", album = "Homework"),
song("Other", album = null),
)
assertEquals(
listOf(
SeedAlbum("Random Access Memories", "Daft Punk"),
SeedAlbum("Homework", "Daft Punk"),
),
albumsOf(collection),
)
}
private fun song(artist: String, album: String? = "Album") = SongRecord(
id = "sg-$artist-$album",
source = MusicSource.DEEZER,
externalId = "1",
title = "Title",
artist = artist,
album = album,
duration = 180,
bitRate = 128,
size = 1,
suffix = "mp3",
contentType = "audio/mpeg",
relativePath = "deezer-1/Title.mp3",
fileMtime = 0,
thumbnail = null,
createdAt = "2026-01-01T00:00:00Z",
updatedAt = "2026-01-01T00:00:00Z",
)
private fun mood(
query: String? = null,
genres: List<String> = emptyList(),