feat: recommend from Deezer and YouTube Music radios
CI / chart (push) Successful in 2s
CI / test (push) Successful in 1m57s
CI / chart-push (push) Successful in 5s
CI / image (push) Successful in 3m39s

- Deezer: /artist/{id}/radio for collection artists, affinity 2.0
- YouTube Music: RDAMVM watch playlist via yt-dlp, affinity 2.0
- seed radios from the 2 newest songs of the mood's own catalog
- dedupe candidates by recording, not only by external id
- clean raw YouTube titles and Topic suffixes when indexing
- tests, smoke marker for the RDAMVM path, docs
This commit is contained in:
TapTap committed 2026-11-17 15:22:05 +00:00
1 parent 62bcefad20
commit 26c41cc7d9
12 files changed
+296 -44

No files matched your search

+5 -3
View File
@@ -38,9 +38,11 @@ keeps playing without a mounted music library.
state. Hidden `new` candidates refill the slots automatically. state. Hidden `new` candidates refill the slots automatically.
- **Discovery uses the whole mood.** Every artist (weighted by how many songs - **Discovery uses the whole mood.** Every artist (weighted by how many songs
the mood has), every album, the hint and the genres feed one pass. On Deezer the mood has), every album, the hint and the genres feed one pass. On Deezer
the server also walks Deezer's artist graph: top tracks of the collection the server also asks Deezer for the artists' **radio** — a Daft Punk mood gets
artists, their related artists, and deep cuts from the collection's albums. MGMT, Caribou and Jamiroquai — plus related artists and album deep cuts. On
Adding a song makes the pool refresh instead of waiting for `refreshHours`. YouTube Music it follows the song's `RDAMVM` **radio queue**, the same mix the
player builds. Adding a song makes the pool refresh instead of waiting for
`refreshHours`.
- **Add / Not interested are permanent.** Adding promotes a recommendation into - **Add / Not interested are permanent.** Adding promotes a recommendation into
the collection. Rejecting is remembered per mood. Both free a slot and the the collection. Rejecting is remembered per mood. Both free a slot and the
server refills it. server refills it.
+17 -9
View File
@@ -260,13 +260,18 @@ States: `new` (hidden candidate pool), `downloading`, `ready`, `failed`,
1. **Text queries**: `query`, then up to 3 collection artists ordered by how 1. **Text queries**: `query`, then up to 3 collection artists ordered by how
many songs the mood has, then up to 2 genres, capped at 4 queries. many songs the mood has, then up to 2 genres, capped at 4 queries.
2. **Catalog similarity graph** (`MusicCatalog.discover`): up to 5 collection 2. **Catalog similarity graph** (`MusicCatalog.discover`): up to 5 collection
artists -> their top tracks; up to 2 of those artists' related artists -> artists -> their **radio** (`/artist/{id}/radio`, the platform's own mix)
their top tracks; up to 2 collection albums -> album deep cuts. Deezer and top tracks; up to 2 of those artists' related artists -> their top
implements this; YouTube Music has no such API and returns nothing. tracks; up to 2 collection albums -> album deep cuts. YouTube Music runs
the `RDAMVM<videoId>` watch-playlist radio for the 2 newest collection
songs, which is the same queue its player builds (`yt-dlp
--flat-playlist`), so it also returns real recommendations.
- Scoring: exact collection artist `3.0 + 0.5` per extra song (capped `+1.5`), - Scoring: exact collection artist `3.0 + 0.5` per extra song (capped `+1.5`),
partial artist `1.5`, album already in the collection `+1.0`, catalog partial artist `1.5`, album already in the collection `+1.0`, catalog
affinity (related artist `1.5`, album deep cut `1.0`, artist top `0.5`), affinity (platform radio `2.0`, related artist `1.5`, album deep cut `1.0`,
mood words in the title `+1.0`, over 8 minutes `-1.0`. artist top `0.5`), mood words in the title `+1.0`, over 8 minutes `-1.0`.
- A recording is deduplicated by `(artist, title)` as well as by external id, so
two uploads of a song already in the mood do not both enter the pool.
- Stores up to a bounded pool of `new` candidates (60). Excludes external ids - Stores up to a bounded pool of `new` candidates (60). Excludes external ids
already in `mood_songs`, `added` or `rejected`. already in `mood_songs`, `added` or `rejected`.
- `topUp(mood, force)`: - `topUp(mood, force)`:
@@ -322,14 +327,17 @@ What this server reproduces without telemetry:
| Signal | Here | There | | Signal | Here | There |
|---|---|---| |---|---|---|
| Taste profile | whole mood collection, artists weighted by song count + albums | per-user history and playlists | | Taste profile | whole mood collection, artists weighted by song count + albums | per-user history and playlists |
| Similarity | catalog graph (related artists, artist top tracks, album deep cuts) + text queries | learned embeddings + co-occurrence | | Similarity | the platform's own radio (Deezer artist radio, YouTube `RDAMVM`) + catalog graph + text queries | learned embeddings + co-occurrence |
| Feedback | add / not interested (permanent) | skips, saves, play counts, survey models | | Feedback | add / not interested (permanent) | skips, saves, play counts, survey models |
| Context | mood hint, genres, refresh window | session, time of day, device, activity | | Context | mood hint, genres, refresh window | session, time of day, device, activity |
| Ranking | weighted score + per-artist diversity penalty | deep rankers with hundreds of features | | Ranking | weighted score + per-artist diversity penalty | deep rankers with hundreds of features |
Not reproducible here: cross-user collaborative filtering, audio analysis and Not reproducible here: cross-user collaborative filtering, audio analysis and
sequence models. The add/reject lifecycle and the collection itself are the sequence models. The add/reject lifecycle and the collection itself are the
only behavioural signals, which is why discovery leans on the catalog graph. only behavioural signals, which is why discovery leans on the platforms' own
radio endpoints. Spotify is not an option: its `/v1/recommendations`, related
artists, audio features and audio analysis were removed for new apps on
2024-11-27, and its terms forbid training on Spotify content.
### 4.4 Download pipeline ### 4.4 Download pipeline
@@ -711,8 +719,8 @@ Phases 1-8 are implemented in both repositories.
- Deezer is the default for new moods; every mood stores its own source and - Deezer is the default for new moods; every mood stores its own source and
can switch it, which clears the active recommendation pool and refills it. can switch it, which clears the active recommendation pool and refills it.
- Chart 0.4.0 adds `DEFAULT_SOURCE`, `DEEZER_*` and an optional ARL secret. - Chart 0.4.0 adds `DEFAULT_SOURCE`, `DEEZER_*` and an optional ARL secret.
- Tests: 94 server tests, 66-step smoke test with fake `yt-dlp`, `rip` and a - Tests: 97 server tests, 67-step smoke test with fake `yt-dlp`, `rip` and a
fake Deezer API (including the artist graph). fake Deezer API (including the artist graph and both radio paths).
### Client ### Client
+17
View File
@@ -60,6 +60,18 @@ for arg in "$@"; do
prev="$arg" prev="$arg"
done done
if [[ " $* " == *"RDAMVM"* ]]; then
# YouTube Music's watch-playlist radio: record that it was requested.
printf '%s\n' "$*" >> "$(dirname "$0")/../yt-radio.log"
cat <<'JSON'
{"entries":[
{"id":"vid-radio","title":"Radio Artist - Radio Track","duration":210,"channel":"Radio Artist"},
{"id":"vid-rec-a","title":"Downloaded Track","artist":"New Artist","album":"New Album","duration":180,"channel":"New Artist"}
]}
JSON
exit 0
fi
if [[ " $* " == *" music.youtube.com/search "* || " $* " == *" --flat-playlist "* ]]; then if [[ " $* " == *" music.youtube.com/search "* || " $* " == *" --flat-playlist "* ]]; then
cat <<'JSON' cat <<'JSON'
{"entries":[ {"entries":[
@@ -417,6 +429,11 @@ log "Recommendations"
expect "refresh recommendations" "200" "POST" "/moods/$MOOD_ID/recommendations/refresh" expect "refresh recommendations" "200" "POST" "/moods/$MOOD_ID/recommendations/refresh"
REC_VIDEO="$(printf '%s' "$HTTP_BODY" | python3 -c 'import json,sys;r=json.load(sys.stdin)["recommendations"];print(r[0]["externalId"] if r else "")')" REC_VIDEO="$(printf '%s' "$HTTP_BODY" | python3 -c 'import json,sys;r=json.load(sys.stdin)["recommendations"];print(r[0]["externalId"] if r else "")')"
wait_for 90 "recommendations download" 'd["mood"]["recommendationReady"] >= 1' "/moods/$MOOD_ID" wait_for 90 "recommendations download" 'd["mood"]["recommendationReady"] >= 1' "/moods/$MOOD_ID"
if [[ -f "$WORK/yt-radio.log" ]]; then
pass "YouTube Music radio discovery"
else
fail "YouTube Music radio discovery" "no RDAMVM playlist was requested"
fi
expect "add a recommendation" "200" "POST" "/moods/$MOOD_ID/recommendations/$REC_VIDEO/add" expect "add a recommendation" "200" "POST" "/moods/$MOOD_ID/recommendations/$REC_VIDEO/add"
SECOND_REC="$(printf '%s' "$HTTP_BODY" | python3 -c 'import json,sys;r=json.load(sys.stdin)["detail"]["recommendations"];print(r[0]["externalId"] if r else "")')" SECOND_REC="$(printf '%s' "$HTTP_BODY" | python3 -c 'import json,sys;r=json.load(sys.stdin)["detail"]["recommendations"];print(r[0]["externalId"] if r else "")')"
if [[ -n "$SECOND_REC" ]]; then if [[ -n "$SECOND_REC" ]]; then
@@ -81,6 +81,11 @@ class DeezerSearch(
artistTopTracks(id, SEED_TOP_LIMIT).forEach { artistTopTracks(id, SEED_TOP_LIMIT).forEach {
add(found, it, SEED_TOP_AFFINITY) add(found, it, SEED_TOP_AFFINITY)
} }
// Deezer's own radio is the recommendation engine at its strongest:
// a Daft Punk radio mixes Daft Punk with MGMT, Caribou, Jamiroquai.
artistRadio(id, SEED_RADIO_LIMIT).forEach {
add(found, it, RADIO_AFFINITY)
}
} }
val related = LinkedHashSet<Long>() val related = LinkedHashSet<Long>()
@@ -176,6 +181,11 @@ class DeezerSearch(
.filter { it > 0 } .filter { it > 0 }
}.getOrDefault(emptyList()) }.getOrDefault(emptyList())
private fun artistRadio(id: Long, limit: Int): List<TrackCandidate> = runCatching {
val body = get("$apiBase/artist/$id/radio?limit=$limit&output=json")
json.decodeFromString<DeezerTracksResponse>(body).data.mapNotNull { it.toCandidate() }
}.getOrDefault(emptyList())
private fun artistTopTracks(id: Long, limit: Int): List<TrackCandidate> = runCatching { private fun artistTopTracks(id: Long, limit: Int): List<TrackCandidate> = runCatching {
val body = get("$apiBase/artist/$id/top?limit=$limit&output=json") val body = get("$apiBase/artist/$id/top?limit=$limit&output=json")
json.decodeFromString<DeezerTracksResponse>(body).data.mapNotNull { it.toCandidate() } json.decodeFromString<DeezerTracksResponse>(body).data.mapNotNull { it.toCandidate() }
@@ -270,6 +280,7 @@ class DeezerSearch(
const val MAX_RELATED_ARTISTS = 6 const val MAX_RELATED_ARTISTS = 6
const val MAX_SEED_ALBUMS = 2 const val MAX_SEED_ALBUMS = 2
const val SEED_TOP_LIMIT = 25 const val SEED_TOP_LIMIT = 25
const val SEED_RADIO_LIMIT = 40
const val RELATED_TOP_LIMIT = 10 const val RELATED_TOP_LIMIT = 10
const val ALBUM_TRACK_LIMIT = 40 const val ALBUM_TRACK_LIMIT = 40
const val ARTIST_LOOKUP_LIMIT = 5 const val ARTIST_LOOKUP_LIMIT = 5
@@ -277,6 +288,8 @@ class DeezerSearch(
/** A hit by an artist already in the mood; the scorer adds the rest. */ /** A hit by an artist already in the mood; the scorer adds the rest. */
const val SEED_TOP_AFFINITY = 0.5 const val SEED_TOP_AFFINITY = 0.5
/** A track Deezer itself puts in that artist's radio. */
const val RADIO_AFFINITY = 2.0
/** A hit by an artist Deezer considers similar to the collection. */ /** A hit by an artist Deezer considers similar to the collection. */
const val RELATED_AFFINITY = 1.5 const val RELATED_AFFINITY = 1.5
/** Another track from an album already in the mood. */ /** Another track from an album already in the mood. */
@@ -45,6 +45,8 @@ class DiscoveryService(
val seeds = DiscoverySeeds( val seeds = DiscoverySeeds(
artists = artists, artists = artists,
albums = albums, albums = albums,
// Only songs from the mood's own catalog can seed its radio.
trackIds = radioSeedIds(collection, mood.source, RADIO_SEEDS),
hint = mood.query, hint = mood.query,
genres = mood.genres, genres = mood.genres,
name = mood.name, name = mood.name,
@@ -95,14 +97,10 @@ class DiscoveryService(
return DiscoveryOutcome(0, error = message) return DiscoveryOutcome(0, error = message)
} }
val candidates = LinkedHashMap<String, TrackCandidate>() val candidates = mergeCandidates(
(searches.mapNotNull { it.getOrNull() }.flatten() + discovered).forEach { candidate -> (searches.mapNotNull { it.getOrNull() }.flatten() + discovered)
if (!isMusicLength(candidate.duration)) return@forEach .filter { isMusicLength(it.duration) },
val existing = candidates[candidate.externalId] )
if (existing == null || existing.affinity < candidate.affinity) {
candidates[candidate.externalId] = candidate
}
}
if (candidates.isEmpty()) { if (candidates.isEmpty()) {
val message = "${catalog.source} returned no usable results" val message = "${catalog.source} returned no usable results"
@@ -118,6 +116,10 @@ class DiscoveryService(
.toSet() .toSet()
val moodText = listOfNotNull(mood.query, mood.description, mood.name).joinToString(" ") val moodText = listOfNotNull(mood.query, mood.description, mood.name).joinToString(" ")
// A different upload of a song already in the mood is still a duplicate.
val collectionRecordings = collection
.map { recordingKey(it.artist, it.title) }
.toSet()
val artistWeights = artists val artistWeights = artists
.associate { matchKey(it.name) to it.songs.toDouble() } .associate { matchKey(it.name) to it.songs.toDouble() }
.filterKeys { it.isNotEmpty() } .filterKeys { it.isNotEmpty() }
@@ -129,6 +131,7 @@ class DiscoveryService(
val ranked = rankCandidates( val ranked = rankCandidates(
candidates.values candidates.values
.filterNot { it.externalId in excluded } .filterNot { it.externalId in excluded }
.filterNot { recordingKey(it.artist, it.title) in collectionRecordings }
.map { candidate -> .map { candidate ->
candidate to scoreCandidate(candidate, artistWeights, albumKeys, moodText) candidate to scoreCandidate(candidate, artistWeights, albumKeys, moodText)
}, },
@@ -175,6 +178,7 @@ class DiscoveryService(
/** The graph expands far more than a single search page does. */ /** The graph expands far more than a single search page does. */
const val DISCOVERY_LIMIT = 200 const val DISCOVERY_LIMIT = 200
const val RADIO_SEEDS = 2
const val CANDIDATE_POOL = 60 const val CANDIDATE_POOL = 60
} }
} }
@@ -216,6 +220,20 @@ internal fun artistsByFrequency(collection: List<SongRecord>): List<SeedArtist>
) )
.map { SeedArtist(it.key, it.value) } .map { SeedArtist(it.key, it.value) }
/**
* The most recent `(source, externalId)` songs of the mood's catalog, one per
* artist: a radio seeded by three tracks of the same artist adds nothing.
*/
internal fun radioSeedIds(
collection: List<SongRecord>,
source: String,
limit: Int,
): List<String> = collection.asReversed()
.filter { it.source == source }
.distinctBy { matchKey(it.artist) }
.take(limit)
.map { it.externalId }
/** Distinct albums of the collection, deduplicated loosely. */ /** Distinct albums of the collection, deduplicated loosely. */
internal fun albumsOf(collection: List<SongRecord>): List<SeedAlbum> = internal fun albumsOf(collection: List<SongRecord>): List<SeedAlbum> =
collection.mapNotNull { song -> collection.mapNotNull { song ->
@@ -223,6 +241,39 @@ internal fun albumsOf(collection: List<SongRecord>): List<SeedAlbum> =
SeedAlbum(album, song.artist) SeedAlbum(album, song.artist)
}.distinctBy { "${matchKey(it.title)}|${matchKey(it.artist)}" } }.distinctBy { "${matchKey(it.title)}|${matchKey(it.artist)}" }
/**
* Keeps one copy per recording. A radio and an album endpoint may return the
* same song twice, and two uploads of one song only differ by external id; the
* copy with the stronger catalog affinity wins.
*/
internal fun mergeCandidates(
items: Iterable<TrackCandidate>,
): LinkedHashMap<String, TrackCandidate> {
val byId = LinkedHashMap<String, TrackCandidate>()
val idForRecording = HashMap<String, String>()
items.forEach { candidate ->
val recording = recordingKey(candidate.artist, candidate.title)
val currentId = idForRecording[recording]
val current = currentId?.let(byId::get)
when {
current == null -> {
byId[candidate.externalId] = candidate
idForRecording[recording] = candidate.externalId
}
candidate.affinity > current.affinity -> {
byId.remove(current.externalId)
byId[candidate.externalId] = candidate
idForRecording[recording] = candidate.externalId
}
}
}
return byId
}
/** Identity of a recording, independent of the catalog's track id. */
internal fun recordingKey(artist: String, title: String): String =
"${matchKey(artist)}|${matchKey(title)}"
/** /**
* Picks the best [limit] candidates while penalising an artist that keeps * Picks the best [limit] candidates while penalising an artist that keeps
* winning: without it, a mood with three songs by one artist would fill every * winning: without it, a mood with three songs by one artist would fill every
@@ -33,11 +33,16 @@ data class SeedAlbum(val title: String, val artist: String)
data class DiscoverySeeds( data class DiscoverySeeds(
val artists: List<SeedArtist> = emptyList(), val artists: List<SeedArtist> = emptyList(),
val albums: List<SeedAlbum> = emptyList(), val albums: List<SeedAlbum> = emptyList(),
/**
* External ids of collection songs that belong to this catalog, most recent
* first. They seed the catalog's own "radio"/play-next queue.
*/
val trackIds: List<String> = emptyList(),
val hint: String? = null, val hint: String? = null,
val genres: List<String> = emptyList(), val genres: List<String> = emptyList(),
val name: String = "", val name: String = "",
) { ) {
fun isEmpty(): Boolean = artists.isEmpty() && albums.isEmpty() fun isEmpty(): Boolean = artists.isEmpty() && albums.isEmpty() && trackIds.isEmpty()
} }
/** A music catalog that can be searched and can resolve a pasted URL. */ /** A music catalog that can be searched and can resolve a pasted URL. */
@@ -0,0 +1,36 @@
package win.taptap.musicmanager.discovery
private const val NOISE_WORDS =
"official|lyrics?|audio|video|visuali[sz]er|mv|hd|4k|remaster(?:ed)?"
private val TITLE_NOISE = Regex(
"\\((?:$NOISE_WORDS)[^)]*\\)" +
"|\\[(?:$NOISE_WORDS)[^\\]]*\\]" +
"|\\s*\\|\\s*(?:$NOISE_WORDS).*$",
RegexOption.IGNORE_CASE,
)
private val WHITESPACE = Regex("\\s+")
/**
* Removes the upload noise that YouTube titles carry ("(Official Video)",
* "[4K Remaster]", "| Lyrics"), so a search hit, a download tag and an existing
* collection row compare as the same song.
*/
internal fun cleanTrackTitle(rawTitle: String): String =
rawTitle.replace(TITLE_NOISE, "")
.replace(WHITESPACE, " ")
.trim()
.ifEmpty { rawTitle.trim() }
/**
* `"Daft Punk - Get Lucky"` tagged by the plain YouTube extractor still has the
* artist in the title. Strip it when it matches the tagged artist.
*/
internal fun stripArtistPrefix(title: String, artist: String): String {
val name = artist.trim()
if (name.isEmpty() || name.equals("Unknown artist", ignoreCase = true)) return title
val prefix = "$name - "
if (!title.startsWith(prefix, ignoreCase = true)) return title
return title.substring(prefix.length).trim().ifEmpty { title }
}
@@ -77,6 +77,51 @@ class YtDlpSearch(
return parseYtSearchOutput(output) return parseYtSearchOutput(output)
} }
/**
* YouTube Music's own radio queue: `list=RDAMVM<videoId>` is the mix the
* player builds when a song ends. It needs no account and works through the
* same yt-dlp call as search, so a YouTube mood gets real recommendations
* instead of only text matches.
*/
override fun discover(seeds: DiscoverySeeds, limit: Int): List<TrackCandidate> {
val ids = seeds.trackIds.take(MAX_RADIO_SEEDS)
if (ids.isEmpty()) return emptyList()
val found = LinkedHashMap<String, TrackCandidate>()
ids.forEach { id ->
runCatching { radio(id) }.getOrDefault(emptyList()).forEach { candidate ->
if (isMusicLength(candidate.duration)) {
found.putIfAbsent(
candidate.externalId,
candidate.copy(affinity = RADIO_AFFINITY),
)
}
}
}
return found.values.take(limit)
}
private fun radio(videoId: String): List<TrackCandidate> {
val output = run(
listOf(
"--flat-playlist",
"--playlist-end",
RADIO_LIMIT.toString(),
"--dump-single-json",
"--no-warnings",
"--ignore-errors",
"--ignore-config",
"--no-cache-dir",
"--socket-timeout",
"10",
"--extractor-retries",
"1",
"https://music.youtube.com/watch?v=$videoId&list=RDAMVM$videoId",
),
timeoutSeconds,
)
return parseYtSearchOutput(output)
}
override fun resolve(url: String): TrackCandidate? { override fun resolve(url: String): TrackCandidate? {
val trimmed = url.trim() val trimmed = url.trim()
if (trimmed.isEmpty()) return null if (trimmed.isEmpty()) return null
@@ -133,6 +178,9 @@ class YtDlpSearch(
private companion object { private companion object {
const val MAX_RESULTS = 25 const val MAX_RESULTS = 25
const val MAX_RADIO_SEEDS = 2
const val RADIO_LIMIT = 25
const val RADIO_AFFINITY = 2.0
/** /**
* YouTube Music's "Songs" filter. Without it the result page also * YouTube Music's "Songs" filter. Without it the result page also
@@ -179,7 +227,7 @@ private fun JsonObject.toCandidate(): TrackCandidate? {
return TrackCandidate( return TrackCandidate(
source = MusicSource.YOUTUBE, source = MusicSource.YOUTUBE,
externalId = externalId, externalId = externalId,
title = cleanTitle(rawTitle), title = cleanTrackTitle(rawTitle),
artist = musicArtist, artist = musicArtist,
album = string("album")?.takeIf { it.isRealValue() }, album = string("album")?.takeIf { it.isRealValue() },
duration = number("duration")?.toInt(), duration = number("duration")?.toInt(),
@@ -216,23 +264,9 @@ private fun String.isRealValue(): Boolean =
private const val MAX_VIDEO_ID_LENGTH = 11 private const val MAX_VIDEO_ID_LENGTH = 11
private const val NOISE_WORDS = "official|lyrics?|audio|video|visuali[sz]er|mv|hd|4k|remaster(?:ed)?"
private val TITLE_NOISE = Regex(
"\\((?:$NOISE_WORDS)[^)]*\\)" +
"|\\[(?:$NOISE_WORDS)[^\\]]*\\]" +
"|\\s*\\|\\s*(?:$NOISE_WORDS).*$",
RegexOption.IGNORE_CASE,
)
private val WHITESPACE = Regex("\\s+")
private fun cleanTitle(rawTitle: String): String =
rawTitle.replace(TITLE_NOISE, "").replace(WHITESPACE, " ").trim().ifEmpty { rawTitle.trim() }
/** "Artist - Track (Official Video)" becomes ("Artist", "Track"). */ /** "Artist - Track (Official Video)" becomes ("Artist", "Track"). */
internal fun splitYtTitle(rawTitle: String, channel: String): Pair<String, String> { internal fun splitYtTitle(rawTitle: String, channel: String): Pair<String, String> {
val cleaned = cleanTitle(rawTitle) val cleaned = cleanTrackTitle(rawTitle)
for (separator in listOf(" - ", " – ", " — ")) { for (separator in listOf(" - ", " – ", " — ")) {
val index = cleaned.indexOf(separator) val index = cleaned.indexOf(separator)
if (index > 0) { if (index > 0) {
@@ -8,6 +8,8 @@ import java.util.logging.Logger
import org.jaudiotagger.audio.AudioFileIO import org.jaudiotagger.audio.AudioFileIO
import org.jaudiotagger.tag.FieldKey import org.jaudiotagger.tag.FieldKey
import org.jaudiotagger.tag.Tag import org.jaudiotagger.tag.Tag
import win.taptap.musicmanager.discovery.cleanTrackTitle
import win.taptap.musicmanager.discovery.stripArtistPrefix
data class ExtractedSong( data class ExtractedSong(
val title: String, val title: String,
@@ -52,7 +54,7 @@ class Mp3TagExtractor {
val fallbackTitle = file.fileName.toString().removeSuffix(".mp3").removeSuffix(".MP3") val fallbackTitle = file.fileName.toString().removeSuffix(".mp3").removeSuffix(".MP3")
val title = tag.values(FieldKey.TITLE).firstOrNull() ?: fallbackTitle val rawTitle = tag.values(FieldKey.TITLE).firstOrNull() ?: fallbackTitle
val artists = (tag.values(FieldKey.ARTIST) + tag.values(FieldKey.ARTISTS)).distinct() val artists = (tag.values(FieldKey.ARTIST) + tag.values(FieldKey.ARTISTS)).distinct()
val albumTitle = tag.values(FieldKey.ALBUM).firstOrNull() val albumTitle = tag.values(FieldKey.ALBUM).firstOrNull()
val albumArtist = (tag.values(FieldKey.ALBUM_ARTIST) + tag.values(FieldKey.ALBUM_ARTISTS)) val albumArtist = (tag.values(FieldKey.ALBUM_ARTIST) + tag.values(FieldKey.ALBUM_ARTISTS))
@@ -62,7 +64,11 @@ class Mp3TagExtractor {
artists.isNotEmpty() -> artists.joinToString(", ") artists.isNotEmpty() -> artists.joinToString(", ")
albumArtist != null -> albumArtist albumArtist != null -> albumArtist
else -> "Unknown artist" else -> "Unknown artist"
} }.removeSuffix(" - Topic").trim()
// The plain YouTube extractor embeds the raw upload title, which keeps
// the artist prefix and the "(Official Video)" noise in the store.
val title = stripArtistPrefix(cleanTrackTitle(rawTitle), artist)
val duration = header.trackLength.takeIf { it > 0 } val duration = header.trackLength.takeIf { it > 0 }
?: header.preciseTrackLength.toInt().coerceAtLeast(0) ?: header.preciseTrackLength.toInt().coerceAtLeast(0)
@@ -36,6 +36,13 @@ class DeezerDiscoverTest {
"artist":{"name":"Daft Punk"}, "artist":{"name":"Daft Punk"},
"album":{"title":"Random Access Memories"}}]}""", "album":{"title":"Random Access Memories"}}]}""",
) )
"/artist/27/radio" ->
ok(
"""{"data":[
{"id":5,"title":"Kids","duration":303,"readable":true,
"artist":{"name":"MGMT"},
"album":{"title":"Oracular Spectacular"}}]}""",
)
"/artist/27/related" -> "/artist/27/related" ->
ok("""{"data":[{"id":99,"name":"Daft Punk Tribute"}]}""") ok("""{"data":[{"id":99,"name":"Daft Punk Tribute"}]}""")
"/artist/99/top" -> "/artist/99/top" ->
@@ -58,8 +65,11 @@ class DeezerDiscoverTest {
val found = catalog(server).discover(seeds(), limit = 50).associateBy { it.externalId } val found = catalog(server).discover(seeds(), limit = 50).associateBy { it.externalId }
// The 15-minute track is not music-length and never reaches the pool. // The 15-minute track is not music-length and never reaches the pool.
assertEquals(setOf("1", "3", "4"), found.keys) assertEquals(setOf("1", "3", "4", "5"), found.keys)
assertEquals(0.5, found.getValue("1").affinity) assertEquals(0.5, found.getValue("1").affinity)
// Deezer's own radio is the strongest affinity signal we can get.
assertEquals(2.0, found.getValue("5").affinity)
assertEquals("MGMT", found.getValue("5").artist)
assertEquals(1.5, found.getValue("3").affinity) assertEquals(1.5, found.getValue("3").affinity)
assertEquals(1.0, found.getValue("4").affinity) assertEquals(1.0, found.getValue("4").affinity)
// `/album/{id}/tracks` has no album object; the search result fills it in. // `/album/{id}/tracks` has no album object; the search result fills it in.
@@ -268,6 +268,34 @@ class DiscoveryServiceTest {
assertEquals(1.0, sameAlbum - unrelated) assertEquals(1.0, sameAlbum - unrelated)
} }
@Test
fun `merge keeps one copy of a recording and prefers the stronger affinity`() {
val album = TrackCandidate(
source = MusicSource.DEEZER,
externalId = "album-copy",
title = "Get Lucky (Radio Edit)",
artist = "Daft Punk",
duration = 248,
)
val radio = album.copy(externalId = "radio-copy", affinity = 2.0)
val cover = TrackCandidate(
source = MusicSource.DEEZER,
externalId = "cover",
title = "Get Lucky",
artist = "Daft Punk Tribute",
duration = 240,
)
val merged = mergeCandidates(listOf(album, cover, radio))
assertEquals(setOf("radio-copy", "cover"), merged.keys)
assertEquals(2.0, merged.getValue("radio-copy").affinity)
assertEquals(
recordingKey("Daft Punk", "Get Lucky"),
recordingKey("Daft Punk", "Get Lucky (Radio Edit)"),
)
}
@Test @Test
fun `ranking spreads picks across artists instead of repeating one`() { fun `ranking spreads picks across artists instead of repeating one`() {
val lead = (1..12).map { val lead = (1..12).map {
@@ -348,6 +376,25 @@ class DiscoveryServiceTest {
) )
} }
@Test
fun `radio seeds use the newest song per artist of the same catalog`() {
val collection = listOf(
song("Alpha", externalId = "yt-1", source = MusicSource.YOUTUBE),
song("Beta", externalId = "yt-2", source = MusicSource.YOUTUBE),
song("Alpha", externalId = "yt-3", source = MusicSource.YOUTUBE),
song("Gamma", externalId = "dz-1", source = MusicSource.DEEZER),
)
assertEquals(
listOf("yt-3", "yt-2"),
radioSeedIds(collection, MusicSource.YOUTUBE, limit = 2),
)
assertEquals(
listOf("dz-1"),
radioSeedIds(collection, MusicSource.DEEZER, limit = 2),
)
}
@Test @Test
fun `albums are collected once per title and artist`() { fun `albums are collected once per title and artist`() {
val collection = listOf( val collection = listOf(
@@ -366,10 +413,15 @@ class DiscoveryServiceTest {
) )
} }
private fun song(artist: String, album: String? = "Album") = SongRecord( private fun song(
id = "sg-$artist-$album", artist: String,
source = MusicSource.DEEZER, album: String? = "Album",
externalId = "1", source: String = MusicSource.DEEZER,
externalId: String = "1",
) = SongRecord(
id = "sg-$artist-$album-$source-$externalId",
source = source,
externalId = externalId,
title = "Title", title = "Title",
artist = artist, artist = artist,
album = album, album = album,
@@ -43,6 +43,24 @@ class Mp3TagExtractorTest {
assertEquals("audio/mpeg", song.contentType) assertEquals("audio/mpeg", song.contentType)
} }
@Test
fun `cleans the raw YouTube title prefix and upload noise`() {
TestAudio.assumeFfmpeg()
val file = tempDir.resolve("video-id/rick.mp3")
TestAudio.createMp3(
file,
metadata = mapOf(
"title" to "Rick Astley - Never Gonna Give You Up (Official Video) (4K Remaster)",
"artist" to "Rick Astley - Topic",
),
)
val song = assertNotNull(Mp3TagExtractor().extract(file))
assertEquals("Never Gonna Give You Up", song.title)
assertEquals("Rick Astley", song.artist)
}
@Test @Test
fun `falls back to the file name and a generic artist`() { fun `falls back to the file name and a generic artist`() {
TestAudio.assumeFfmpeg() TestAudio.assumeFfmpeg()