feat: search and download from YouTube Music
- search music.youtube.com with the Songs filter and resolve each hit so results carry canonical artist, album and duration - fall back to plain ytsearch when the Music search fails - download the Music watch URL first and the regular YouTube URL second - parse explicit artist fields and drop channel/album/playlist entries - update the smoke stub, docs and tests
This commit is contained in:
1 parent
61d8b58223
commit
515b9bbe1b
6 files changed
+215
-29
No files matched your search
@@ -1,9 +1,9 @@
|
|||||||
# AudioManager
|
# AudioManager
|
||||||
|
|
||||||
A self-hosted, mood-first music server. You create private moods, add tracks
|
A self-hosted, mood-first music server. You create private moods, add tracks
|
||||||
from YouTube search or a pasted URL, and the server downloads them into a shared
|
from YouTube Music search or a pasted URL, and the server downloads them into a
|
||||||
media store. Recommendations are discovered and downloaded automatically so a
|
shared media store. Recommendations are discovered and downloaded automatically
|
||||||
mood keeps playing without a mounted music library.
|
so a mood keeps playing without a mounted music library.
|
||||||
|
|
||||||
- Server: Kotlin, Ktor, SQLite (Exposed + Flyway), `yt-dlp`, `ffmpeg`.
|
- Server: Kotlin, Ktor, SQLite (Exposed + Flyway), `yt-dlp`, `ffmpeg`.
|
||||||
- Client: Compose Multiplatform (Android, desktop, Wasm).
|
- Client: Compose Multiplatform (Android, desktop, Wasm).
|
||||||
@@ -61,8 +61,8 @@ Requirements: JDK 21, `ffmpeg`, `yt-dlp`, and (for the smoke test) `sqlite3`,
|
|||||||
The server creates `./data/media` (the media store), `./data/downloads`
|
The server creates `./data/media` (the media store), `./data/downloads`
|
||||||
(download temp files) and `./data/musicmanager.db`.
|
(download temp files) and `./data/musicmanager.db`.
|
||||||
|
|
||||||
Downloads require `yt-dlp` to be able to reach YouTube. If YouTube applies bot
|
Downloads require `yt-dlp` to be able to reach YouTube Music. If YouTube applies
|
||||||
checks, search and downloads fail with `502 search_failed` / `failed`
|
bot checks, search and downloads fail with `502 search_failed` / `failed`
|
||||||
recommendations; retry later or update `yt-dlp`.
|
recommendations; retry later or update `yt-dlp`.
|
||||||
|
|
||||||
## Configuration
|
## Configuration
|
||||||
@@ -133,10 +133,16 @@ finishes.
|
|||||||
|
|
||||||
| Method | Path | Purpose |
|
| Method | Path | Purpose |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| `GET` | `/search?q=...&limit=...` | YouTube search |
|
| `GET` | `/search?q=...&limit=...` | YouTube Music search |
|
||||||
|
|
||||||
Results longer than 10 minutes (or shorter than 30 seconds) are hidden. Each
|
Results longer than 10 minutes (or shorter than 30 seconds) are hidden. Each
|
||||||
result reports `downloaded` and `songId` when the track is already in the store.
|
result reports `downloaded`, `songId` and `inMood` when applicable.
|
||||||
|
|
||||||
|
Search uses the YouTube Music catalog (the "Songs" filter), so results are
|
||||||
|
canonical tracks with an artist, album and duration instead of arbitrary
|
||||||
|
uploads. yt-dlp has to resolve every hit to read those fields, which takes
|
||||||
|
roughly 10–15 seconds for ten results. If the Music search fails, the server
|
||||||
|
falls back to the regular YouTube search so the feature keeps working.
|
||||||
|
|
||||||
### Playback
|
### Playback
|
||||||
|
|
||||||
@@ -166,7 +172,7 @@ Ktor routes
|
|||||||
└── SongRepository ── shared song store keyed by video id
|
└── SongRepository ── shared song store keyed by video id
|
||||||
└── RadioService ── mixes collection + ready recommendations
|
└── RadioService ── mixes collection + ready recommendations
|
||||||
└── DownloadService── single worker: yt-dlp → media store → index → attach
|
└── DownloadService── single worker: yt-dlp → media store → index → attach
|
||||||
└── DiscoveryService── yt-dlp search → hidden `new` candidates
|
└── DiscoveryService── yt-dlp YouTube Music search → hidden `new` candidates
|
||||||
└── TopUpService ── keeps active slots at recommendationCount
|
└── TopUpService ── keeps active slots at recommendationCount
|
||||||
└── MediaScanner ── rebuilds the index from the media root
|
└── MediaScanner ── rebuilds the index from the media root
|
||||||
└── MediaGcService ── refcounted file deletion
|
└── MediaGcService ── refcounted file deletion
|
||||||
@@ -188,6 +194,12 @@ depends on tags. One row and one file exist per video, no matter how many moods
|
|||||||
or users reference it. Covers are cached under `<DATA_DIR>/covers` keyed by song
|
or users reference it. Covers are cached under `<DATA_DIR>/covers` keyed by song
|
||||||
id.
|
id.
|
||||||
|
|
||||||
|
Downloads are requested from the YouTube Music player
|
||||||
|
(`music.youtube.com/watch?v=<id>`), which makes yt-dlp embed the song's artist,
|
||||||
|
album and release year into the MP3. Videos that are not on the Music host fall
|
||||||
|
back to the regular YouTube URL. Files are MP3 only
|
||||||
|
(`-x --audio-format mp3 --audio-quality 0`).
|
||||||
|
|
||||||
A file is deleted only when it has no references from `mood_songs`, an active
|
A file is deleted only when it has no references from `mood_songs`, an active
|
||||||
recommendation (`downloading`, `ready`, `added`, `failed`), or an open playback
|
recommendation (`downloading`, `ready`, `added`, `failed`), or an open playback
|
||||||
session. GC runs after removals, after rejections, after mood deletion, and
|
session. GC runs after removals, after rejections, after mood deletion, and
|
||||||
|
|||||||
@@ -13,8 +13,9 @@ Repos:
|
|||||||
|
|
||||||
- Users have **moods only**. There is no mounted music directory, no file
|
- Users have **moods only**. There is no mounted music directory, no file
|
||||||
upload and no longform/podcast content.
|
upload and no longform/podcast content.
|
||||||
- All content enters the server by downloading from YouTube through `yt-dlp`.
|
- All content enters the server by downloading from YouTube Music through
|
||||||
- Search is a remote YouTube-Music-style search. Results are shown before
|
`yt-dlp` (regular YouTube is only a fallback host).
|
||||||
|
- Search is a remote YouTube Music search. Results are shown before
|
||||||
anything is added.
|
anything is added.
|
||||||
- A mood contains:
|
- A mood contains:
|
||||||
- **Songs** — the user's collection. Added from search, from a recommendation,
|
- **Songs** — the user's collection. Added from search, from a recommendation,
|
||||||
@@ -104,7 +105,7 @@ Current files and responsibilities, and what happens to them:
|
|||||||
| `db/RadioRepository.kt` | sessions, items, affinity | keep, minus folder/library assumptions |
|
| `db/RadioRepository.kt` | sessions, items, affinity | keep, minus folder/library assumptions |
|
||||||
| `db/DownloadRepository.kt` | per-user/mood jobs, longform progress | rewrite job model; move longform repository out |
|
| `db/DownloadRepository.kt` | per-user/mood jobs, longform progress | rewrite job model; move longform repository out |
|
||||||
| `discovery/DiscoveryService.kt` | seeds + genres/query -> candidates, review | same inputs from collection; store `new` candidates; no `matchedSongId` library matching |
|
| `discovery/DiscoveryService.kt` | seeds + genres/query -> candidates, review | same inputs from collection; store `new` candidates; no `matchedSongId` library matching |
|
||||||
| `discovery/YtSearch.kt` | yt-dlp flat search | keep; change music length to 45..600s; add URL resolve |
|
| `discovery/YtSearch.kt` | yt-dlp flat search | YouTube Music songs search with resolved metadata; regular `ytsearch` fallback; add URL resolve |
|
||||||
| `download/DownloadService.kt` | accept -> import folder -> scan -> seed | index directly into media store; attach song/recommendation; GC on failure |
|
| `download/DownloadService.kt` | accept -> import folder -> scan -> seed | index directly into media store; attach song/recommendation; GC on failure |
|
||||||
| `download/YtDlpDownloader.kt` | MP3 + embedded metadata | keep; target `<videoId>/<title>.mp3` |
|
| `download/YtDlpDownloader.kt` | MP3 + embedded metadata | keep; target `<videoId>/<title>.mp3` |
|
||||||
| `scanner/MediaScanner.kt` | walks folder config | walks `MEDIA_DIR`; derive video id from path |
|
| `scanner/MediaScanner.kt` | walks folder config | walks `MEDIA_DIR`; derive video id from path |
|
||||||
@@ -630,11 +631,16 @@ Phases 1-8 are implemented in both repositories.
|
|||||||
`downloading: true` when a new video was queued.
|
`downloading: true` when a new video was queued.
|
||||||
- `AppConfig` is `port`, `dataDir`, `usersFile`, `ffmpegPath`, `ytDlpPath`,
|
- `AppConfig` is `port`, `dataDir`, `usersFile`, `ffmpegPath`, `ytDlpPath`,
|
||||||
`mediaDir`, `downloadDir`. `LibraryConfig` and the folder APIs are deleted.
|
`mediaDir`, `downloadDir`. `LibraryConfig` and the folder APIs are deleted.
|
||||||
|
- `YtDlpSearch` searches `music.youtube.com` with the Songs filter and resolves
|
||||||
|
each hit, so results carry the canonical artist, album and duration.
|
||||||
|
`YtDlpDownloader` requests the Music watch URL first and falls back to
|
||||||
|
`www.youtube.com` for videos that are not on the Music host. If the Music
|
||||||
|
search itself fails, search falls back to `ytsearch`.
|
||||||
- Background loops: download worker (continuous), recommendation top-up
|
- Background loops: download worker (continuous), recommendation top-up
|
||||||
(hourly), media GC (hourly), startup scan.
|
(hourly), media GC (hourly), startup scan.
|
||||||
- Chart 0.3.0 uses one `/data` PVC, `MEDIA_DIR=/data/media`, no hostPath, no
|
- Chart 0.3.0 uses one `/data` PVC, `MEDIA_DIR=/data/media`, no hostPath, no
|
||||||
library ConfigMap and no init container.
|
library ConfigMap and no init container.
|
||||||
- Tests: 73 server tests, 48-step smoke test with a fake `yt-dlp`.
|
- Tests: 75 server tests, 48-step smoke test with a fake `yt-dlp`.
|
||||||
|
|
||||||
### Client
|
### Client
|
||||||
|
|
||||||
|
|||||||
+6
-6
@@ -59,14 +59,14 @@ for arg in "$@"; do
|
|||||||
prev="$arg"
|
prev="$arg"
|
||||||
done
|
done
|
||||||
|
|
||||||
if [[ " $* " == *" --flat-playlist "* ]]; then
|
if [[ " $* " == *" music.youtube.com/search "* || " $* " == *" --flat-playlist "* ]]; then
|
||||||
cat <<'JSON'
|
cat <<'JSON'
|
||||||
{"entries":[
|
{"entries":[
|
||||||
{"id":"vid-seed-1","title":"Seed Artist - Seed One","duration":180,"channel":"Seed Artist"},
|
{"id":"vid-seed-1","title":"Seed One","artist":"Seed Artist","album":"Seed Album","duration":180,"channel":"Seed Artist"},
|
||||||
{"id":"vid-rec-a","title":"New Artist - Downloaded Track","duration":180,"channel":"New Artist","thumbnail":"https://example/a.jpg"},
|
{"id":"vid-rec-a","title":"Downloaded Track","artist":"New Artist","album":"New Album","duration":180,"channel":"New Artist","thumbnail":"https://example/a.jpg"},
|
||||||
{"id":"vid-rec-b","title":"Other Artist - Another Track","duration":200,"channel":"Other Artist"},
|
{"id":"vid-rec-b","title":"Another Track","artist":"Other Artist","album":"Other Album","duration":200,"channel":"Other Artist"},
|
||||||
{"id":"vid-long","title":"Long Set","duration":900,"channel":"DJ"},
|
{"id":"vid-long","title":"Long Set","artist":"DJ","duration":900,"channel":"DJ"},
|
||||||
{"id":"vid-short","title":"Interlude","duration":10,"channel":"DJ"}
|
{"id":"vid-short","title":"Interlude","artist":"DJ","duration":10,"channel":"DJ"}
|
||||||
]}
|
]}
|
||||||
JSON
|
JSON
|
||||||
exit 0
|
exit 0
|
||||||
|
|||||||
@@ -1,5 +1,6 @@
|
|||||||
package win.taptap.musicmanager.discovery
|
package win.taptap.musicmanager.discovery
|
||||||
|
|
||||||
|
import java.net.URLEncoder
|
||||||
import java.util.concurrent.TimeUnit
|
import java.util.concurrent.TimeUnit
|
||||||
import kotlin.concurrent.thread
|
import kotlin.concurrent.thread
|
||||||
import kotlinx.serialization.json.Json
|
import kotlinx.serialization.json.Json
|
||||||
@@ -9,7 +10,7 @@ import kotlinx.serialization.json.contentOrNull
|
|||||||
import kotlinx.serialization.json.jsonArray
|
import kotlinx.serialization.json.jsonArray
|
||||||
import kotlinx.serialization.json.jsonObject
|
import kotlinx.serialization.json.jsonObject
|
||||||
|
|
||||||
/** One YouTube search hit, already split into an artist and a track title. */
|
/** One YouTube Music search hit, already split into an artist and a track title. */
|
||||||
data class YtCandidate(
|
data class YtCandidate(
|
||||||
val videoId: String,
|
val videoId: String,
|
||||||
val title: String,
|
val title: String,
|
||||||
@@ -29,25 +30,63 @@ fun interface YtResolver {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Talks to YouTube through `yt-dlp`. Flat playlist output avoids resolving every
|
* Talks to YouTube through `yt-dlp`, preferring the YouTube Music catalog.
|
||||||
* video, which keeps a discovery run to a single request per query.
|
*
|
||||||
|
* The music search returns canonical songs with an artist, album and duration,
|
||||||
|
* so discovery does not pick arbitrary user uploads, live rips or compilations.
|
||||||
|
* Each hit has to be resolved (a flat playlist only exposes ids and titles),
|
||||||
|
* which is why a search is slower than the plain YouTube search. `ytsearch` is
|
||||||
|
* kept as a fallback for bot checks or extractor changes.
|
||||||
*/
|
*/
|
||||||
class YtDlpSearch(
|
class YtDlpSearch(
|
||||||
private val executable: String,
|
private val executable: String,
|
||||||
private val timeoutSeconds: Long = 30,
|
private val timeoutSeconds: Long = 60,
|
||||||
) : YtSearch, YtResolver {
|
) : YtSearch, YtResolver {
|
||||||
|
|
||||||
override fun search(query: String, limit: Int): List<YtCandidate> {
|
override fun search(query: String, limit: Int): List<YtCandidate> {
|
||||||
|
val count = limit.coerceIn(1, MAX_RESULTS)
|
||||||
|
|
||||||
|
// yt-dlp resolves every entry of the search result page, so keep the
|
||||||
|
// retry counts low: a bot check must fail in seconds, not minutes.
|
||||||
|
val music = try {
|
||||||
|
run(
|
||||||
|
listOf(
|
||||||
|
"--playlist-end",
|
||||||
|
count.toString(),
|
||||||
|
"--dump-single-json",
|
||||||
|
"--no-warnings",
|
||||||
|
"--ignore-errors",
|
||||||
|
"--ignore-config",
|
||||||
|
"--no-cache-dir",
|
||||||
|
"--socket-timeout",
|
||||||
|
"10",
|
||||||
|
"--extractor-retries",
|
||||||
|
"1",
|
||||||
|
"--retries",
|
||||||
|
"2",
|
||||||
|
musicSearchUrl(query),
|
||||||
|
),
|
||||||
|
timeoutSeconds,
|
||||||
|
)
|
||||||
|
} catch (e: Exception) {
|
||||||
|
null
|
||||||
|
}
|
||||||
|
val fromMusic = music?.let(::parseYtSearchOutput).orEmpty()
|
||||||
|
if (fromMusic.isNotEmpty()) return fromMusic
|
||||||
|
|
||||||
val output = run(
|
val output = run(
|
||||||
listOf(
|
listOf(
|
||||||
"--flat-playlist",
|
"--flat-playlist",
|
||||||
"--dump-single-json",
|
"--dump-single-json",
|
||||||
"--no-warnings",
|
"--no-warnings",
|
||||||
"--ignore-errors",
|
"--ignore-errors",
|
||||||
|
"--ignore-config",
|
||||||
"--no-cache-dir",
|
"--no-cache-dir",
|
||||||
"--socket-timeout",
|
"--socket-timeout",
|
||||||
"10",
|
"10",
|
||||||
"ytsearch${limit.coerceIn(1, 25)}:$query",
|
"--extractor-retries",
|
||||||
|
"1",
|
||||||
|
"ytsearch$count:$query",
|
||||||
),
|
),
|
||||||
timeoutSeconds,
|
timeoutSeconds,
|
||||||
)
|
)
|
||||||
@@ -62,6 +101,7 @@ class YtDlpSearch(
|
|||||||
"--dump-single-json",
|
"--dump-single-json",
|
||||||
"--no-playlist",
|
"--no-playlist",
|
||||||
"--no-warnings",
|
"--no-warnings",
|
||||||
|
"--ignore-config",
|
||||||
"--no-cache-dir",
|
"--no-cache-dir",
|
||||||
"--socket-timeout",
|
"--socket-timeout",
|
||||||
"10",
|
"10",
|
||||||
@@ -106,6 +146,22 @@ class YtDlpSearch(
|
|||||||
}
|
}
|
||||||
return output.toString()
|
return output.toString()
|
||||||
}
|
}
|
||||||
|
|
||||||
|
private companion object {
|
||||||
|
const val MAX_RESULTS = 25
|
||||||
|
|
||||||
|
/**
|
||||||
|
* YouTube Music's "Songs" filter. Without it the result page also
|
||||||
|
* contains channels, albums and playlists, and yt-dlp would spend most
|
||||||
|
* of the time resolving those instead of tracks.
|
||||||
|
*/
|
||||||
|
const val MUSIC_SONGS_FILTER = "EgWKAQIIAWoKEAkQBRAKEAMQBA%3D%3D"
|
||||||
|
|
||||||
|
fun musicSearchUrl(query: String): String {
|
||||||
|
val encoded = URLEncoder.encode(query, Charsets.UTF_8).replace("+", "%20")
|
||||||
|
return "https://music.youtube.com/search?q=$encoded&sp=$MUSIC_SONGS_FILTER"
|
||||||
|
}
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
internal fun parseYtSearchOutput(json: String): List<YtCandidate> {
|
internal fun parseYtSearchOutput(json: String): List<YtCandidate> {
|
||||||
@@ -127,7 +183,25 @@ internal fun parseYtVideo(json: String): YtCandidate? {
|
|||||||
|
|
||||||
private fun JsonObject.toCandidate(): YtCandidate? {
|
private fun JsonObject.toCandidate(): YtCandidate? {
|
||||||
val videoId = string("id")?.takeIf { it.isNotBlank() } ?: return null
|
val videoId = string("id")?.takeIf { it.isNotBlank() } ?: return null
|
||||||
|
// Search result pages also contain channels, albums and playlists. Their
|
||||||
|
// browse ids are longer than an 11-character video id.
|
||||||
|
if (videoId.length > MAX_VIDEO_ID_LENGTH) return null
|
||||||
val rawTitle = string("title")?.takeIf { it.isNotBlank() } ?: return null
|
val rawTitle = string("title")?.takeIf { it.isNotBlank() } ?: return null
|
||||||
|
|
||||||
|
val musicArtist = artistName()
|
||||||
|
if (musicArtist != null) {
|
||||||
|
// YouTube Music already separates artist and title, so do not split on
|
||||||
|
// " - " (a song title may legitimately contain a dash).
|
||||||
|
return YtCandidate(
|
||||||
|
videoId = videoId,
|
||||||
|
title = cleanTitle(rawTitle),
|
||||||
|
artist = musicArtist,
|
||||||
|
album = string("album")?.takeIf { it.isRealValue() },
|
||||||
|
duration = number("duration")?.toInt(),
|
||||||
|
thumbnail = thumbnail(),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
val channel = string("channel") ?: string("uploader").orEmpty()
|
val channel = string("channel") ?: string("uploader").orEmpty()
|
||||||
val (artist, title) = splitYtTitle(rawTitle, channel)
|
val (artist, title) = splitYtTitle(rawTitle, channel)
|
||||||
|
|
||||||
@@ -135,12 +209,27 @@ private fun JsonObject.toCandidate(): YtCandidate? {
|
|||||||
videoId = videoId,
|
videoId = videoId,
|
||||||
title = title.ifEmpty { rawTitle },
|
title = title.ifEmpty { rawTitle },
|
||||||
artist = artist.ifEmpty { "Unknown artist" },
|
artist = artist.ifEmpty { "Unknown artist" },
|
||||||
album = string("album"),
|
album = string("album")?.takeIf { it.isRealValue() },
|
||||||
duration = number("duration")?.toInt(),
|
duration = number("duration")?.toInt(),
|
||||||
thumbnail = thumbnail(),
|
thumbnail = thumbnail(),
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** Reads the explicit artist fields that resolved YouTube Music hits provide. */
|
||||||
|
private fun JsonObject.artistName(): String? {
|
||||||
|
string("artist")?.takeIf { it.isRealValue() }?.let { return it }
|
||||||
|
val artists = runCatching { get("artists")?.jsonArray }.getOrNull()
|
||||||
|
?.mapNotNull { (it as? JsonPrimitive)?.contentOrNull?.trim() }
|
||||||
|
?.filter { it.isNotEmpty() && it != "NA" }
|
||||||
|
.orEmpty()
|
||||||
|
return artists.takeIf { it.isNotEmpty() }?.joinToString(", ")
|
||||||
|
}
|
||||||
|
|
||||||
|
private fun String.isRealValue(): Boolean =
|
||||||
|
isNotBlank() && !equals("NA", ignoreCase = true) && !equals("null", ignoreCase = true)
|
||||||
|
|
||||||
|
private const val MAX_VIDEO_ID_LENGTH = 11
|
||||||
|
|
||||||
private const val NOISE_WORDS = "official|lyrics?|audio|video|visuali[sz]er|mv|hd|4k|remaster(?:ed)?"
|
private const val NOISE_WORDS = "official|lyrics?|audio|video|visuali[sz]er|mv|hd|4k|remaster(?:ed)?"
|
||||||
|
|
||||||
private val TITLE_NOISE = Regex(
|
private val TITLE_NOISE = Regex(
|
||||||
@@ -150,9 +239,14 @@ private val TITLE_NOISE = Regex(
|
|||||||
RegexOption.IGNORE_CASE,
|
RegexOption.IGNORE_CASE,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
private val WHITESPACE = Regex("\\s+")
|
||||||
|
|
||||||
|
private fun cleanTitle(rawTitle: String): String =
|
||||||
|
rawTitle.replace(TITLE_NOISE, "").replace(WHITESPACE, " ").trim().ifEmpty { rawTitle.trim() }
|
||||||
|
|
||||||
/** "Artist - Track (Official Video)" becomes ("Artist", "Track"). */
|
/** "Artist - Track (Official Video)" becomes ("Artist", "Track"). */
|
||||||
internal fun splitYtTitle(rawTitle: String, channel: String): Pair<String, String> {
|
internal fun splitYtTitle(rawTitle: String, channel: String): Pair<String, String> {
|
||||||
val cleaned = rawTitle.replace(TITLE_NOISE, "").trim()
|
val cleaned = cleanTitle(rawTitle)
|
||||||
for (separator in listOf(" - ", " – ", " — ")) {
|
for (separator in listOf(" - ", " – ", " — ")) {
|
||||||
val index = cleaned.indexOf(separator)
|
val index = cleaned.indexOf(separator)
|
||||||
if (index > 0) {
|
if (index > 0) {
|
||||||
|
|||||||
@@ -6,14 +6,20 @@ import java.util.concurrent.TimeUnit
|
|||||||
import kotlin.concurrent.thread
|
import kotlin.concurrent.thread
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Downloads a single YouTube video as an MP3 with embedded metadata, so the
|
* Downloads a single track as an MP3 with embedded metadata, so the indexer can
|
||||||
* indexer can derive artist and title from the file itself.
|
* derive artist, title, album and cover art from the file itself.
|
||||||
*/
|
*/
|
||||||
fun interface AudioDownloader {
|
fun interface AudioDownloader {
|
||||||
/** Downloads [videoId] into [targetDir] and returns the resulting MP3 file. */
|
/** Downloads [videoId] into [targetDir] and returns the resulting MP3 file. */
|
||||||
fun download(videoId: String, targetDir: Path): Path
|
fun download(videoId: String, targetDir: Path): Path
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Downloads from the YouTube Music player first, which is what a search hit
|
||||||
|
* points at: yt-dlp then exposes the song's artist, album and release year and
|
||||||
|
* `--embed-metadata` writes them into the MP3. Some videos only exist on the
|
||||||
|
* regular YouTube host, so that URL is tried once as a fallback.
|
||||||
|
*/
|
||||||
class YtDlpDownloader(
|
class YtDlpDownloader(
|
||||||
private val executable: String,
|
private val executable: String,
|
||||||
private val timeoutSeconds: Long = 600,
|
private val timeoutSeconds: Long = 600,
|
||||||
@@ -21,6 +27,17 @@ class YtDlpDownloader(
|
|||||||
|
|
||||||
override fun download(videoId: String, targetDir: Path): Path {
|
override fun download(videoId: String, targetDir: Path): Path {
|
||||||
Files.createDirectories(targetDir)
|
Files.createDirectories(targetDir)
|
||||||
|
return try {
|
||||||
|
runDownload("$MUSIC_WATCH_URL$videoId", videoId, targetDir)
|
||||||
|
} catch (e: YtDlpTimeoutException) {
|
||||||
|
throw e
|
||||||
|
} catch (e: IllegalStateException) {
|
||||||
|
clearPartials(videoId, targetDir)
|
||||||
|
runDownload("$YOUTUBE_WATCH_URL$videoId", videoId, targetDir)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private fun runDownload(url: String, videoId: String, targetDir: Path): Path {
|
||||||
val process = ProcessBuilder(
|
val process = ProcessBuilder(
|
||||||
executable,
|
executable,
|
||||||
"-x",
|
"-x",
|
||||||
@@ -33,13 +50,14 @@ class YtDlpDownloader(
|
|||||||
"--convert-thumbnails",
|
"--convert-thumbnails",
|
||||||
"jpg",
|
"jpg",
|
||||||
"--no-playlist",
|
"--no-playlist",
|
||||||
|
"--ignore-config",
|
||||||
"--no-cache-dir",
|
"--no-cache-dir",
|
||||||
"--no-progress",
|
"--no-progress",
|
||||||
"--print",
|
"--print",
|
||||||
"after_move:filepath",
|
"after_move:filepath",
|
||||||
"-o",
|
"-o",
|
||||||
targetDir.resolve("%(id)s.%(ext)s").toString(),
|
targetDir.resolve("%(id)s.%(ext)s").toString(),
|
||||||
"https://www.youtube.com/watch?v=$videoId",
|
url,
|
||||||
)
|
)
|
||||||
.redirectErrorStream(false)
|
.redirectErrorStream(false)
|
||||||
.start()
|
.start()
|
||||||
@@ -61,7 +79,7 @@ class YtDlpDownloader(
|
|||||||
process.destroyForcibly()
|
process.destroyForcibly()
|
||||||
reader.join(2_000)
|
reader.join(2_000)
|
||||||
errorReader.join(2_000)
|
errorReader.join(2_000)
|
||||||
throw IllegalStateException("yt-dlp download timed out after ${timeoutSeconds}s")
|
throw YtDlpTimeoutException("yt-dlp download timed out after ${timeoutSeconds}s")
|
||||||
}
|
}
|
||||||
reader.join(2_000)
|
reader.join(2_000)
|
||||||
errorReader.join(2_000)
|
errorReader.join(2_000)
|
||||||
@@ -79,4 +97,23 @@ class YtDlpDownloader(
|
|||||||
}
|
}
|
||||||
return file
|
return file
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** A failed first attempt may leave `.part`/`.webm` files behind. */
|
||||||
|
private fun clearPartials(videoId: String, targetDir: Path) {
|
||||||
|
runCatching {
|
||||||
|
Files.list(targetDir).use { stream ->
|
||||||
|
stream
|
||||||
|
.filter { it.fileName.toString().startsWith(videoId) }
|
||||||
|
.forEach { runCatching { Files.deleteIfExists(it) } }
|
||||||
}
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private companion object {
|
||||||
|
const val MUSIC_WATCH_URL = "https://music.youtube.com/watch?v="
|
||||||
|
const val YOUTUBE_WATCH_URL = "https://www.youtube.com/watch?v="
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/** A timeout is not retried against the other host: the network is the problem. */
|
||||||
|
class YtDlpTimeoutException(message: String) : IllegalStateException(message)
|
||||||
@@ -31,6 +31,43 @@ class DiscoveryServiceTest {
|
|||||||
assertEquals(YtCandidate("c", "Loud", "Gamma", null, 90, null), parsed[2])
|
assertEquals(YtCandidate("c", "Loud", "Gamma", null, 90, null), parsed[2])
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
fun `parses resolved YouTube Music songs and drops browse entries`() {
|
||||||
|
val json = """
|
||||||
|
{"id":"daft punk","entries":[
|
||||||
|
{"id":"wU26xVT_vBU","title":"One More Time (Radio Edit)","artist":"Daft Punk",
|
||||||
|
"artists":["Daft Punk"],"album":"One More Time","duration":320,
|
||||||
|
"thumbnails":[{"url":"http://t/1.jpg"},{"url":"http://t/2.jpg"}]},
|
||||||
|
{"id":"4D7u5KF7SP8","title":"Get Lucky (feat. Pharrell Williams and Nile Rodgers)",
|
||||||
|
"artists":["Daft Punk","Pharrell Williams","Nile Rodgers"],
|
||||||
|
"album":"Random Access Memories","duration":370},
|
||||||
|
{"id":"UCRr1xG_2WIDs18a6cIiCxeA","ie_key":"YoutubeTab","title":"Daft Punk - Topic"},
|
||||||
|
{"id":"MPREb_uWQeJO6Z53u","ie_key":"YoutubeTab","title":"Album - One More Time"},
|
||||||
|
{"id":"khnokW3Mw24","title":"Instant Crush (feat. Julian Casablancas)","artist":"Daft Punk",
|
||||||
|
"album":"NA","duration":338}
|
||||||
|
]}
|
||||||
|
""".trimIndent()
|
||||||
|
|
||||||
|
val parsed = parseYtSearchOutput(json)
|
||||||
|
|
||||||
|
assertEquals(3, parsed.size)
|
||||||
|
assertEquals(
|
||||||
|
YtCandidate(
|
||||||
|
videoId = "wU26xVT_vBU",
|
||||||
|
title = "One More Time (Radio Edit)",
|
||||||
|
artist = "Daft Punk",
|
||||||
|
album = "One More Time",
|
||||||
|
duration = 320,
|
||||||
|
thumbnail = "http://t/2.jpg",
|
||||||
|
),
|
||||||
|
parsed[0],
|
||||||
|
)
|
||||||
|
// A dash inside a real song title must not be treated as an artist split.
|
||||||
|
assertEquals("Get Lucky (feat. Pharrell Williams and Nile Rodgers)", parsed[1].title)
|
||||||
|
assertEquals("Daft Punk, Pharrell Williams, Nile Rodgers", parsed[1].artist)
|
||||||
|
assertEquals(null, parsed[2].album)
|
||||||
|
}
|
||||||
|
|
||||||
@Test
|
@Test
|
||||||
fun `parses a single resolved video`() {
|
fun `parses a single resolved video`() {
|
||||||
val json = """
|
val json = """
|
||||||
|
|||||||
Reference in new issue
Block a user