feat: accept, store and expose client diagnostics
CI / chart (push) Successful in 3s
CI / test (push) Successful in 2m20s
CI / chart-push (push) Successful in 6s
CI / image (push) Successful in 3m39s

- V11 client_logs table with per-user storage and retention/pruning
- POST/GET/DELETE /logs with batch validation, rate limit and admin access
- redact bearer tokens, Authorization headers and query/JSON secrets twice
- add X-Request-Id to responses and include it in the server log pattern
- report client error entries immediately in the server log
- scripts/logs.sh for source-side retrieval and clearing
- chart knobs CLIENT_LOG_RETENTION_DAYS and CLIENT_LOG_MAX_ROWS_PER_USER
This commit is contained in:
TapTap committed 2026-11-20 13:43:05 +00:00
1 parent 91312faa1c
commit 44b75c4d4d
19 files changed
+981 -5

No files matched your search

+42 -1
View File
@@ -100,6 +100,8 @@ Everything is environment variables:
| `DEEZER_API_BASE` | `https://api.deezer.com` | Public Deezer API base | | `DEEZER_API_BASE` | `https://api.deezer.com` | Public Deezer API base |
| `LOUDNESS_ENABLED` | `true` | Measure each file and expose a per-track gain | | `LOUDNESS_ENABLED` | `true` | Measure each file and expose a per-track gain |
| `LOUDNESS_TARGET_LUFS` | `-16` | Integrated loudness target (`-30` to `-5`) | | `LOUDNESS_TARGET_LUFS` | `-16` | Integrated loudness target (`-30` to `-5`) |
| `CLIENT_LOG_RETENTION_DAYS` | `14` | Days uploaded client diagnostics are kept |
| `CLIENT_LOG_MAX_ROWS_PER_USER` | `20000` | Newest diagnostic rows kept per user |
### `users.yaml` ### `users.yaml`
@@ -114,6 +116,37 @@ Passwords live only in this file (use a Vault-managed Secret in Kubernetes).
`admin` is only needed for the maintenance scan endpoints. Old `folders:` keys `admin` is only needed for the maintenance scan endpoints. Old `folders:` keys
are accepted and ignored. are accepted and ignored.
## Client diagnostics
The apps keep a small local ring buffer (500 entries) of warnings, errors and
crashes. Warnings and errors are flushed immediately, so the line that explains
a crash survives it; info entries are batched. When signed in, the client uploads
up to 100 entries per minute to `POST /api/v1/logs`, tagged with a random install
id, platform, app version and per-run session id.
Privacy and safety:
- `Authorization` headers, `Bearer` tokens, `token=`/`password=`/`arl=` query
parameters and JSON secret fields are replaced with `***` on the client
**and** again on the server.
- The server stores rows per user; a user can only read their own logs. An
administrator can pass `user=<name>`.
- Retention defaults to 14 days and 20,000 rows per user; both are configurable.
- A rejected batch (HTTP 400/413/422) is dropped so it cannot block later logs.
Inspect them with the helper:
```bash
scripts/logs.sh --server https://audiomanager.example.com --user alice \
--level warn --limit 100
# administrators can read another account
scripts/logs.sh --server ... --user admin --target alice
# clear after investigating
scripts/logs.sh --server ... --user alice --clear
```
The password is read from `LOGS_PASSWORD` or prompted; it is never stored.
## Native API ## Native API
All endpoints are under `/api/v1` and require a bearer token, except login. All endpoints are under `/api/v1` and require a bearer token, except login.
@@ -126,10 +159,18 @@ Media URLs may use `?token=` because HTML audio and images cannot send headers.
| `POST` | `/auth/login` | `{username,password,deviceName,platform}` → token | | `POST` | `/auth/login` | `{username,password,deviceName,platform}` → token |
| `POST` | `/auth/logout` | Revoke the current token | | `POST` | `/auth/logout` | Revoke the current token |
| `GET` | `/me` | Current user | | `GET` | `/me` | Current user |
| `GET` | `/info` | Default source, Deezer quality and loudness settings | | `GET` | `/info` | Default source, Deezer quality, loudness and log retention |
| `GET` | `/auth/devices` | List device tokens | | `GET` | `/auth/devices` | List device tokens |
| `DELETE` | `/auth/devices/{id}` | Revoke a device | | `DELETE` | `/auth/devices/{id}` | Revoke a device |
### Diagnostics
| Method | Path | Purpose |
|---|---|---|
| `POST` | `/logs` | Upload a client diagnostics batch |
| `GET` | `/logs` | List diagnostics (`level`, `device`, `since`, `before`, `limit`; admins may add `user`) |
| `DELETE` | `/logs` | Delete the current user's (or, for admins, a named user's) diagnostics |
### Moods ### Moods
| Method | Path | Purpose | | Method | Path | Purpose |
+1
View File
@@ -36,6 +36,7 @@ dependencies {
implementation("io.ktor:ktor-server-call-logging-jvm:3.1.2") implementation("io.ktor:ktor-server-call-logging-jvm:3.1.2")
implementation("io.ktor:ktor-server-auto-head-response-jvm:3.1.2") implementation("io.ktor:ktor-server-auto-head-response-jvm:3.1.2")
implementation("io.ktor:ktor-server-cors-jvm:3.1.2") implementation("io.ktor:ktor-server-cors-jvm:3.1.2")
implementation("io.ktor:ktor-server-call-id-jvm:3.1.2")
// Database // Database
implementation("org.jetbrains.exposed:exposed-core:0.60.0") implementation("org.jetbrains.exposed:exposed-core:0.60.0")
+1 -1
View File
@@ -2,5 +2,5 @@ apiVersion: v2
name: audiomanager name: audiomanager
description: Mood-first self-hosted music server description: Mood-first self-hosted music server
type: application type: application
version: 0.4.1 version: 0.5.0
appVersion: "0.1.0" appVersion: "0.1.0"
@@ -65,6 +65,10 @@ spec:
value: {{ .Values.loudness.enabled | quote }} value: {{ .Values.loudness.enabled | quote }}
- name: LOUDNESS_TARGET_LUFS - name: LOUDNESS_TARGET_LUFS
value: {{ .Values.loudness.targetLufs | quote }} value: {{ .Values.loudness.targetLufs | quote }}
- name: CLIENT_LOG_RETENTION_DAYS
value: {{ .Values.clientLogs.retentionDays | quote }}
- name: CLIENT_LOG_MAX_ROWS_PER_USER
value: {{ .Values.clientLogs.maxRowsPerUser | quote }}
{{- if .Values.deezer.existingSecret }} {{- if .Values.deezer.existingSecret }}
- name: DEEZER_ARL - name: DEEZER_ARL
valueFrom: valueFrom:
+6
View File
@@ -47,6 +47,12 @@ loudness:
# -16 LUFS is the ReplayGain-style target; -14 matches most streaming services. # -16 LUFS is the ReplayGain-style target; -14 matches most streaming services.
targetLufs: -16 targetLufs: -16
# Client diagnostics uploaded by the apps. The server redacts credentials again
# and keeps only a short window so bug reports do not grow the database.
clientLogs:
retentionDays: 14
maxRowsPerUser: 20000
# The media root, database, cover cache and download temp dir all live here. # The media root, database, cover cache and download temp dir all live here.
data: data:
persistence: persistence:
+12 -1
View File
@@ -750,7 +750,18 @@ Phases 1-8 are implemented in both repositories.
additions in sync on unmetered networks, three at a time. additions in sync on unmetered networks, three at a time.
- **Sleep timer** adds "Finish this track"; the full-screen player shows large - **Sleep timer** adds "Finish this track"; the full-screen player shows large
artwork, transport, add-to-mood, queue and Up next. artwork, transport, add-to-mood, queue and Up next.
- Tests: 21 client tests; desktop, Android debug and Wasm compile. - **Diagnostics (V11)**: the client keeps 500 local info/warn/error entries and
uploads up to 100 per minute to `POST /logs`, tagged with a random install id,
platform, app version and per-process session id. Warnings and errors flush
immediately so a crash keeps its cause. `Authorization`, `Bearer`, `token=`,
`password=` and `arl=` values are redacted client-side and again server-side.
Rows are per user, admins may read another user with `?user=`, and retention is
`CLIENT_LOG_RETENTION_DAYS` (14) plus `CLIENT_LOG_MAX_ROWS_PER_USER` (20000).
Uncaught exceptions are stored on Android and desktop and uploaded on the next
launch. Every response carries `X-Request-Id`, which the client includes in
server-error diagnostics so a report can be matched to the server log.
- Tests: 26 client tests; desktop, Android debug and Wasm compile.
### Known follow-ups ### Known follow-ups
+177
View File
@@ -0,0 +1,177 @@
#!/usr/bin/env bash
#
# Fetch and print client diagnostics stored by the server.
#
# The password is read from LOGS_PASSWORD or prompted, and is only used for the
# login request. The token is revoked before the script exits.
#
# Usage:
# scripts/logs.sh --server https://music.example.com --user alice
# scripts/logs.sh --server ... --user admin --target bob --level error
# scripts/logs.sh --server ... --user alice --since 7d --limit 500
# scripts/logs.sh --server ... --user alice --clear
set -euo pipefail
SERVER=""
USER_NAME=""
PASSWORD="${LOGS_PASSWORD:-}"
TARGET=""
LEVEL=""
DEVICE=""
LIMIT="200"
SINCE=""
CLEAR=0
RAW=0
usage() {
cat <<'EOF'
Usage: scripts/logs.sh --server URL --user NAME [options]
Options:
--password PW Password (otherwise LOGS_PASSWORD or a prompt)
--target NAME Read another user's logs (administrators only)
--level LEVEL debug, info, warn or error
--device ID Only this install id
--since VALUE Epoch milliseconds or a window such as 7d / 12h
--limit N Rows to fetch (1..1000, default 200)
--clear Delete the selected user's logs instead of printing them
--raw Print the JSON response unchanged
-h, --help Show this help
EOF
}
while [[ $# -gt 0 ]]; do
case "$1" in
--server) SERVER="${2:?--server needs a value}"; shift 2 ;;
--user) USER_NAME="${2:?--user needs a value}"; shift 2 ;;
--password) PASSWORD="${2:?--password needs a value}"; shift 2 ;;
--target) TARGET="${2:?--target needs a value}"; shift 2 ;;
--level) LEVEL="${2:?--level needs a value}"; shift 2 ;;
--device) DEVICE="${2:?--device needs a value}"; shift 2 ;;
--since) SINCE="${2:?--since needs a value}"; shift 2 ;;
--limit) LIMIT="${2:?--limit needs a value}"; shift 2 ;;
--clear) CLEAR=1; shift ;;
--raw) RAW=1; shift ;;
-h|--help) usage; exit 0 ;;
*) echo "Unknown argument: $1" >&2; usage >&2; exit 2 ;;
esac
done
if [[ -z "$SERVER" || -z "$USER_NAME" ]]; then
usage >&2
exit 2
fi
SERVER="${SERVER%/}"
if [[ -z "$PASSWORD" ]]; then
read -rsp "Password for $USER_NAME: " PASSWORD
echo
fi
since_millis() {
local value="$1"
if [[ "$value" =~ ^([0-9]+)([dh])$ ]]; then
python3 - "$value" <<'PY'
import re, sys, time
value = sys.argv[1]
amount = int(re.match(r"([0-9]+)", value).group(1))
unit = value[-1]
seconds = amount * (86400 if unit == "d" else 3600)
print(int((time.time() - seconds) * 1000))
PY
else
printf '%s' "$value"
fi
}
# Keep the password out of argv: it is piped to the JSON encoder as stdin.
login_body="$(printf '%s\n%s\n' "$USER_NAME" "$PASSWORD" | python3 -c '
import json, sys
user = sys.stdin.readline().rstrip("\n")
password = sys.stdin.readline().rstrip("\n")
print(json.dumps({
"username": user,
"password": password,
"deviceName": "diagnostics-cli",
"platform": "cli",
}))
')"
login_response="$(printf '%s' "$login_body" | curl -sS -w '\n%{http_code}' -X POST \
"$SERVER/api/v1/auth/login" \
-H 'Content-Type: application/json' \
--data-binary @-)"
login_status="$(printf '%s' "$login_response" | tail -1)"
login_body="$(printf '%s' "$login_response" | sed '$d')"
if [[ "$login_status" != "200" ]]; then
echo "Login failed (HTTP $login_status): $login_body" >&2
exit 1
fi
TOKEN="$(printf '%s' "$login_body" | python3 -c 'import json,sys; print(json.load(sys.stdin)["token"])')"
# The token stays in a 0600 curl config instead of the process list.
AUTH_CONFIG="$(mktemp "${TMPDIR:-/tmp}/audiomanager-logs.XXXXXX")"
PAYLOAD_FILE=""
chmod 600 "$AUTH_CONFIG"
printf 'header = "Authorization: Bearer %s"\n' "$TOKEN" > "$AUTH_CONFIG"
cleanup() {
curl -sS -o /dev/null -X POST "$SERVER/api/v1/auth/logout" \
-K "$AUTH_CONFIG" 2>/dev/null || true
rm -f "$AUTH_CONFIG" "$PAYLOAD_FILE"
}
trap cleanup EXIT
query=()
[[ -n "$LEVEL" ]] && query+=(--data-urlencode "level=$LEVEL")
[[ -n "$DEVICE" ]] && query+=(--data-urlencode "device=$DEVICE")
[[ -n "$TARGET" ]] && query+=(--data-urlencode "user=$TARGET")
[[ -n "$SINCE" ]] && query+=(--data-urlencode "since=$(since_millis "$SINCE")")
if [[ "$CLEAR" -eq 1 ]]; then
response="$(curl -sS -G -X DELETE "$SERVER/api/v1/logs" \
-K "$AUTH_CONFIG" "${query[@]}")"
printf '%s' "$response" | python3 -c 'import json,sys; print("deleted", json.load(sys.stdin)["deleted"], "entries")'
exit 0
fi
if [[ "$LIMIT" != "200" ]]; then
query+=(--data-urlencode "limit=$LIMIT")
fi
response="$(curl -sS -G "$SERVER/api/v1/logs" \
-K "$AUTH_CONFIG" "${query[@]}")"
PAYLOAD_FILE="$(mktemp "${TMPDIR:-/tmp}/audiomanager-logs-payload.XXXXXX")"
printf '%s' "$response" > "$PAYLOAD_FILE"
python3 - "$PAYLOAD_FILE" "$RAW" <<'PY'
import json
import sys
with open(sys.argv[1], encoding="utf-8") as handle:
data = json.load(handle)
if sys.argv[2] == "1":
print(json.dumps(data, indent=2))
raise SystemExit(0)
logs = data.get("logs", [])
if not logs:
print("No diagnostics stored for this user.")
raise SystemExit(0)
print(f"{len(logs)} entr{'y' if len(logs) == 1 else 'ies'} (newest first)")
print()
for row in logs:
when = row.get("createdAt") or row.get("receivedAt") or "?"
device = f"{row.get('platform', '?')} {row.get('appVersion', '?')} {row.get('deviceId', '?')}"
session = row.get("sessionId")
suffix = f" session={session}" if session else ""
print(f"{when} {row.get('level', '?').upper():5} {row.get('tag', '?')} [{device}]{suffix}")
print(f" {row.get('message', '')}")
detail = row.get("detail")
if detail:
for line in detail.splitlines():
print(f" | {line}")
print()
PY
+23
View File
@@ -498,6 +498,29 @@ fi
expect "scan status" "200" "GET" "/scan" expect "scan status" "200" "GET" "/scan"
expect "start scan" "202" "POST" "/scan" expect "start scan" "202" "POST" "/scan"
log ""
log "Diagnostics"
expect "upload client logs" "200" "POST" "/logs" \
'{"deviceId":"smoke-dev","platform":"desktop","appVersion":"0.4.3","entries":[{"timestamp":0,"level":"error","tag":"smoke","message":"failed with Bearer smoke-secret","detail":"https://music/stream?token=abc123"}]}'
expect_eq "client log accepted" "1" "$(printf '%s' "$HTTP_BODY" | json_field '["accepted"]')"
expect "list client logs" "200" "GET" "/logs"
LOGS_RESULT="$(printf '%s' "$HTTP_BODY" | python3 -c '
import json, sys
logs = json.load(sys.stdin)["logs"]
text = json.dumps(logs)
ok = len(logs) == 1 and logs[0]["level"] == "error" and "smoke-secret" not in text and "abc123" not in text
print("ok" if ok else "bad")
')"
expect_eq "client logs are stored and redacted" "ok" "$LOGS_RESULT"
LOGS_OUTPUT="$(LOGS_PASSWORD="$PASS" "$ROOT/scripts/logs.sh" --server "$BASE" --user "$USER" --limit 5 2>&1)" || true
if printf '%s' "$LOGS_OUTPUT" | grep -q 'failed with Bearer \*\*\*'; then
pass "logs.sh prints redacted diagnostics"
else
fail "logs.sh prints redacted diagnostics" "${LOGS_OUTPUT:0:240}"
fi
expect "clear client logs" "200" "DELETE" "/logs"
expect_eq "client logs deleted" "1" "$(printf '%s' "$HTTP_BODY" | json_field '["deleted"]')"
log "" log ""
log "Cleanup" log "Cleanup"
expect "delete mood" "204" "DELETE" "/moods/$MOOD_ID" expect "delete mood" "204" "DELETE" "/moods/$MOOD_ID"
@@ -9,6 +9,8 @@ import io.ktor.server.application.install
import io.ktor.server.engine.embeddedServer import io.ktor.server.engine.embeddedServer
import io.ktor.server.netty.Netty import io.ktor.server.netty.Netty
import io.ktor.server.plugins.autohead.AutoHeadResponse import io.ktor.server.plugins.autohead.AutoHeadResponse
import io.ktor.server.plugins.callid.CallId
import io.ktor.server.plugins.callid.callIdMdc
import io.ktor.server.plugins.calllogging.CallLogging import io.ktor.server.plugins.calllogging.CallLogging
import io.ktor.server.plugins.contentnegotiation.ContentNegotiation import io.ktor.server.plugins.contentnegotiation.ContentNegotiation
import io.ktor.server.plugins.cors.routing.CORS import io.ktor.server.plugins.cors.routing.CORS
@@ -20,6 +22,7 @@ import io.ktor.server.response.respondText
import io.ktor.server.routing.get import io.ktor.server.routing.get
import io.ktor.server.routing.routing import io.ktor.server.routing.routing
import java.nio.file.Files import java.nio.file.Files
import java.util.UUID
import java.util.concurrent.TimeUnit import java.util.concurrent.TimeUnit
import kotlinx.coroutines.CoroutineScope import kotlinx.coroutines.CoroutineScope
import kotlinx.coroutines.Dispatchers import kotlinx.coroutines.Dispatchers
@@ -38,6 +41,7 @@ import win.taptap.musicmanager.api.ApiException
import win.taptap.musicmanager.api.apiRoutes import win.taptap.musicmanager.api.apiRoutes
import win.taptap.musicmanager.config.AppConfig import win.taptap.musicmanager.config.AppConfig
import win.taptap.musicmanager.config.UsersConfig import win.taptap.musicmanager.config.UsersConfig
import win.taptap.musicmanager.db.ClientLogRepository
import win.taptap.musicmanager.db.ConfigRepository import win.taptap.musicmanager.db.ConfigRepository
import win.taptap.musicmanager.db.DatabaseFactory import win.taptap.musicmanager.db.DatabaseFactory
import win.taptap.musicmanager.db.MoodRepository import win.taptap.musicmanager.db.MoodRepository
@@ -184,6 +188,25 @@ fun main() {
} }
} }
// Client diagnostics are kept for a short window so a bug can be
// investigated without letting the table grow without bound.
applicationScope.launch(Dispatchers.IO) {
delay(TimeUnit.MINUTES.toMillis(1))
while (isActive) {
runCatching {
ClientLogRepository.prune(
config.clientLogRetentionDays,
config.clientLogMaxRowsPerUser,
)
}
.onSuccess { count ->
if (count > 0) log.info("Pruned {} client diagnostics rows", count)
}
.onFailure { log.warn("Client diagnostics prune failed", it) }
delay(TimeUnit.HOURS.toMillis(6))
}
}
// Delete files that no mood, recommendation or open session references. // Delete files that no mood, recommendation or open session references.
applicationScope.launch(Dispatchers.IO) { applicationScope.launch(Dispatchers.IO) {
delay(TimeUnit.MINUTES.toMillis(10)) delay(TimeUnit.MINUTES.toMillis(10))
@@ -221,7 +244,15 @@ fun main() {
} }
install(PartialContent) install(PartialContent)
install(AutoHeadResponse) install(AutoHeadResponse)
install(CallLogging) // A short id on every response makes server log lines and client
// diagnostics meet in the middle.
install(CallId) {
generate { UUID.randomUUID().toString().take(8) }
replyToHeader("X-Request-Id")
}
install(CallLogging) {
callIdMdc("requestId")
}
install(StatusPages) { install(StatusPages) {
exception<ApiException> { call, cause -> exception<ApiException> { call, cause ->
call.respond( call.respond(
@@ -271,6 +302,7 @@ fun main() {
deezerQuality = config.deezerQuality, deezerQuality = config.deezerQuality,
loudnessEnabled = config.loudnessEnabled && loudnessAnalyzer != null, loudnessEnabled = config.loudnessEnabled && loudnessAnalyzer != null,
loudnessTargetLufs = config.loudnessTargetLufs, loudnessTargetLufs = config.loudnessTargetLufs,
clientLogRetentionDays = config.clientLogRetentionDays,
startScan = { scanner.start(applicationScope) }, startScan = { scanner.start(applicationScope) },
scanStatus = { scanner.status() }, scanStatus = { scanner.status() },
) )
@@ -44,6 +44,49 @@ data class DeviceDto(
val current: Boolean, val current: Boolean,
) )
@Serializable
data class ClientLogEntryDto(
/** Device clock in epoch milliseconds; informational, the server clamps it. */
val timestamp: Long = 0,
val level: String,
val tag: String,
val message: String,
val detail: String? = null,
val sessionId: String? = null,
)
@Serializable
data class ClientLogBatchDto(
val deviceId: String = "unknown",
val platform: String = "unknown",
val appVersion: String = "unknown",
val entries: List<ClientLogEntryDto> = emptyList(),
)
@Serializable
data class ClientLogDto(
val id: Long,
val deviceId: String,
val platform: String,
val appVersion: String,
val sessionId: String? = null,
val level: String,
val tag: String,
val message: String,
val detail: String? = null,
val createdAt: String,
val receivedAt: String,
)
@Serializable
data class ClientLogsDto(val logs: List<ClientLogDto>)
@Serializable
data class ClientLogsAcceptedDto(val accepted: Int)
@Serializable
data class ClientLogsDeletedDto(val deleted: Int)
@Serializable @Serializable
data class ServerInfoDto( data class ServerInfoDto(
val defaultSource: String, val defaultSource: String,
@@ -51,6 +94,8 @@ data class ServerInfoDto(
val deezerQuality: Int, val deezerQuality: Int,
val loudnessEnabled: Boolean, val loudnessEnabled: Boolean,
val loudnessTargetLufs: Double, val loudnessTargetLufs: Double,
/** How long the server keeps uploaded client diagnostics. */
val clientLogRetentionDays: Int,
) )
@Serializable @Serializable
@@ -40,6 +40,7 @@ fun Route.apiRoutes(
deezerQuality: Int, deezerQuality: Int,
loudnessEnabled: Boolean, loudnessEnabled: Boolean,
loudnessTargetLufs: Double, loudnessTargetLufs: Double,
clientLogRetentionDays: Int,
startScan: () -> Boolean, startScan: () -> Boolean,
scanStatus: () -> ScanStatus, scanStatus: () -> ScanStatus,
) { ) {
@@ -89,6 +90,7 @@ fun Route.apiRoutes(
deezerQuality = deezerQuality, deezerQuality = deezerQuality,
loudnessEnabled = loudnessEnabled, loudnessEnabled = loudnessEnabled,
loudnessTargetLufs = loudnessTargetLufs, loudnessTargetLufs = loudnessTargetLufs,
clientLogRetentionDays = clientLogRetentionDays,
), ),
) )
} }
@@ -151,6 +153,7 @@ fun Route.apiRoutes(
defaultSource = defaultSource, defaultSource = defaultSource,
) )
sessionRoutes(users, radioService, gc) sessionRoutes(users, radioService, gc)
logRoutes(users)
} }
} }
@@ -0,0 +1,142 @@
package win.taptap.musicmanager.api
import io.ktor.http.HttpStatusCode
import io.ktor.server.application.ApplicationCall
import io.ktor.server.request.contentLength
import io.ktor.server.response.respond
import io.ktor.server.routing.Route
import io.ktor.server.routing.delete
import io.ktor.server.routing.get
import io.ktor.server.routing.post
import java.time.Instant
import java.time.temporal.ChronoUnit
import org.slf4j.LoggerFactory
import win.taptap.musicmanager.config.UserAccount
import win.taptap.musicmanager.config.UsersConfig
import win.taptap.musicmanager.db.ClientLogRepository
import win.taptap.musicmanager.db.ClientLogSanitizer
import win.taptap.musicmanager.db.toDto
private val log = LoggerFactory.getLogger("win.taptap.musicmanager.api.Logs")
/** A diagnostic batch is small; anything bigger is a bug or abuse. */
private const val MAX_BODY_BYTES = 2L * 1024 * 1024
/** Keep the insert transaction short and the per-request cost predictable. */
private const val MAX_ENTRIES_PER_BATCH = 200
/** Per-user ingest ceiling, measured over the last hour. */
private const val MAX_ROWS_PER_HOUR = 20_000
/**
* Client diagnostics endpoints.
*
* The app uploads warnings/errors and a few context entries in batches. Each
* row belongs to the authenticated user (or to the user an administrator names),
* so one account can never read another account's logs. The server redacts
* credentials again before storing anything.
*/
fun Route.logRoutes(users: UsersConfig) {
post("/logs") {
val user = call.requireApiUser(users)
val length = call.request.contentLength()
if (length != null && length > MAX_BODY_BYTES) {
throw ApiException(
HttpStatusCode.PayloadTooLarge,
"logs_too_large",
"A log batch may not exceed ${MAX_BODY_BYTES / 1024} KiB",
)
}
val batch = call.receiveJson<ClientLogBatchDto>()
if (batch.entries.isEmpty()) badRequest("A log batch needs at least one entry")
if (batch.entries.size > MAX_ENTRIES_PER_BATCH) {
badRequest("A log batch may contain at most $MAX_ENTRIES_PER_BATCH entries")
}
val payloadBytes = batch.entries.sumOf { entry ->
entry.message.length + (entry.detail?.length ?: 0) + entry.tag.length
}
if (payloadBytes > MAX_BODY_BYTES) {
throw ApiException(
HttpStatusCode.PayloadTooLarge,
"logs_too_large",
"A log batch may not exceed ${MAX_BODY_BYTES / 1024} KiB",
)
}
batch.entries.forEach { entry ->
if (ClientLogSanitizer.normalizeLevel(entry.level) == null) {
badRequest("Invalid log level: ${entry.level.take(16)}")
}
}
val since = Instant.now().minus(1, ChronoUnit.HOURS)
if (ClientLogRepository.countSince(user.name, since) >= MAX_ROWS_PER_HOUR) {
throw ApiException(
HttpStatusCode.TooManyRequests,
"logs_rate_limited",
"Too many diagnostic entries in the last hour",
)
}
val accepted = ClientLogRepository.insert(user.name, batch)
// Errors should show up in the server log immediately, not only when
// someone remembers to query the diagnostics endpoint.
batch.entries
.filter { ClientLogSanitizer.normalizeLevel(it.level) == "error" }
.take(3)
.forEach { entry ->
log.warn(
"Client error [{}] {}: {}",
user.name,
ClientLogSanitizer.tag(entry.tag),
ClientLogSanitizer.message(entry.message).take(300),
)
}
call.respond(ClientLogsAcceptedDto(accepted))
}
get("/logs") {
val user = call.requireApiUser(users)
val target = call.logTargetUser(users, user)
val level = call.request.queryParameters["level"]
?.takeIf { it.isNotBlank() }
?.let {
ClientLogSanitizer.normalizeLevel(it)
?: badRequest("level must be one of ${ClientLogSanitizer.levels.sorted().joinToString()}")
}
val deviceId = call.request.queryParameters["device"]?.takeIf { it.isNotBlank() }
val beforeId = call.request.queryParameters["before"]?.let { raw ->
raw.toLongOrNull()?.takeIf { it > 0 } ?: badRequest("before must be a positive row id")
}
val sinceMillis = call.request.queryParameters["since"]?.let { raw ->
raw.toLongOrNull()?.takeIf { it > 0 } ?: badRequest("since must be epoch milliseconds")
}
val limit = call.request.queryParameters["limit"]?.let { raw ->
raw.toIntOrNull()?.takeIf { it in 1..1_000 } ?: badRequest("limit must be between 1 and 1000")
} ?: 200
val logs = ClientLogRepository.list(
userName = target,
level = level,
deviceId = deviceId,
beforeId = beforeId,
sinceMillis = sinceMillis,
limit = limit,
)
call.respond(ClientLogsDto(logs.map { it.toDto() }))
}
delete("/logs") {
val user = call.requireApiUser(users)
val target = call.logTargetUser(users, user)
val deleted = ClientLogRepository.delete(target)
call.respond(ClientLogsDeletedDto(deleted))
}
}
private fun ApplicationCall.logTargetUser(users: UsersConfig, current: UserAccount): String {
val requested = request.queryParameters["user"]?.takeIf { it.isNotBlank() }
if (requested == null || requested == current.name) return current.name
if (!current.admin) forbidden("Only administrators can read another user's logs")
return users.byName(requested)?.name ?: notFound("User")
}
@@ -26,6 +26,10 @@ data class AppConfig(
val loudnessEnabled: Boolean = true, val loudnessEnabled: Boolean = true,
/** Integrated loudness target; -16 LUFS is the ReplayGain-style value. */ /** Integrated loudness target; -16 LUFS is the ReplayGain-style value. */
val loudnessTargetLufs: Double = -16.0, val loudnessTargetLufs: Double = -16.0,
/** Client diagnostics: rows older than this are pruned. */
val clientLogRetentionDays: Int = 14,
/** Client diagnostics: newest rows kept per user. */
val clientLogMaxRowsPerUser: Int = 20_000,
val mediaDir: Path = dataDir.resolve("media"), val mediaDir: Path = dataDir.resolve("media"),
val downloadDir: Path = dataDir.resolve("downloads"), val downloadDir: Path = dataDir.resolve("downloads"),
) { ) {
@@ -62,6 +66,18 @@ data class AppConfig(
require(loudnessTargetLufs in -30.0..-5.0) { require(loudnessTargetLufs in -30.0..-5.0) {
"LOUDNESS_TARGET_LUFS must be between -30 and -5, got: $loudnessTargetLufs" "LOUDNESS_TARGET_LUFS must be between -30 and -5, got: $loudnessTargetLufs"
} }
val rawRetention = value("CLIENT_LOG_RETENTION_DAYS", "14")
val clientLogRetentionDays = rawRetention.toIntOrNull()
?: throw IllegalArgumentException("CLIENT_LOG_RETENTION_DAYS must be an integer")
require(clientLogRetentionDays in 1..365) {
"CLIENT_LOG_RETENTION_DAYS must be between 1 and 365, got: $clientLogRetentionDays"
}
val rawLogRows = value("CLIENT_LOG_MAX_ROWS_PER_USER", "20000")
val clientLogMaxRowsPerUser = rawLogRows.toIntOrNull()
?: throw IllegalArgumentException("CLIENT_LOG_MAX_ROWS_PER_USER must be an integer")
require(clientLogMaxRowsPerUser in 100..1_000_000) {
"CLIENT_LOG_MAX_ROWS_PER_USER must be between 100 and 1000000, got: $clientLogMaxRowsPerUser"
}
return AppConfig( return AppConfig(
port = port, port = port,
@@ -76,6 +92,8 @@ data class AppConfig(
deezerApiBase = value("DEEZER_API_BASE", "https://api.deezer.com"), deezerApiBase = value("DEEZER_API_BASE", "https://api.deezer.com"),
loudnessEnabled = loudnessEnabled, loudnessEnabled = loudnessEnabled,
loudnessTargetLufs = loudnessTargetLufs, loudnessTargetLufs = loudnessTargetLufs,
clientLogRetentionDays = clientLogRetentionDays,
clientLogMaxRowsPerUser = clientLogMaxRowsPerUser,
mediaDir = Path.of(value("MEDIA_DIR", dataDir.resolve("media").toString())), mediaDir = Path.of(value("MEDIA_DIR", dataDir.resolve("media").toString())),
downloadDir = Path.of(value("DOWNLOAD_DIR", dataDir.resolve("downloads").toString())), downloadDir = Path.of(value("DOWNLOAD_DIR", dataDir.resolve("downloads").toString())),
) )
@@ -0,0 +1,234 @@
package win.taptap.musicmanager.db
import java.time.Instant
import java.time.temporal.ChronoUnit
import org.jetbrains.exposed.sql.Op
import org.jetbrains.exposed.sql.ResultRow
import org.jetbrains.exposed.sql.SortOrder
import org.jetbrains.exposed.sql.SqlExpressionBuilder.eq
import org.jetbrains.exposed.sql.SqlExpressionBuilder.greaterEq
import org.jetbrains.exposed.sql.SqlExpressionBuilder.less
import org.jetbrains.exposed.sql.and
import org.jetbrains.exposed.sql.deleteWhere
import org.jetbrains.exposed.sql.insert
import org.jetbrains.exposed.sql.selectAll
import org.jetbrains.exposed.sql.transactions.transaction
import win.taptap.musicmanager.api.ClientLogBatchDto
import win.taptap.musicmanager.api.ClientLogDto
data class ClientLogRecord(
val id: Long,
val deviceId: String,
val platform: String,
val appVersion: String,
val sessionId: String?,
val level: String,
val tag: String,
val message: String,
val detail: String?,
val createdAt: String,
val receivedAt: String,
)
fun ClientLogRecord.toDto(): ClientLogDto = ClientLogDto(
id = id,
deviceId = deviceId,
platform = platform,
appVersion = appVersion,
sessionId = sessionId,
level = level,
tag = tag,
message = message,
detail = detail,
createdAt = createdAt,
receivedAt = receivedAt,
)
/**
* Removes credentials before anything is persisted. The client redacts as well,
* but the server must not trust a buggy or older build: a token in a log line
* would give whoever reads the table access to the account.
*/
object ClientLogSanitizer {
private const val MAX_MESSAGE = 4_000
private const val MAX_DETAIL = 16_000
private const val MAX_TAG = 64
private const val MAX_DEVICE_ID = 80
private const val MAX_PLATFORM = 32
private const val MAX_APP_VERSION = 40
private const val MAX_SESSION_ID = 64
private val bearer = Regex("""(?i)\b(bearer\s+)[A-Za-z0-9._~+/=-]{8,}""")
private val authorization = Regex("""(?i)\b(authorization\s*:\s*)[^\r\n]+""")
private val querySecret = Regex(
"""(?i)(\b(?:token|access_token|password|passwd|arl|key)=)[^&\s"'<>]+""",
)
private val jsonSecret = Regex(
"""(?i)("(?:token|access_token|password|arl|key)"\s*:\s*")[^"]*(")""",
)
val levels = setOf("debug", "info", "warn", "error")
fun redact(value: String): String = value
.replace(bearer) { match -> match.groupValues[1] + "***" }
.replace(authorization) { match -> match.groupValues[1] + "***" }
.replace(querySecret) { match -> match.groupValues[1] + "***" }
.replace(jsonSecret) { match -> match.groupValues[1] + "***" + match.groupValues[2] }
fun normalizeLevel(value: String): String? =
value.trim().lowercase().takeIf { it in levels }
fun deviceId(value: String): String = clean(value, MAX_DEVICE_ID).ifBlank { "unknown" }
fun platform(value: String): String = clean(value, MAX_PLATFORM).ifBlank { "unknown" }
fun appVersion(value: String): String = clean(value, MAX_APP_VERSION).ifBlank { "unknown" }
fun sessionId(value: String?): String? =
value?.let { clean(it, MAX_SESSION_ID) }?.takeIf { it.isNotBlank() }
fun tag(value: String): String = clean(value, MAX_TAG).ifBlank { "app" }
fun message(value: String): String = clean(value, MAX_MESSAGE).ifBlank { "(empty)" }
fun detail(value: String?): String? =
value?.let { clean(it, MAX_DETAIL) }?.takeIf { it.isNotBlank() }
private fun clean(value: String, max: Int): String =
redact(value).trim().let { if (it.length > max) it.take(max) else it }
}
/**
* Client diagnostics storage. A batch is written in one transaction so a broken
* request cannot leave half a log behind, and retention runs on the server clock.
*/
object ClientLogRepository {
private const val MAX_CLOCK_SKEW_MILLIS = 5 * 60_000L
private const val MAX_ENTRY_AGE_MILLIS = 90L * 24 * 60 * 60_000
fun insert(userName: String, batch: ClientLogBatchDto): Int {
val now = Instant.now()
val receivedAt = now.toString()
val deviceId = ClientLogSanitizer.deviceId(batch.deviceId)
val platform = ClientLogSanitizer.platform(batch.platform)
val appVersion = ClientLogSanitizer.appVersion(batch.appVersion)
return transaction {
var inserted = 0
batch.entries.forEach { entry ->
val level = ClientLogSanitizer.normalizeLevel(entry.level) ?: return@forEach
val millis = entry.timestamp
.takeIf { it > 0 }
?.coerceIn(
now.toEpochMilli() - MAX_ENTRY_AGE_MILLIS,
now.toEpochMilli() + MAX_CLOCK_SKEW_MILLIS,
)
?: now.toEpochMilli()
ClientLogsTable.insert {
it[ClientLogsTable.userName] = userName
it[ClientLogsTable.deviceId] = deviceId
it[ClientLogsTable.platform] = platform
it[ClientLogsTable.appVersion] = appVersion
it[sessionId] = ClientLogSanitizer.sessionId(entry.sessionId)
it[ClientLogsTable.level] = level
it[tag] = ClientLogSanitizer.tag(entry.tag)
it[message] = ClientLogSanitizer.message(entry.message)
it[detail] = ClientLogSanitizer.detail(entry.detail)
it[createdAt] = Instant.ofEpochMilli(millis).toString()
it[ClientLogsTable.receivedAt] = receivedAt
}
inserted += 1
}
inserted
}
}
fun list(
userName: String,
level: String? = null,
deviceId: String? = null,
beforeId: Long? = null,
sinceMillis: Long? = null,
limit: Int = 200,
): List<ClientLogRecord> {
var condition: Op<Boolean> = ClientLogsTable.userName eq userName
level?.let { condition = condition and (ClientLogsTable.level eq it) }
deviceId?.let { condition = condition and (ClientLogsTable.deviceId eq it) }
beforeId?.let { condition = condition and (ClientLogsTable.id less it) }
sinceMillis?.let {
val since = Instant.ofEpochMilli(it).toString()
condition = condition and (ClientLogsTable.receivedAt greaterEq since)
}
return transaction {
ClientLogsTable.selectAll()
.where(condition)
.orderBy(ClientLogsTable.id to SortOrder.DESC)
.limit(limit.coerceIn(1, 1_000))
.map { it.toRecord() }
}
}
fun countSince(userName: String, since: Instant): Long = transaction {
ClientLogsTable.selectAll()
.where {
(ClientLogsTable.userName eq userName) and
(ClientLogsTable.receivedAt greaterEq since.toString())
}
.count()
}
fun delete(userName: String): Int = transaction {
ClientLogsTable.deleteWhere { ClientLogsTable.userName eq userName }
}
/**
* Deletes rows older than [retentionDays] and trims every user to
* [maxRowsPerUser], newest first. Returns the number of deleted rows.
*/
fun prune(retentionDays: Int, maxRowsPerUser: Int): Int {
val cutoff = Instant.now()
.minus(retentionDays.coerceAtLeast(1).toLong(), ChronoUnit.DAYS)
.toString()
var deleted = transaction {
ClientLogsTable.deleteWhere { ClientLogsTable.receivedAt less cutoff }
}
val users = transaction {
ClientLogsTable.select(ClientLogsTable.userName)
.withDistinct()
.map { it[ClientLogsTable.userName] }
}
users.forEach { user -> deleted += trim(user, maxRowsPerUser) }
return deleted
}
private fun trim(userName: String, maxRows: Int): Int {
if (maxRows <= 0) return delete(userName)
return transaction {
val cutoff = ClientLogsTable.select(ClientLogsTable.id)
.where { ClientLogsTable.userName eq userName }
.orderBy(ClientLogsTable.id to SortOrder.DESC)
.limit(1).offset((maxRows - 1).toLong())
.map { it[ClientLogsTable.id] }
.firstOrNull() ?: return@transaction 0
ClientLogsTable.deleteWhere {
(ClientLogsTable.userName eq userName) and
(ClientLogsTable.id less cutoff)
}
}
}
private fun ResultRow.toRecord(): ClientLogRecord = ClientLogRecord(
id = this[ClientLogsTable.id],
deviceId = this[ClientLogsTable.deviceId],
platform = this[ClientLogsTable.platform],
appVersion = this[ClientLogsTable.appVersion],
sessionId = this[ClientLogsTable.sessionId],
level = this[ClientLogsTable.level],
tag = this[ClientLogsTable.tag],
message = this[ClientLogsTable.message],
detail = this[ClientLogsTable.detail],
createdAt = this[ClientLogsTable.createdAt],
receivedAt = this[ClientLogsTable.receivedAt],
)
}
@@ -163,3 +163,24 @@ object SongAffinityTable : Table("song_affinity") {
override val primaryKey = PrimaryKey(userName, songId) override val primaryKey = PrimaryKey(userName, songId)
} }
/**
* Batches of client diagnostics. `received_at` is the server clock and is what
* retention uses; `created_at` is the device clock and is only for display.
*/
object ClientLogsTable : Table("client_logs") {
val id = long("id").autoIncrement()
val userName = reference("user_name", UsersTable.name, onDelete = ReferenceOption.CASCADE)
val deviceId = varchar("device_id", 80)
val platform = varchar("platform", 32)
val appVersion = varchar("app_version", 40)
val sessionId = varchar("session_id", 64).nullable()
val level = varchar("level", 8)
val tag = varchar("tag", 64)
val message = text("message")
val detail = text("detail").nullable()
val createdAt = varchar("created_at", 40)
val receivedAt = varchar("received_at", 40)
override val primaryKey = PrimaryKey(id)
}
@@ -0,0 +1,24 @@
-- Client diagnostics. The apps keep a small local ring buffer of warnings and
-- errors (plus an occasional info entry) and upload it in batches so a bug can
-- be investigated with the server-side request id and the client context.
--
-- Rows are per user and pruned by retention days and a per-user row cap. The
-- client timestamp is informational; ordering uses the server-assigned id so a
-- device with a wrong clock cannot reorder the log.
CREATE TABLE client_logs (
id INTEGER PRIMARY KEY AUTOINCREMENT,
user_name TEXT NOT NULL REFERENCES users(name) ON DELETE CASCADE,
device_id TEXT NOT NULL,
platform TEXT NOT NULL,
app_version TEXT NOT NULL,
session_id TEXT,
level TEXT NOT NULL CHECK (level IN ('debug', 'info', 'warn', 'error')),
tag TEXT NOT NULL,
message TEXT NOT NULL,
detail TEXT,
created_at TEXT NOT NULL,
received_at TEXT NOT NULL
);
CREATE INDEX idx_client_logs_user_id ON client_logs (user_name, id);
CREATE INDEX idx_client_logs_received ON client_logs (received_at);
+1 -1
View File
@@ -1,7 +1,7 @@
<configuration> <configuration>
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender"> <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<encoder> <encoder>
<pattern>%d{yyyy-MM-dd'T'HH:mm:ss.SSSXXX} %-5level [%thread] %logger{36} - %msg%n</pattern> <pattern>%d{yyyy-MM-dd'T'HH:mm:ss.SSSXXX} %-5level [%thread] %logger{36} %X{requestId:-} - %msg%n</pattern>
</encoder> </encoder>
</appender> </appender>
@@ -94,6 +94,7 @@ class ApiRoutesTest {
assertEquals(0, info["deezerQuality"]!!.jsonPrimitive.int) assertEquals(0, info["deezerQuality"]!!.jsonPrimitive.int)
assertTrue(info["loudnessEnabled"]!!.jsonPrimitive.content.toBoolean()) assertTrue(info["loudnessEnabled"]!!.jsonPrimitive.content.toBoolean())
assertEquals("-16.0", info["loudnessTargetLufs"]!!.jsonPrimitive.content) assertEquals("-16.0", info["loudnessTargetLufs"]!!.jsonPrimitive.content)
assertEquals(14, info["clientLogRetentionDays"]!!.jsonPrimitive.int)
val malformed = client.post("/api/v1/moods") { val malformed = client.post("/api/v1/moods") {
bearer(token) bearer(token)
@@ -105,6 +106,88 @@ class ApiRoutesTest {
} }
} }
@Test
fun `client diagnostics are private and redacted`() {
seededContext().use { seed ->
testApplication {
setupApp(seed)
val alice = client.token("alice", "alice-pw")
val bob = client.token("bob", "bob-pw")
val batch = """
{"deviceId":"pixel-7","platform":"android","appVersion":"0.4.3",
"entries":[
{"timestamp":1700000000000,"level":"warn","tag":"playback","message":"stalled","detail":"Authorization: Bearer keep-secret","sessionId":"run-1"},
{"timestamp":1700000000000,"level":"error","tag":"api","message":"GET /stream -> 500","detail":"https://music/stream?token=abc123"}
]}
""".trimIndent()
val upload = client.post("/api/v1/logs") {
bearer(alice)
contentType(ContentType.Application.Json)
setBody(batch)
}
assertEquals(HttpStatusCode.OK, upload.status)
assertEquals(2, parse(upload.bodyAsText())["accepted"]!!.jsonPrimitive.int)
val list = parse(client.get("/api/v1/logs") { bearer(alice) }.bodyAsText())
val logs = list["logs"]!!.jsonArray
assertEquals(2, logs.size)
// Newest first: the error entry was inserted last.
assertEquals("api", logs[0].jsonObject.string("tag"))
assertEquals("error", logs[0].jsonObject.string("level"))
assertFalse(logs[0].jsonObject.string("detail").contains("abc123"))
assertFalse(logs[1].jsonObject.string("detail").contains("keep-secret"))
assertTrue(logs[1].jsonObject.string("detail").contains("***"))
val levelFiltered = parse(
client.get("/api/v1/logs?level=warn") { bearer(alice) }.bodyAsText(),
)
assertEquals(1, levelFiltered["logs"]!!.jsonArray.size)
val bobLogs = parse(client.get("/api/v1/logs") { bearer(bob) }.bodyAsText())
assertTrue(bobLogs["logs"]!!.jsonArray.isEmpty())
val forbidden = client.get("/api/v1/logs?user=alice") { bearer(bob) }
assertEquals(HttpStatusCode.Forbidden, forbidden.status)
val adminView = parse(
client.get("/api/v1/logs?user=alice") { bearer(alice) }.bodyAsText(),
)
assertEquals(2, adminView["logs"]!!.jsonArray.size)
val invalid = client.post("/api/v1/logs") {
bearer(alice)
contentType(ContentType.Application.Json)
setBody(
"""{"deviceId":"d","platform":"android","appVersion":"1",
"entries":[{"level":"verbose","tag":"x","message":"y"}]}""",
)
}
assertEquals(HttpStatusCode.BadRequest, invalid.status)
val many = (1..201).joinToString(",") {
"""{"level":"info","tag":"x","message":"m"}"""
}
val tooMany = client.post("/api/v1/logs") {
bearer(alice)
contentType(ContentType.Application.Json)
setBody(
"""{"deviceId":"d","platform":"android","appVersion":"1","entries":[$many]}""",
)
}
assertEquals(HttpStatusCode.BadRequest, tooMany.status)
val cleared = client.delete("/api/v1/logs") { bearer(alice) }
assertEquals(HttpStatusCode.OK, cleared.status)
assertEquals(2, parse(cleared.bodyAsText())["deleted"]!!.jsonPrimitive.int)
assertTrue(
parse(client.get("/api/v1/logs") { bearer(alice) }.bodyAsText())["logs"]!!
.jsonArray.isEmpty(),
)
}
}
}
@Test @Test
fun `moods own their songs and are private`() { fun `moods own their songs and are private`() {
seededContext().use { seed -> seededContext().use { seed ->
@@ -644,6 +727,7 @@ class ApiRoutesTest {
deezerQuality = 0, deezerQuality = 0,
loudnessEnabled = true, loudnessEnabled = true,
loudnessTargetLufs = -16.0, loudnessTargetLufs = -16.0,
clientLogRetentionDays = 14,
startScan = { false }, startScan = { false },
scanStatus = { ScanStatus(false, 0) }, scanStatus = { ScanStatus(false, 0) },
) )
@@ -0,0 +1,110 @@
package win.taptap.musicmanager.db
import java.nio.file.Files
import java.time.Instant
import java.time.temporal.ChronoUnit
import kotlin.test.Test
import kotlin.test.assertEquals
import kotlin.test.assertFalse
import kotlin.test.assertTrue
import org.jetbrains.exposed.sql.SqlExpressionBuilder.eq
import org.jetbrains.exposed.sql.transactions.transaction
import org.jetbrains.exposed.sql.update
import win.taptap.musicmanager.api.ClientLogBatchDto
import win.taptap.musicmanager.api.ClientLogEntryDto
import win.taptap.musicmanager.config.AppConfig
import win.taptap.musicmanager.config.UserAccount
import win.taptap.musicmanager.config.UsersConfig
class ClientLogRepositoryTest {
@Test
fun `credentials are redacted before they are stored`() {
withDatabase { db ->
assertEquals(1, ClientLogRepository.insert("alice", batch("playback", "Authorization: Bearer supersecret")))
val row = ClientLogRepository.list("alice").single()
assertFalse(row.message.contains("supersecret"))
assertFalse(row.detail.orEmpty().contains("supersecret"))
assertTrue(row.detail.orEmpty().contains("***"))
}
}
@Test
fun `retention removes old rows and the per-user cap keeps the newest`() {
withDatabase { db ->
ClientLogRepository.insert("alice", batch("one", "first"))
ClientLogRepository.insert("alice", batch("two", "second"))
ClientLogRepository.insert("alice", batch("three", "third"))
// Three rows, keep two: the newest two survive.
assertEquals(1, ClientLogRepository.prune(365, 2))
assertEquals(listOf("third", "second"), ClientLogRepository.list("alice").map { it.message })
// Age the remaining rows past a one-day retention window.
transaction {
ClientLogsTable.update({ ClientLogsTable.userName eq "alice" }) {
it[receivedAt] = Instant.now().minus(10, ChronoUnit.DAYS).toString()
}
}
assertEquals(2, ClientLogRepository.prune(7, 2))
assertTrue(ClientLogRepository.list("alice").isEmpty())
}
}
@Test
fun `users cannot see each other's diagnostics`() {
withDatabase { db ->
ClientLogRepository.insert("alice", batch("a", "alice line"))
ClientLogRepository.insert("bob", batch("b", "bob line"))
assertEquals(listOf("alice line"), ClientLogRepository.list("alice").map { it.message })
assertEquals(listOf("bob line"), ClientLogRepository.list("bob").map { it.message })
assertEquals(1, ClientLogRepository.delete("alice"))
assertTrue(ClientLogRepository.list("alice").isEmpty())
assertEquals(1, ClientLogRepository.list("bob").size)
}
}
private fun withDatabase(block: (DatabaseHandle) -> Unit) {
val dir = Files.createTempDirectory("client-logs-test")
val users = UsersConfig(
listOf(
UserAccount("alice", "alice-pw", admin = true),
UserAccount("bob", "bob-pw", admin = false),
),
)
val db = DatabaseFactory.connect(
AppConfig(
port = 0,
dataDir = dir,
usersFile = dir.resolve("users.yaml"),
ffmpegPath = "ffmpeg",
),
)
try {
ConfigRepository.sync(users)
block(db)
} finally {
db.close()
}
}
private fun batch(tag: String, message: String): ClientLogBatchDto =
ClientLogBatchDto(
deviceId = "test-device",
platform = "desktop",
appVersion = "0.4.3",
entries = listOf(
ClientLogEntryDto(
timestamp = System.currentTimeMillis(),
level = "warn",
tag = tag,
message = message,
detail = "request failed: Authorization: Bearer supersecret",
sessionId = "run-test",
),
),
)
}