From 6e2bb400d14af51603ca3ba4a8b6725b81029683 Mon Sep 17 00:00:00 2001 From: TapTap Date: Wed, 11 Nov 2026 17:51:49 +0100 Subject: [PATCH] add readme --- README.md | 429 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 429 insertions(+) create mode 100644 README.md diff --git a/README.md b/README.md new file mode 100644 index 0000000..f05998d --- /dev/null +++ b/README.md @@ -0,0 +1,429 @@ +# TapTap Homelab — Kubernetes on Proxmox + +A self-hosted homelab built around a single-node **K3s** cluster running on **NixOS** inside a **Proxmox VE** VM. Routing is handled by an **OPNsense** VM, storage and backups by a **TrueNAS** VM, and the Kubernetes layer is delivered declaratively with **ArgoCD**, **Kustomize** and **Helm**. + +![Kubernetes](https://img.shields.io/badge/Kubernetes-K3s-326CE5?logo=kubernetes&logoColor=white) +![Proxmox](https://img.shields.io/badge/Proxmox-VE-E57000?logo=proxmox&logoColor=white) +![NixOS](https://img.shields.io/badge/NixOS-26.05-5277C3?logo=nixos&logoColor=white) +![OPNsense](https://img.shields.io/badge/OPNsense-Router-D94F00) +![TrueNAS](https://img.shields.io/badge/TrueNAS-Storage-0095D5) +![ArgoCD](https://img.shields.io/badge/ArgoCD-GitOps-EF7B4D?logo=argo&logoColor=white) +![Traefik](https://img.shields.io/badge/Traefik-Ingress-24A1C1?logo=traefik&logoColor=white) +![Vault](https://img.shields.io/badge/Vault-Secrets-FFEC6E?logo=vault&logoColor=black) +![Cloudflare](https://img.shields.io/badge/Cloudflare-DNS-F38020?logo=cloudflare&logoColor=white) + +> **Last verified:** 2026-11-11 against the live cluster and host. + +--- + +## Contents + +- [Overview](#overview) +- [At a glance](#at-a-glance) +- [Architecture](#architecture) +- [Infrastructure layers](#infrastructure-layers) +- [Network and edge](#network-and-edge) +- [Platform components](#platform-components) +- [Infrastructure as Code](#infrastructure-as-code) +- [Repository layout](#repository-layout) +- [Service catalogue](#service-catalogue) +- [Backup and recovery](#backup-and-recovery) +- [Observability](#observability) +- [Security](#security) +- [Engineering highlights](#engineering-highlights) +- [Known limitations and roadmap](#known-limitations-and-roadmap) + +--- + +## Overview + +This repository contains the Kubernetes manifests for a personal homelab. The goal is to run useful self-hosted services while treating the whole stack as code: the host is provisioned and configured with a declarative NixOS flake, and the cluster is reconciled continuously by ArgoCD from Git. + +The platform runs as a set of virtual machines on a single Proxmox VE host: + +- **OPNsense** — routing, firewall, DHCP, NAT and port forwarding. +- **TrueNAS** — mirrored storage, NFS exports and the primary data copy. +- **NixOS / K3s** — the Kubernetes node, host-level automation and Borg backups. + +The cluster is intentionally small and single-node, but the delivery model is the same one used in larger environments: declarative configuration, Git as the source of truth, automated reconciliation, runtime secret injection, TLS automation and tested backups. + +## At a glance + +| Layer | Technology | Role | +| --- | --- | --- | +| Virtualisation | Proxmox VE | Hosts the OPNsense, TrueNAS and K3s VMs | +| Routing / firewall | OPNsense VM | LAN gateway, DHCP, NAT, port forwarding | +| Storage | TrueNAS VM | Mirrored storage, NFS export | +| Host OS | NixOS 26.05 | Declarative, reproducible host configuration | +| Kubernetes | K3s `v1.35.4+k3s1` | Single-node cluster with embedded etcd | +| Ingress | Traefik `3.6.13` via K3s ServiceLB | HTTP/HTTPS entry point on `192.168.1.119` | +| GitOps | ArgoCD `v3.5.4` | Continuous reconciliation from Gitea | +| Packaging | Kustomize + Helm | Plain YAML, Kustomize composition, rendered Helm charts | +| Secrets | HashiCorp Vault + External Secrets Operator `2.12.0` | Runtime secret injection | +| TLS | cert-manager `1.13.2` + Let's Encrypt | Automated certificates via Cloudflare DNS-01 | +| Source control / CI | Gitea + Gitea Actions runner | Git hosting and CI job execution | +| Remote access | Twingate | Zero-trust access to private resources | +| Backups | BorgBackup + rclone | Weekly local backups plus offsite copy | + +## Architecture + +```mermaid +flowchart TB + Internet((Internet)) + + subgraph CF["Cloudflare"] + DNS["DNS + DDNS
*.tap-tap.win"] + end + + subgraph PVE["Proxmox VE host"] + OPN["OPNsense VM
Router / Firewall / DHCP / NAT"] + NAS["TrueNAS VM
Mirrored storage
192.168.1.110"] + + subgraph K3S["K3s VM — NixOS 26.05 — 192.168.1.119"] + SVC["Traefik Ingress
via K3s ServiceLB"] + ARGO["ArgoCD"] + APPS["Application workloads"] + VAULT["HashiCorp Vault"] + ESO["External Secrets Operator"] + CERT["cert-manager"] + + ARGO -->|"reconciles"| APPS + ESO -->|"reads secrets"| VAULT + CERT -->|"issues TLS"| SVC + SVC --> APPS + end + end + + TW["Twingate connector"] + + Internet --> DNS + DNS --> OPN + OPN -->|"HTTP / HTTPS"| SVC + NAS -->|"NFS"| K3S + Internet --> TW + TW --> K3S +``` + +Traffic flow: + +1. Public DNS records live in Cloudflare and are kept current by `cloudflare-ddns`. +2. OPNsense forwards HTTP/HTTPS to the K3s node at `192.168.1.119`. +3. Traefik routes requests to the correct Kubernetes Service based on the ingress host. +4. Twingate provides private, identity-aware access without exposing services publicly. +5. TrueNAS exports storage over NFS to the Kubernetes node for Nextcloud and media workloads. + +## Infrastructure layers + +| Component | Platform | Address | Responsibility | +| --- | --- | --- | --- | +| Proxmox VE host | Host | — | Virtualisation for all infrastructure VMs | +| OPNsense | VM on Proxmox | LAN gateway | Routing, firewall, DHCP, NAT, port forwarding | +| TrueNAS | VM on Proxmox | `192.168.1.110` | Mirrored storage pool and NFS export | +| K3s node | NixOS VM on Proxmox | `192.168.1.119` | Single-node Kubernetes, host automation, Borg repository | + +### Proxmox VE + +The Proxmox host is the virtualisation layer. Keeping routing, storage and compute in separate VMs makes each role independently configurable and snapshot-able, while still fitting the whole platform on one physical machine. + +### OPNsense + +OPNsense is the LAN router and firewall. It provides: + +- Routing between the LAN and the internet. +- DHCP for the local network. +- NAT and port forwarding for HTTP/HTTPS to the Kubernetes ingress. +- A natural segmentation point between the public edge and internal services. + +### TrueNAS + +TrueNAS provides the mirrored storage pool and exports the Nextcloud data set over NFS: + +```text +192.168.1.110:/mnt/MainMirroredStorage/nextcloud +``` + +The K3s node mounts it at `/root/kubernetes/nextcloud/data/mount` with systemd automount and a 10-minute idle timeout, so the host can boot even when the NAS is unavailable. + +### NixOS / K3s + +The Kubernetes node is a NixOS VM managed by a separate NixOS flake configuration. The server profile configures: + +- K3s in `server` mode with `clusterInit = true`. +- Node IP and TLS SAN `192.168.1.119`. +- Firewall rules for Kubernetes, HTTP/HTTPS, Matter, Thread and mDNS. +- The TrueNAS NFS mount and the external backup disk. +- Weekly Borg backups and the Nextcloud cron job. + +The live node reports `NixOS 26.05` and `K3s v1.35.4+k3s1` with embedded etcd. + +## Network and edge + +| Item | Value | +| --- | --- | +| LAN | `192.168.1.0/24` | +| Kubernetes node | `192.168.1.119` | +| TrueNAS | `192.168.1.110` | +| Public domain | `*.tap-tap.win` | +| Ingress controller | Traefik `3.6.13` | +| LoadBalancer implementation | K3s ServiceLB (no MetalLB) | +| Ingress address | `192.168.1.119` on ports 80/443 | +| Traefik NodePorts | `30123` (HTTP), `32467` (HTTPS) | + +**Load balancing:** MetalLB was previously used and has been removed. K3s built-in ServiceLB now implements `LoadBalancer` services by binding the node IP. MetalLB CRDs remain in the cluster as leftovers from the previous installation, but there is no MetalLB namespace, controller or address pool. + +**DNS and exposure:** Cloudflare hosts public DNS, and `cloudflare-ddns` keeps A records pointed at the current public address. OPNsense forwards HTTP/HTTPS to Traefik. Twingate is used for zero-trust access to private services. A `cloudflared` manifest exists in the repository, but the connector is not deployed in the cluster. + +**TLS:** cert-manager issues certificates from Let's Encrypt using the Cloudflare DNS-01 challenge. Certificates are currently ready for ArgoCD, Gitea, Vault, Home Assistant, Jellyfin, Nextcloud, nginx2, OnlyOffice and Plex. The nginx demo and Vaultwarden ingresses are HTTP-only at the ingress level. + +**Internal DNS:** Kubernetes uses CoreDNS for cluster service discovery. Public names resolve through Cloudflare, while TrueNAS provides persistent storage over NFS. + +## Platform components + +| Component | Namespace | Version / image | Purpose | +| --- | --- | --- | --- | +| ArgoCD | `argocd` | `v3.5.4` | GitOps reconciliation and application management | +| Traefik | `kube-system` | `3.6.13` | Ingress controller and TLS termination | +| K3s ServiceLB | `kube-system` | K3s built-in | LoadBalancer implementation on the node IP | +| cert-manager | `cert-manager` | `1.13.2` | Automated Let's Encrypt certificates | +| HashiCorp Vault | `hashicorp-vault` | chart `0.34.1` | Central secret storage (KV v2) | +| External Secrets Operator | `external-secrets` | `2.12.0` | Syncs Vault secrets into Kubernetes Secrets | +| cloudflare-ddns | `cloudflare-ddns` | `favonia/cloudflare-ddns` | Keeps public DNS records current | +| Gitea | `gitea` | `gitea/gitea` | Git hosting for this repository | +| Gitea runner | `gitea` | `gitea/runner` + Docker-in-Docker | CI job execution | +| metrics-server | `kube-system` | `0.8.1` | Resource metrics for the Kubernetes API | +| local-path provisioner | `kube-system` | `v0.0.35` | Local persistent volumes | +| CoreDNS | `kube-system` | `1.14.2` | Cluster DNS | + +`local-path` is the default StorageClass. `local-persistent-storage` is a retained local-path class used by Vault. Most stateful services use `hostPath` volumes on the K3s VM or the TrueNAS NFS mount. + +## Infrastructure as Code + +The infrastructure is split into two declarative layers. + +### Host layer — NixOS flake + +The NixOS host configuration is a declarative flake with separate profiles for laptops, an AI lab machine and the `server` node. Inputs target specific Nixpkgs release channels (`26.05`, `25.11` and `unstable`) plus `nixos-hardware`, `nixos-wsl` and `nixos-generators`. + +The server profile is the source of truth for: + +- K3s server configuration and firewall rules. +- NFS mounts and backup disk mounts. +- Borg backup jobs. +- Kernel modules, timezone, SSH and base packages. +- The Nextcloud OCC cron job. + +Typical host change: + +```bash +sudo nixos-rebuild switch --flake .#server +``` + +Provisioning is reproducible too: the flake can build a custom NixOS install ISO, and an automated install script partitions the disks, clones the configuration repositories and runs `nixos-install`. Rebuilding the node does not rely on hand-edited files on the machine. + +### Cluster layer — GitOps + +Kubernetes resources use plain YAML, Kustomize and Helm charts. ArgoCD runs an `ApplicationSet` that watches a fixed list of directories in this repository and creates one ArgoCD Application per directory. + +Key properties: + +- Automated sync from the Gitea repository. +- `CreateNamespace=true` for new applications. +- `ServerSideApply=true` to avoid apply conflicts. +- Helm charts rendered through Kustomize with `--enable-helm`. +- A root application also tracks the repository root. + +Adding a service follows a predictable flow: + +1. Create a directory with a `kustomization.yaml` and the required resources. +2. Add the directory to the `ApplicationSet` path list. +3. Commit and push to Gitea. +4. ArgoCD detects the change and reconciles the cluster. + +A manual render/apply for debugging looks like: + +```bash +kustomize build --enable-helm | kubectl apply --server-side -f - +``` + +### Separation of concerns + +| Layer | Managed by | Examples | +| --- | --- | --- | +| Host | NixOS flake | K3s, NFS mounts, firewall, Borg | +| Platform | ArgoCD / Helm / Kustomize | Traefik, cert-manager, Vault, ESO, ArgoCD | +| Applications | Kustomize | Nextcloud, Plex, Jellyfin, OnlyOffice, Home Assistant | + +## Repository layout + +| Path | Purpose | +| --- | --- | +| `argocd/` | ArgoCD installation, root application and ApplicationSet | +| `cert-manager/` | cert-manager, Let's Encrypt ClusterIssuer, Cloudflare token sync | +| `external-secrets/` | External Secrets Operator and Vault ClusterSecretStore | +| `hashicorp-vault/` | Vault Helm chart, UI/ingress and storage configuration | +| `cloudflare-ddns/` | Dynamic DNS updater for public hostnames | +| `gitea/` | Gitea server and CI runner | +| `homeassistant/` | Home Assistant, Mosquitto, Zigbee2MQTT, Matter and OTBR | +| `nextcloud/` | Nextcloud, MariaDB, Redis, ingress and secrets | +| `onlyoffice/` | OnlyOffice Document Server | +| `plex/` | Plex Media Server | +| `jellyfin/` | Jellyfin media server | +| `vaultwarden/` | Vaultwarden password manager | +| `twingate/` | Twingate connector | +| `nginx/` | nginx demo workload and an alternative Terraform/Helm deployment | +| `local-persistent-storage-class.yaml` | Retained local-path StorageClass | + +`data/` and `charts/` are ignored by Git. Application data lives on the host, on TrueNAS storage, or on the external backup disk and is not tracked in this repository. + +Several directories are kept for reference or future use but are **not currently deployed**. These are listed in the service catalogue. + +## Service catalogue + +### Running now + +| Service | Namespace | Public hostname | Notes | +| --- | --- | --- | --- | +| ArgoCD | `argocd` | `argocd.tap-tap.win` | GitOps UI and API | +| HashiCorp Vault | `hashicorp-vault` | `hashicorp-vault.tap-tap.win` | KV v2 secret store and UI | +| Gitea | `gitea` | `gitea.tap-tap.win` | Git hosting; includes an Actions runner with Docker-in-Docker | +| Nextcloud | `nextcloud` | `nextcloud.tap-tap.win` | MariaDB 11.4.4 and Redis 7 backends | +| OnlyOffice | `onlyoffice` | `onlyoffice.tap-tap.win` | OnlyOffice Document Server | +| Plex | `plex` | `plex.tap-tap.win` | Media server using the Nextcloud media library | +| Jellyfin | `jellyfin` | `jellyfin.tap-tap.win` | Media server using the Nextcloud media library | +| Vaultwarden | `vaultwarden` | `vaultwarden.tap-tap.win` | Password manager (HTTP ingress) | +| Home Assistant | `homeassistant` | `homeassistant.tap-tap.win` | Smart-home hub | +| Mosquitto | `homeassistant` | internal | MQTT broker for Zigbee2MQTT and Home Assistant | +| Zigbee2MQTT | `homeassistant` | NodePort `30080` / `30081` | Zigbee bridge with USB stick passthrough | +| Matter Server | `homeassistant` | internal | Matter device integration | +| OpenThread Border Router | `homeassistant` | host network | Thread border router with USB radio passthrough | +| Twingate | `twingate` | — | Zero-trust connector | +| Cloudflare DDNS | `cloudflare-ddns` | — | Updates public DNS records | +| nginx demo | `nginx` | `nginx.tap-tap.win`, `nginx2.tap-tap.win` | Simple ingress demo; `nginx.tap-tap.win` is HTTP-only, `nginx2` uses TLS | + +ArgoCD currently reports 14 application directories plus the root application, all `Synced` and `Healthy`. + +### Present in the repository but not deployed + +These directories contain manifests that are not currently running in the cluster: + +| Path | What it contains | +| --- | --- | +| `monitoring/` | kube-prometheus-stack (Prometheus, Grafana, Alertmanager) | +| `grafana/` | Standalone Grafana deployment, superseded by the monitoring chart | +| `pihole/` | Pi-hole DNS and ad blocking | +| `papertrader/` | Custom PaperTrader server and browser images | +| `homarr/` | Homarr dashboard | +| `audiobookshelf/` | Audiobookshelf server | +| `navidrome/` | Navidrome music server | +| `gitlab/` | GitLab CE | +| `cloudflared/` | Cloudflare Tunnel connector | +| `metallb/` | Previous MetalLB manifests; replaced by K3s ServiceLB | + +### Legacy and local working directories + +| Path | Status | +| --- | --- | +| `idrive/` | Tracked but unused. Backups are handled by Borg, not IDrive. | +| `oracle/` | Local, untracked Terraform experiment for an Oracle Cloud free-tier VM; not part of the live platform. | +| `old/` | Local, untracked early prototype scripts and migration experiments. | +| `downloader/` | Local, untracked placeholder directory with no active manifests. | + +## Backup and recovery + +The backup design keeps multiple independent copies of the data: + +| Copy | Location | Method | Frequency | +| --- | --- | --- | --- | +| Primary | TrueNAS mirrored storage | NFS export to the K3s node | Automount, 10-minute idle timeout | +| Local backup | `/mnt/backup/borg` on a 3.6 TB external USB disk | BorgBackup | Weekly | +| Offsite | Google Drive `Backup` folder | `rclone sync` of the Borg repository | Weekly, Thursday 04:30 | + +### Borg + +The NixOS host runs a weekly Borg job: + +- **Paths:** `/root` and `/mnt/data`. +- **Repository:** `/mnt/backup/borg`. +- **Compression:** `auto,zstd`. +- **Encryption:** keyfile mode. +- **Checkpoint interval:** 300 seconds. +- **Retention:** 1 daily, 2 weekly, 2 monthly. +- **Failure handling:** restart on failure after 30 seconds. + +The latest run completed successfully. Deduplication keeps incremental snapshots small: a run representing roughly 3 TB of logical data produced about 387 MB of deduplicated archive data. + +### Offsite copy + +A systemd timer runs every Thursday at 04:30 and synchronises the entire Borg repository to Google Drive: + +```text +rclone sync /mnt/backup/borg gdrive:Backup +``` + +The timer is persistent, so a missed run is executed after the next boot. + +### What is not backed up + +There is currently **no active Proxmox VM-level backup** and no active Proxmox Backup Server datastore. The host configuration contains a prepared NFS export for a PBS datastore, but it is not in use. + +### Recovery outline + +1. Reinstall or rebuild the NixOS host from the flake (or restore the VM). +2. Mount the TrueNAS NFS export and the external backup disk. +3. Restore application data and configuration from the Borg repository. +4. Let ArgoCD reconcile the Kubernetes manifests from Git. + +## Observability + +The live cluster currently runs only `metrics-server` for resource metrics. The `monitoring/` directory contains a complete kube-prometheus-stack configuration (Prometheus, Grafana, Alertmanager, node-exporter and kube-state-metrics), but it is **not currently deployed**. + +Observability is therefore a known gap and a roadmap item rather than a running feature. + +## Security + +- **Automated TLS:** cert-manager obtains Let's Encrypt certificates through the Cloudflare DNS-01 challenge. Nine certificates are currently issued and ready. +- **Runtime secrets:** Vault stores KV v2 secrets. External Secrets Operator authenticates using Kubernetes service-account auth and syncs only the secrets each workload needs. +- **No credentials in this README:** tokens, passwords, keys and private endpoints are intentionally excluded. +- **Zero-trust remote access:** Twingate provides identity-aware access to private services without exposing them directly. +- **Network separation:** Public ingress passes through OPNsense to Traefik; Twingate provides a separate identity-aware path for private access. The host firewall also permits cluster, Matter/Thread and NodePort ports, so isolation relies on OPNsense and the private LAN. +- **Minimal public surface:** services such as Vault, ArgoCD, Gitea and the media servers are protected by ingress and TLS; platform internals do not have public ingresses. +- **Data protection:** Borg repositories are encrypted, and the offsite copy is synchronised to Google Drive. + +## Engineering highlights + +- **Two-layer Infrastructure as Code:** NixOS owns the host, ArgoCD owns the cluster. Both are declarative and managed as code. +- **Immutable host configuration:** packages, services, filesystems, firewall rules and backup jobs are declared in the NixOS flake, not configured by hand. +- **GitOps delivery:** every application change is reconciled automatically; manual `kubectl apply` is only used for debugging. +- **Versioned platform components:** K3s, ArgoCD, Traefik, cert-manager, Vault and External Secrets use explicit versions. +- **Secretless application manifests:** workloads consume Kubernetes Secrets produced by External Secrets rather than embedding credentials in Git. +- **Layered storage:** TrueNAS provides mirrored network storage, local-path classes provide fast cluster-local volumes, and an external disk holds the Borg repository. +- **3-copy backup chain:** mirrored primary storage, a local Borg repository and an offsite Google Drive copy. +- **Practical home automation:** Home Assistant, MQTT, Zigbee, Matter and Thread run alongside the Kubernetes platform with USB device passthrough and host networking where required. + +## Known limitations and roadmap + +This is a personal homelab, not a production platform. Known limitations: + +- **Single-node cluster:** no high availability; the node and its embedded etcd are a single point of failure. +- **`hostPath` coupling:** many applications bind directly to paths on the K3s VM, which makes rescheduling or multi-node operation harder. +- **Repo/live drift:** several manifests are kept in the repository but are not deployed. This README distinguishes live services from repo-only services. +- **Unpinned images:** some workloads use `:latest` tags; pinning digests or versions would make rollbacks deterministic. +- **Monitoring gap:** Prometheus, Grafana and Alertmanager are defined but not deployed. +- **No VM-level backup:** recovery relies on the NixOS flake plus data-level Borg backups. +- **Host-level secrets:** remaining host secrets should move to a declarative secret manager such as sops-nix or agenix. +- **Flake lock not committed:** the NixOS flake lock file is ignored by Git, so host builds should commit it for fully reproducible package versions. +- **MetalLB leftovers:** CRDs remain from the previous load-balancer setup and can be cleaned up. +- **No automated restore drills:** backups complete successfully, but restores should be tested on a schedule. + +Planned improvements: + +- Deploy the monitoring stack and configure alert routing. +- Pin all container images and add automated manifest validation. +- Move host secrets to sops-nix or agenix. +- Commit the NixOS flake lock file for reproducible host builds. +- Replace selected `hostPath` volumes with PVCs backed by network storage. +- Add restore tests and document recovery time. + +--- + +*This repository documents a personal homelab. It is shared for learning and portfolio purposes and is not intended as a production reference architecture.*