# TapTap Homelab — Kubernetes on Proxmox
A self-hosted homelab built around a single-node **K3s** cluster running on **NixOS** inside a **Proxmox VE** VM. Routing is handled by an **OPNsense** VM, storage and backups by a **TrueNAS** VM, and the Kubernetes layer is delivered declaratively with **ArgoCD**, **Kustomize** and **Helm**.









> **Last verified:** 2026-11-11 against the live cluster and host.
---
## Contents
- [Overview](#overview)
- [At a glance](#at-a-glance)
- [Architecture](#architecture)
- [Infrastructure layers](#infrastructure-layers)
- [Network and edge](#network-and-edge)
- [Platform components](#platform-components)
- [Infrastructure as Code](#infrastructure-as-code)
- [Repository layout](#repository-layout)
- [Service catalogue](#service-catalogue)
- [Backup and recovery](#backup-and-recovery)
- [Observability](#observability)
- [Security](#security)
- [Engineering highlights](#engineering-highlights)
- [Known limitations and roadmap](#known-limitations-and-roadmap)
---
## Overview
This repository contains the Kubernetes manifests for a personal homelab. The goal is to run useful self-hosted services while treating the whole stack as code: the host is provisioned and configured with a declarative NixOS flake, and the cluster is reconciled continuously by ArgoCD from Git.
The platform runs as a set of virtual machines on a single Proxmox VE host:
- **OPNsense** — routing, firewall, DHCP, NAT and port forwarding.
- **TrueNAS** — mirrored storage, NFS exports and the primary data copy.
- **NixOS / K3s** — the Kubernetes node, host-level automation and Borg backups.
The cluster is intentionally small and single-node, but the delivery model is the same one used in larger environments: declarative configuration, Git as the source of truth, automated reconciliation, runtime secret injection, TLS automation and tested backups.
## At a glance
| Layer | Technology | Role |
| --- | --- | --- |
| Virtualisation | Proxmox VE | Hosts the OPNsense, TrueNAS and K3s VMs |
| Routing / firewall | OPNsense VM | LAN gateway, DHCP, NAT, port forwarding |
| Storage | TrueNAS VM | Mirrored storage, NFS export |
| Host OS | NixOS 26.05 | Declarative, reproducible host configuration |
| Kubernetes | K3s `v1.35.4+k3s1` | Single-node cluster with embedded etcd |
| Ingress | Traefik `3.6.13` via K3s ServiceLB | HTTP/HTTPS entry point on `192.168.1.119` |
| GitOps | ArgoCD `v3.5.4` | Continuous reconciliation from Gitea |
| Packaging | Kustomize + Helm | Plain YAML, Kustomize composition, rendered Helm charts |
| Secrets | HashiCorp Vault + External Secrets Operator `2.12.0` | Runtime secret injection |
| TLS | cert-manager `1.13.2` + Let's Encrypt | Automated certificates via Cloudflare DNS-01 |
| Source control / CI | Gitea + Gitea Actions runner | Git hosting and CI job execution |
| Remote access | Twingate | Zero-trust access to private resources |
| Backups | BorgBackup + rclone | Weekly local backups plus offsite copy |
## Architecture
```mermaid
flowchart TB
Internet((Internet))
subgraph CF["Cloudflare"]
DNS["DNS + DDNS
*.tap-tap.win"]
end
subgraph PVE["Proxmox VE host"]
OPN["OPNsense VM
Router / Firewall / DHCP / NAT"]
NAS["TrueNAS VM
Mirrored storage
192.168.1.110"]
subgraph K3S["K3s VM — NixOS 26.05 — 192.168.1.119"]
SVC["Traefik Ingress
via K3s ServiceLB"]
ARGO["ArgoCD"]
APPS["Application workloads"]
VAULT["HashiCorp Vault"]
ESO["External Secrets Operator"]
CERT["cert-manager"]
ARGO -->|"reconciles"| APPS
ESO -->|"reads secrets"| VAULT
CERT -->|"issues TLS"| SVC
SVC --> APPS
end
end
TW["Twingate connector"]
Internet --> DNS
DNS --> OPN
OPN -->|"HTTP / HTTPS"| SVC
NAS -->|"NFS"| K3S
Internet --> TW
TW --> K3S
```
Traffic flow:
1. Public DNS records live in Cloudflare and are kept current by `cloudflare-ddns`.
2. OPNsense forwards HTTP/HTTPS to the K3s node at `192.168.1.119`.
3. Traefik routes requests to the correct Kubernetes Service based on the ingress host.
4. Twingate provides private, identity-aware access without exposing services publicly.
5. TrueNAS exports storage over NFS to the Kubernetes node for Nextcloud and media workloads.
## Infrastructure layers
| Component | Platform | Address | Responsibility |
| --- | --- | --- | --- |
| Proxmox VE host | Host | — | Virtualisation for all infrastructure VMs |
| OPNsense | VM on Proxmox | LAN gateway | Routing, firewall, DHCP, NAT, port forwarding |
| TrueNAS | VM on Proxmox | `192.168.1.110` | Mirrored storage pool and NFS export |
| K3s node | NixOS VM on Proxmox | `192.168.1.119` | Single-node Kubernetes, host automation, Borg repository |
### Proxmox VE
The Proxmox host is the virtualisation layer. Keeping routing, storage and compute in separate VMs makes each role independently configurable and snapshot-able, while still fitting the whole platform on one physical machine.
### OPNsense
OPNsense is the LAN router and firewall. It provides:
- Routing between the LAN and the internet.
- DHCP for the local network.
- NAT and port forwarding for HTTP/HTTPS to the Kubernetes ingress.
- A natural segmentation point between the public edge and internal services.
### TrueNAS
TrueNAS provides the mirrored storage pool and exports the Nextcloud data set over NFS:
```text
192.168.1.110:/mnt/MainMirroredStorage/nextcloud
```
The K3s node mounts it at `/root/kubernetes/nextcloud/data/mount` with systemd automount and a 10-minute idle timeout, so the host can boot even when the NAS is unavailable.
### NixOS / K3s
The Kubernetes node is a NixOS VM managed by a separate NixOS flake configuration. The server profile configures:
- K3s in `server` mode with `clusterInit = true`.
- Node IP and TLS SAN `192.168.1.119`.
- Firewall rules for Kubernetes, HTTP/HTTPS, Matter, Thread and mDNS.
- The TrueNAS NFS mount and the external backup disk.
- Weekly Borg backups and the Nextcloud cron job.
The live node reports `NixOS 26.05` and `K3s v1.35.4+k3s1` with embedded etcd.
## Network and edge
| Item | Value |
| --- | --- |
| LAN | `192.168.1.0/24` |
| Kubernetes node | `192.168.1.119` |
| TrueNAS | `192.168.1.110` |
| Public domain | `*.tap-tap.win` |
| Ingress controller | Traefik `3.6.13` |
| LoadBalancer implementation | K3s ServiceLB (no MetalLB) |
| Ingress address | `192.168.1.119` on ports 80/443 |
| Traefik NodePorts | `30123` (HTTP), `32467` (HTTPS) |
**Load balancing:** MetalLB was previously used and has been removed. K3s built-in ServiceLB now implements `LoadBalancer` services by binding the node IP. MetalLB CRDs remain in the cluster as leftovers from the previous installation, but there is no MetalLB namespace, controller or address pool.
**DNS and exposure:** Cloudflare hosts public DNS, and `cloudflare-ddns` keeps A records pointed at the current public address. OPNsense forwards HTTP/HTTPS to Traefik. Twingate is used for zero-trust access to private services. A `cloudflared` manifest exists in the repository, but the connector is not deployed in the cluster.
**TLS:** cert-manager issues certificates from Let's Encrypt using the Cloudflare DNS-01 challenge. Certificates are currently ready for ArgoCD, Gitea, Vault, Home Assistant, Jellyfin, Nextcloud, nginx2, OnlyOffice and Plex. The nginx demo and Vaultwarden ingresses are HTTP-only at the ingress level.
**Internal DNS:** Kubernetes uses CoreDNS for cluster service discovery. Public names resolve through Cloudflare, while TrueNAS provides persistent storage over NFS.
## Platform components
| Component | Namespace | Version / image | Purpose |
| --- | --- | --- | --- |
| ArgoCD | `argocd` | `v3.5.4` | GitOps reconciliation and application management |
| Traefik | `kube-system` | `3.6.13` | Ingress controller and TLS termination |
| K3s ServiceLB | `kube-system` | K3s built-in | LoadBalancer implementation on the node IP |
| cert-manager | `cert-manager` | `1.13.2` | Automated Let's Encrypt certificates |
| HashiCorp Vault | `hashicorp-vault` | chart `0.34.1` | Central secret storage (KV v2) |
| External Secrets Operator | `external-secrets` | `2.12.0` | Syncs Vault secrets into Kubernetes Secrets |
| cloudflare-ddns | `cloudflare-ddns` | `favonia/cloudflare-ddns` | Keeps public DNS records current |
| Gitea | `gitea` | `gitea/gitea` | Git hosting for this repository |
| Gitea runner | `gitea` | `gitea/runner` + Docker-in-Docker | CI job execution |
| metrics-server | `kube-system` | `0.8.1` | Resource metrics for the Kubernetes API |
| local-path provisioner | `kube-system` | `v0.0.35` | Local persistent volumes |
| CoreDNS | `kube-system` | `1.14.2` | Cluster DNS |
`local-path` is the default StorageClass. `local-persistent-storage` is a retained local-path class used by Vault. Most stateful services use `hostPath` volumes on the K3s VM or the TrueNAS NFS mount.
## Infrastructure as Code
The infrastructure is split into two declarative layers.
### Host layer — NixOS flake
The NixOS host configuration is a declarative flake with separate profiles for laptops, an AI lab machine and the `server` node. Inputs target specific Nixpkgs release channels (`26.05`, `25.11` and `unstable`) plus `nixos-hardware`, `nixos-wsl` and `nixos-generators`.
The server profile is the source of truth for:
- K3s server configuration and firewall rules.
- NFS mounts and backup disk mounts.
- Borg backup jobs.
- Kernel modules, timezone, SSH and base packages.
- The Nextcloud OCC cron job.
Typical host change:
```bash
sudo nixos-rebuild switch --flake .#server
```
Provisioning is reproducible too: the flake can build a custom NixOS install ISO, and an automated install script partitions the disks, clones the configuration repositories and runs `nixos-install`. Rebuilding the node does not rely on hand-edited files on the machine.
### Cluster layer — GitOps
Kubernetes resources use plain YAML, Kustomize and Helm charts. ArgoCD runs an `ApplicationSet` that watches a fixed list of directories in this repository and creates one ArgoCD Application per directory.
Key properties:
- Automated sync from the Gitea repository.
- `CreateNamespace=true` for new applications.
- `ServerSideApply=true` to avoid apply conflicts.
- Helm charts rendered through Kustomize with `--enable-helm`.
- A root application also tracks the repository root.
Adding a service follows a predictable flow:
1. Create a directory with a `kustomization.yaml` and the required resources.
2. Add the directory to the `ApplicationSet` path list.
3. Commit and push to Gitea.
4. ArgoCD detects the change and reconciles the cluster.
A manual render/apply for debugging looks like:
```bash
kustomize build --enable-helm | kubectl apply --server-side -f -
```
### Separation of concerns
| Layer | Managed by | Examples |
| --- | --- | --- |
| Host | NixOS flake | K3s, NFS mounts, firewall, Borg |
| Platform | ArgoCD / Helm / Kustomize | Traefik, cert-manager, Vault, ESO, ArgoCD |
| Applications | Kustomize | Nextcloud, Plex, Jellyfin, OnlyOffice, Home Assistant |
## Repository layout
| Path | Purpose |
| --- | --- |
| `argocd/` | ArgoCD installation, root application and ApplicationSet |
| `cert-manager/` | cert-manager, Let's Encrypt ClusterIssuer, Cloudflare token sync |
| `external-secrets/` | External Secrets Operator and Vault ClusterSecretStore |
| `hashicorp-vault/` | Vault Helm chart, UI/ingress and storage configuration |
| `cloudflare-ddns/` | Dynamic DNS updater for public hostnames |
| `gitea/` | Gitea server and CI runner |
| `homeassistant/` | Home Assistant, Mosquitto, Zigbee2MQTT, Matter and OTBR |
| `nextcloud/` | Nextcloud, MariaDB, Redis, ingress and secrets |
| `onlyoffice/` | OnlyOffice Document Server |
| `plex/` | Plex Media Server |
| `jellyfin/` | Jellyfin media server |
| `vaultwarden/` | Vaultwarden password manager |
| `twingate/` | Twingate connector |
| `nginx/` | nginx demo workload and an alternative Terraform/Helm deployment |
| `local-persistent-storage-class.yaml` | Retained local-path StorageClass |
`data/` and `charts/` are ignored by Git. Application data lives on the host, on TrueNAS storage, or on the external backup disk and is not tracked in this repository.
Several directories are kept for reference or future use but are **not currently deployed**. These are listed in the service catalogue.
## Service catalogue
### Running now
| Service | Namespace | Public hostname | Notes |
| --- | --- | --- | --- |
| ArgoCD | `argocd` | `argocd.tap-tap.win` | GitOps UI and API |
| HashiCorp Vault | `hashicorp-vault` | `hashicorp-vault.tap-tap.win` | KV v2 secret store and UI |
| Gitea | `gitea` | `gitea.tap-tap.win` | Git hosting; includes an Actions runner with Docker-in-Docker |
| Nextcloud | `nextcloud` | `nextcloud.tap-tap.win` | MariaDB 11.4.4 and Redis 7 backends |
| OnlyOffice | `onlyoffice` | `onlyoffice.tap-tap.win` | OnlyOffice Document Server |
| Plex | `plex` | `plex.tap-tap.win` | Media server using the Nextcloud media library |
| Jellyfin | `jellyfin` | `jellyfin.tap-tap.win` | Media server using the Nextcloud media library |
| Vaultwarden | `vaultwarden` | `vaultwarden.tap-tap.win` | Password manager (HTTP ingress) |
| Home Assistant | `homeassistant` | `homeassistant.tap-tap.win` | Smart-home hub |
| Mosquitto | `homeassistant` | internal | MQTT broker for Zigbee2MQTT and Home Assistant |
| Zigbee2MQTT | `homeassistant` | NodePort `30080` / `30081` | Zigbee bridge with USB stick passthrough |
| Matter Server | `homeassistant` | internal | Matter device integration |
| OpenThread Border Router | `homeassistant` | host network | Thread border router with USB radio passthrough |
| Twingate | `twingate` | — | Zero-trust connector |
| Cloudflare DDNS | `cloudflare-ddns` | — | Updates public DNS records |
| nginx demo | `nginx` | `nginx.tap-tap.win`, `nginx2.tap-tap.win` | Simple ingress demo; `nginx.tap-tap.win` is HTTP-only, `nginx2` uses TLS |
ArgoCD currently reports 14 application directories plus the root application, all `Synced` and `Healthy`.
### Present in the repository but not deployed
These directories contain manifests that are not currently running in the cluster:
| Path | What it contains |
| --- | --- |
| `monitoring/` | kube-prometheus-stack (Prometheus, Grafana, Alertmanager) |
| `grafana/` | Standalone Grafana deployment, superseded by the monitoring chart |
| `pihole/` | Pi-hole DNS and ad blocking |
| `papertrader/` | Custom PaperTrader server and browser images |
| `homarr/` | Homarr dashboard |
| `audiobookshelf/` | Audiobookshelf server |
| `navidrome/` | Navidrome music server |
| `gitlab/` | GitLab CE |
| `cloudflared/` | Cloudflare Tunnel connector |
| `metallb/` | Previous MetalLB manifests; replaced by K3s ServiceLB |
### Legacy and local working directories
| Path | Status |
| --- | --- |
| `idrive/` | Tracked but unused. Backups are handled by Borg, not IDrive. |
| `oracle/` | Local, untracked Terraform experiment for an Oracle Cloud free-tier VM; not part of the live platform. |
| `old/` | Local, untracked early prototype scripts and migration experiments. |
| `downloader/` | Local, untracked placeholder directory with no active manifests. |
## Backup and recovery
The backup design keeps multiple independent copies of the data:
| Copy | Location | Method | Frequency |
| --- | --- | --- | --- |
| Primary | TrueNAS mirrored storage | NFS export to the K3s node | Automount, 10-minute idle timeout |
| Local backup | `/mnt/backup/borg` on a 3.6 TB external USB disk | BorgBackup | Weekly |
| Offsite | Google Drive `Backup` folder | `rclone sync` of the Borg repository | Weekly, Thursday 04:30 |
### Borg
The NixOS host runs a weekly Borg job:
- **Paths:** `/root` and `/mnt/data`.
- **Repository:** `/mnt/backup/borg`.
- **Compression:** `auto,zstd`.
- **Encryption:** keyfile mode.
- **Checkpoint interval:** 300 seconds.
- **Retention:** 1 daily, 2 weekly, 2 monthly.
- **Failure handling:** restart on failure after 30 seconds.
The latest run completed successfully. Deduplication keeps incremental snapshots small: a run representing roughly 3 TB of logical data produced about 387 MB of deduplicated archive data.
### Offsite copy
A systemd timer runs every Thursday at 04:30 and synchronises the entire Borg repository to Google Drive:
```text
rclone sync /mnt/backup/borg gdrive:Backup
```
The timer is persistent, so a missed run is executed after the next boot.
### What is not backed up
There is currently **no active Proxmox VM-level backup** and no active Proxmox Backup Server datastore. The host configuration contains a prepared NFS export for a PBS datastore, but it is not in use.
### Recovery outline
1. Reinstall or rebuild the NixOS host from the flake (or restore the VM).
2. Mount the TrueNAS NFS export and the external backup disk.
3. Restore application data and configuration from the Borg repository.
4. Let ArgoCD reconcile the Kubernetes manifests from Git.
## Observability
The live cluster currently runs only `metrics-server` for resource metrics. The `monitoring/` directory contains a complete kube-prometheus-stack configuration (Prometheus, Grafana, Alertmanager, node-exporter and kube-state-metrics), but it is **not currently deployed**.
Observability is therefore a known gap and a roadmap item rather than a running feature.
## Security
- **Automated TLS:** cert-manager obtains Let's Encrypt certificates through the Cloudflare DNS-01 challenge. Nine certificates are currently issued and ready.
- **Runtime secrets:** Vault stores KV v2 secrets. External Secrets Operator authenticates using Kubernetes service-account auth and syncs only the secrets each workload needs.
- **No credentials in this README:** tokens, passwords, keys and private endpoints are intentionally excluded.
- **Zero-trust remote access:** Twingate provides identity-aware access to private services without exposing them directly.
- **Network separation:** Public ingress passes through OPNsense to Traefik; Twingate provides a separate identity-aware path for private access. The host firewall also permits cluster, Matter/Thread and NodePort ports, so isolation relies on OPNsense and the private LAN.
- **Minimal public surface:** services such as Vault, ArgoCD, Gitea and the media servers are protected by ingress and TLS; platform internals do not have public ingresses.
- **Data protection:** Borg repositories are encrypted, and the offsite copy is synchronised to Google Drive.
## Engineering highlights
- **Two-layer Infrastructure as Code:** NixOS owns the host, ArgoCD owns the cluster. Both are declarative and managed as code.
- **Immutable host configuration:** packages, services, filesystems, firewall rules and backup jobs are declared in the NixOS flake, not configured by hand.
- **GitOps delivery:** every application change is reconciled automatically; manual `kubectl apply` is only used for debugging.
- **Versioned platform components:** K3s, ArgoCD, Traefik, cert-manager, Vault and External Secrets use explicit versions.
- **Secretless application manifests:** workloads consume Kubernetes Secrets produced by External Secrets rather than embedding credentials in Git.
- **Layered storage:** TrueNAS provides mirrored network storage, local-path classes provide fast cluster-local volumes, and an external disk holds the Borg repository.
- **3-copy backup chain:** mirrored primary storage, a local Borg repository and an offsite Google Drive copy.
- **Practical home automation:** Home Assistant, MQTT, Zigbee, Matter and Thread run alongside the Kubernetes platform with USB device passthrough and host networking where required.
## Known limitations and roadmap
This is a personal homelab, not a production platform. Known limitations:
- **Single-node cluster:** no high availability; the node and its embedded etcd are a single point of failure.
- **`hostPath` coupling:** many applications bind directly to paths on the K3s VM, which makes rescheduling or multi-node operation harder.
- **Repo/live drift:** several manifests are kept in the repository but are not deployed. This README distinguishes live services from repo-only services.
- **Unpinned images:** some workloads use `:latest` tags; pinning digests or versions would make rollbacks deterministic.
- **Monitoring gap:** Prometheus, Grafana and Alertmanager are defined but not deployed.
- **No VM-level backup:** recovery relies on the NixOS flake plus data-level Borg backups.
- **Host-level secrets:** remaining host secrets should move to a declarative secret manager such as sops-nix or agenix.
- **Flake lock not committed:** the NixOS flake lock file is ignored by Git, so host builds should commit it for fully reproducible package versions.
- **MetalLB leftovers:** CRDs remain from the previous load-balancer setup and can be cleaned up.
- **No automated restore drills:** backups complete successfully, but restores should be tested on a schedule.
Planned improvements:
- Deploy the monitoring stack and configure alert routing.
- Pin all container images and add automated manifest validation.
- Move host secrets to sops-nix or agenix.
- Commit the NixOS flake lock file for reproducible host builds.
- Replace selected `hostPath` volumes with PVCs backed by network storage.
- Add restore tests and document recovery time.
---
*This repository documents a personal homelab. It is shared for learning and portfolio purposes and is not intended as a production reference architecture.*