add readme
This commit is contained in:
1 parent
1ca684d267
commit
6e2bb400d1
1 file changed
+429
@@ -0,0 +1,429 @@
|
|||||||
|
# TapTap Homelab — Kubernetes on Proxmox
|
||||||
|
|
||||||
|
A self-hosted homelab built around a single-node **K3s** cluster running on **NixOS** inside a **Proxmox VE** VM. Routing is handled by an **OPNsense** VM, storage and backups by a **TrueNAS** VM, and the Kubernetes layer is delivered declaratively with **ArgoCD**, **Kustomize** and **Helm**.
|
||||||
|
|
||||||
|

|
||||||
|

|
||||||
|

|
||||||
|

|
||||||
|

|
||||||
|

|
||||||
|

|
||||||
|

|
||||||
|

|
||||||
|
|
||||||
|
> **Last verified:** 2026-11-11 against the live cluster and host.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Contents
|
||||||
|
|
||||||
|
- [Overview](#overview)
|
||||||
|
- [At a glance](#at-a-glance)
|
||||||
|
- [Architecture](#architecture)
|
||||||
|
- [Infrastructure layers](#infrastructure-layers)
|
||||||
|
- [Network and edge](#network-and-edge)
|
||||||
|
- [Platform components](#platform-components)
|
||||||
|
- [Infrastructure as Code](#infrastructure-as-code)
|
||||||
|
- [Repository layout](#repository-layout)
|
||||||
|
- [Service catalogue](#service-catalogue)
|
||||||
|
- [Backup and recovery](#backup-and-recovery)
|
||||||
|
- [Observability](#observability)
|
||||||
|
- [Security](#security)
|
||||||
|
- [Engineering highlights](#engineering-highlights)
|
||||||
|
- [Known limitations and roadmap](#known-limitations-and-roadmap)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
This repository contains the Kubernetes manifests for a personal homelab. The goal is to run useful self-hosted services while treating the whole stack as code: the host is provisioned and configured with a declarative NixOS flake, and the cluster is reconciled continuously by ArgoCD from Git.
|
||||||
|
|
||||||
|
The platform runs as a set of virtual machines on a single Proxmox VE host:
|
||||||
|
|
||||||
|
- **OPNsense** — routing, firewall, DHCP, NAT and port forwarding.
|
||||||
|
- **TrueNAS** — mirrored storage, NFS exports and the primary data copy.
|
||||||
|
- **NixOS / K3s** — the Kubernetes node, host-level automation and Borg backups.
|
||||||
|
|
||||||
|
The cluster is intentionally small and single-node, but the delivery model is the same one used in larger environments: declarative configuration, Git as the source of truth, automated reconciliation, runtime secret injection, TLS automation and tested backups.
|
||||||
|
|
||||||
|
## At a glance
|
||||||
|
|
||||||
|
| Layer | Technology | Role |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| Virtualisation | Proxmox VE | Hosts the OPNsense, TrueNAS and K3s VMs |
|
||||||
|
| Routing / firewall | OPNsense VM | LAN gateway, DHCP, NAT, port forwarding |
|
||||||
|
| Storage | TrueNAS VM | Mirrored storage, NFS export |
|
||||||
|
| Host OS | NixOS 26.05 | Declarative, reproducible host configuration |
|
||||||
|
| Kubernetes | K3s `v1.35.4+k3s1` | Single-node cluster with embedded etcd |
|
||||||
|
| Ingress | Traefik `3.6.13` via K3s ServiceLB | HTTP/HTTPS entry point on `192.168.1.119` |
|
||||||
|
| GitOps | ArgoCD `v3.5.4` | Continuous reconciliation from Gitea |
|
||||||
|
| Packaging | Kustomize + Helm | Plain YAML, Kustomize composition, rendered Helm charts |
|
||||||
|
| Secrets | HashiCorp Vault + External Secrets Operator `2.12.0` | Runtime secret injection |
|
||||||
|
| TLS | cert-manager `1.13.2` + Let's Encrypt | Automated certificates via Cloudflare DNS-01 |
|
||||||
|
| Source control / CI | Gitea + Gitea Actions runner | Git hosting and CI job execution |
|
||||||
|
| Remote access | Twingate | Zero-trust access to private resources |
|
||||||
|
| Backups | BorgBackup + rclone | Weekly local backups plus offsite copy |
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
```mermaid
|
||||||
|
flowchart TB
|
||||||
|
Internet((Internet))
|
||||||
|
|
||||||
|
subgraph CF["Cloudflare"]
|
||||||
|
DNS["DNS + DDNS<br/>*.tap-tap.win"]
|
||||||
|
end
|
||||||
|
|
||||||
|
subgraph PVE["Proxmox VE host"]
|
||||||
|
OPN["OPNsense VM<br/>Router / Firewall / DHCP / NAT"]
|
||||||
|
NAS["TrueNAS VM<br/>Mirrored storage<br/>192.168.1.110"]
|
||||||
|
|
||||||
|
subgraph K3S["K3s VM — NixOS 26.05 — 192.168.1.119"]
|
||||||
|
SVC["Traefik Ingress<br/>via K3s ServiceLB"]
|
||||||
|
ARGO["ArgoCD"]
|
||||||
|
APPS["Application workloads"]
|
||||||
|
VAULT["HashiCorp Vault"]
|
||||||
|
ESO["External Secrets Operator"]
|
||||||
|
CERT["cert-manager"]
|
||||||
|
|
||||||
|
ARGO -->|"reconciles"| APPS
|
||||||
|
ESO -->|"reads secrets"| VAULT
|
||||||
|
CERT -->|"issues TLS"| SVC
|
||||||
|
SVC --> APPS
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
TW["Twingate connector"]
|
||||||
|
|
||||||
|
Internet --> DNS
|
||||||
|
DNS --> OPN
|
||||||
|
OPN -->|"HTTP / HTTPS"| SVC
|
||||||
|
NAS -->|"NFS"| K3S
|
||||||
|
Internet --> TW
|
||||||
|
TW --> K3S
|
||||||
|
```
|
||||||
|
|
||||||
|
Traffic flow:
|
||||||
|
|
||||||
|
1. Public DNS records live in Cloudflare and are kept current by `cloudflare-ddns`.
|
||||||
|
2. OPNsense forwards HTTP/HTTPS to the K3s node at `192.168.1.119`.
|
||||||
|
3. Traefik routes requests to the correct Kubernetes Service based on the ingress host.
|
||||||
|
4. Twingate provides private, identity-aware access without exposing services publicly.
|
||||||
|
5. TrueNAS exports storage over NFS to the Kubernetes node for Nextcloud and media workloads.
|
||||||
|
|
||||||
|
## Infrastructure layers
|
||||||
|
|
||||||
|
| Component | Platform | Address | Responsibility |
|
||||||
|
| --- | --- | --- | --- |
|
||||||
|
| Proxmox VE host | Host | — | Virtualisation for all infrastructure VMs |
|
||||||
|
| OPNsense | VM on Proxmox | LAN gateway | Routing, firewall, DHCP, NAT, port forwarding |
|
||||||
|
| TrueNAS | VM on Proxmox | `192.168.1.110` | Mirrored storage pool and NFS export |
|
||||||
|
| K3s node | NixOS VM on Proxmox | `192.168.1.119` | Single-node Kubernetes, host automation, Borg repository |
|
||||||
|
|
||||||
|
### Proxmox VE
|
||||||
|
|
||||||
|
The Proxmox host is the virtualisation layer. Keeping routing, storage and compute in separate VMs makes each role independently configurable and snapshot-able, while still fitting the whole platform on one physical machine.
|
||||||
|
|
||||||
|
### OPNsense
|
||||||
|
|
||||||
|
OPNsense is the LAN router and firewall. It provides:
|
||||||
|
|
||||||
|
- Routing between the LAN and the internet.
|
||||||
|
- DHCP for the local network.
|
||||||
|
- NAT and port forwarding for HTTP/HTTPS to the Kubernetes ingress.
|
||||||
|
- A natural segmentation point between the public edge and internal services.
|
||||||
|
|
||||||
|
### TrueNAS
|
||||||
|
|
||||||
|
TrueNAS provides the mirrored storage pool and exports the Nextcloud data set over NFS:
|
||||||
|
|
||||||
|
```text
|
||||||
|
192.168.1.110:/mnt/MainMirroredStorage/nextcloud
|
||||||
|
```
|
||||||
|
|
||||||
|
The K3s node mounts it at `/root/kubernetes/nextcloud/data/mount` with systemd automount and a 10-minute idle timeout, so the host can boot even when the NAS is unavailable.
|
||||||
|
|
||||||
|
### NixOS / K3s
|
||||||
|
|
||||||
|
The Kubernetes node is a NixOS VM managed by a separate NixOS flake configuration. The server profile configures:
|
||||||
|
|
||||||
|
- K3s in `server` mode with `clusterInit = true`.
|
||||||
|
- Node IP and TLS SAN `192.168.1.119`.
|
||||||
|
- Firewall rules for Kubernetes, HTTP/HTTPS, Matter, Thread and mDNS.
|
||||||
|
- The TrueNAS NFS mount and the external backup disk.
|
||||||
|
- Weekly Borg backups and the Nextcloud cron job.
|
||||||
|
|
||||||
|
The live node reports `NixOS 26.05` and `K3s v1.35.4+k3s1` with embedded etcd.
|
||||||
|
|
||||||
|
## Network and edge
|
||||||
|
|
||||||
|
| Item | Value |
|
||||||
|
| --- | --- |
|
||||||
|
| LAN | `192.168.1.0/24` |
|
||||||
|
| Kubernetes node | `192.168.1.119` |
|
||||||
|
| TrueNAS | `192.168.1.110` |
|
||||||
|
| Public domain | `*.tap-tap.win` |
|
||||||
|
| Ingress controller | Traefik `3.6.13` |
|
||||||
|
| LoadBalancer implementation | K3s ServiceLB (no MetalLB) |
|
||||||
|
| Ingress address | `192.168.1.119` on ports 80/443 |
|
||||||
|
| Traefik NodePorts | `30123` (HTTP), `32467` (HTTPS) |
|
||||||
|
|
||||||
|
**Load balancing:** MetalLB was previously used and has been removed. K3s built-in ServiceLB now implements `LoadBalancer` services by binding the node IP. MetalLB CRDs remain in the cluster as leftovers from the previous installation, but there is no MetalLB namespace, controller or address pool.
|
||||||
|
|
||||||
|
**DNS and exposure:** Cloudflare hosts public DNS, and `cloudflare-ddns` keeps A records pointed at the current public address. OPNsense forwards HTTP/HTTPS to Traefik. Twingate is used for zero-trust access to private services. A `cloudflared` manifest exists in the repository, but the connector is not deployed in the cluster.
|
||||||
|
|
||||||
|
**TLS:** cert-manager issues certificates from Let's Encrypt using the Cloudflare DNS-01 challenge. Certificates are currently ready for ArgoCD, Gitea, Vault, Home Assistant, Jellyfin, Nextcloud, nginx2, OnlyOffice and Plex. The nginx demo and Vaultwarden ingresses are HTTP-only at the ingress level.
|
||||||
|
|
||||||
|
**Internal DNS:** Kubernetes uses CoreDNS for cluster service discovery. Public names resolve through Cloudflare, while TrueNAS provides persistent storage over NFS.
|
||||||
|
|
||||||
|
## Platform components
|
||||||
|
|
||||||
|
| Component | Namespace | Version / image | Purpose |
|
||||||
|
| --- | --- | --- | --- |
|
||||||
|
| ArgoCD | `argocd` | `v3.5.4` | GitOps reconciliation and application management |
|
||||||
|
| Traefik | `kube-system` | `3.6.13` | Ingress controller and TLS termination |
|
||||||
|
| K3s ServiceLB | `kube-system` | K3s built-in | LoadBalancer implementation on the node IP |
|
||||||
|
| cert-manager | `cert-manager` | `1.13.2` | Automated Let's Encrypt certificates |
|
||||||
|
| HashiCorp Vault | `hashicorp-vault` | chart `0.34.1` | Central secret storage (KV v2) |
|
||||||
|
| External Secrets Operator | `external-secrets` | `2.12.0` | Syncs Vault secrets into Kubernetes Secrets |
|
||||||
|
| cloudflare-ddns | `cloudflare-ddns` | `favonia/cloudflare-ddns` | Keeps public DNS records current |
|
||||||
|
| Gitea | `gitea` | `gitea/gitea` | Git hosting for this repository |
|
||||||
|
| Gitea runner | `gitea` | `gitea/runner` + Docker-in-Docker | CI job execution |
|
||||||
|
| metrics-server | `kube-system` | `0.8.1` | Resource metrics for the Kubernetes API |
|
||||||
|
| local-path provisioner | `kube-system` | `v0.0.35` | Local persistent volumes |
|
||||||
|
| CoreDNS | `kube-system` | `1.14.2` | Cluster DNS |
|
||||||
|
|
||||||
|
`local-path` is the default StorageClass. `local-persistent-storage` is a retained local-path class used by Vault. Most stateful services use `hostPath` volumes on the K3s VM or the TrueNAS NFS mount.
|
||||||
|
|
||||||
|
## Infrastructure as Code
|
||||||
|
|
||||||
|
The infrastructure is split into two declarative layers.
|
||||||
|
|
||||||
|
### Host layer — NixOS flake
|
||||||
|
|
||||||
|
The NixOS host configuration is a declarative flake with separate profiles for laptops, an AI lab machine and the `server` node. Inputs target specific Nixpkgs release channels (`26.05`, `25.11` and `unstable`) plus `nixos-hardware`, `nixos-wsl` and `nixos-generators`.
|
||||||
|
|
||||||
|
The server profile is the source of truth for:
|
||||||
|
|
||||||
|
- K3s server configuration and firewall rules.
|
||||||
|
- NFS mounts and backup disk mounts.
|
||||||
|
- Borg backup jobs.
|
||||||
|
- Kernel modules, timezone, SSH and base packages.
|
||||||
|
- The Nextcloud OCC cron job.
|
||||||
|
|
||||||
|
Typical host change:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo nixos-rebuild switch --flake .#server
|
||||||
|
```
|
||||||
|
|
||||||
|
Provisioning is reproducible too: the flake can build a custom NixOS install ISO, and an automated install script partitions the disks, clones the configuration repositories and runs `nixos-install`. Rebuilding the node does not rely on hand-edited files on the machine.
|
||||||
|
|
||||||
|
### Cluster layer — GitOps
|
||||||
|
|
||||||
|
Kubernetes resources use plain YAML, Kustomize and Helm charts. ArgoCD runs an `ApplicationSet` that watches a fixed list of directories in this repository and creates one ArgoCD Application per directory.
|
||||||
|
|
||||||
|
Key properties:
|
||||||
|
|
||||||
|
- Automated sync from the Gitea repository.
|
||||||
|
- `CreateNamespace=true` for new applications.
|
||||||
|
- `ServerSideApply=true` to avoid apply conflicts.
|
||||||
|
- Helm charts rendered through Kustomize with `--enable-helm`.
|
||||||
|
- A root application also tracks the repository root.
|
||||||
|
|
||||||
|
Adding a service follows a predictable flow:
|
||||||
|
|
||||||
|
1. Create a directory with a `kustomization.yaml` and the required resources.
|
||||||
|
2. Add the directory to the `ApplicationSet` path list.
|
||||||
|
3. Commit and push to Gitea.
|
||||||
|
4. ArgoCD detects the change and reconciles the cluster.
|
||||||
|
|
||||||
|
A manual render/apply for debugging looks like:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
kustomize build <directory> --enable-helm | kubectl apply --server-side -f -
|
||||||
|
```
|
||||||
|
|
||||||
|
### Separation of concerns
|
||||||
|
|
||||||
|
| Layer | Managed by | Examples |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| Host | NixOS flake | K3s, NFS mounts, firewall, Borg |
|
||||||
|
| Platform | ArgoCD / Helm / Kustomize | Traefik, cert-manager, Vault, ESO, ArgoCD |
|
||||||
|
| Applications | Kustomize | Nextcloud, Plex, Jellyfin, OnlyOffice, Home Assistant |
|
||||||
|
|
||||||
|
## Repository layout
|
||||||
|
|
||||||
|
| Path | Purpose |
|
||||||
|
| --- | --- |
|
||||||
|
| `argocd/` | ArgoCD installation, root application and ApplicationSet |
|
||||||
|
| `cert-manager/` | cert-manager, Let's Encrypt ClusterIssuer, Cloudflare token sync |
|
||||||
|
| `external-secrets/` | External Secrets Operator and Vault ClusterSecretStore |
|
||||||
|
| `hashicorp-vault/` | Vault Helm chart, UI/ingress and storage configuration |
|
||||||
|
| `cloudflare-ddns/` | Dynamic DNS updater for public hostnames |
|
||||||
|
| `gitea/` | Gitea server and CI runner |
|
||||||
|
| `homeassistant/` | Home Assistant, Mosquitto, Zigbee2MQTT, Matter and OTBR |
|
||||||
|
| `nextcloud/` | Nextcloud, MariaDB, Redis, ingress and secrets |
|
||||||
|
| `onlyoffice/` | OnlyOffice Document Server |
|
||||||
|
| `plex/` | Plex Media Server |
|
||||||
|
| `jellyfin/` | Jellyfin media server |
|
||||||
|
| `vaultwarden/` | Vaultwarden password manager |
|
||||||
|
| `twingate/` | Twingate connector |
|
||||||
|
| `nginx/` | nginx demo workload and an alternative Terraform/Helm deployment |
|
||||||
|
| `local-persistent-storage-class.yaml` | Retained local-path StorageClass |
|
||||||
|
|
||||||
|
`data/` and `charts/` are ignored by Git. Application data lives on the host, on TrueNAS storage, or on the external backup disk and is not tracked in this repository.
|
||||||
|
|
||||||
|
Several directories are kept for reference or future use but are **not currently deployed**. These are listed in the service catalogue.
|
||||||
|
|
||||||
|
## Service catalogue
|
||||||
|
|
||||||
|
### Running now
|
||||||
|
|
||||||
|
| Service | Namespace | Public hostname | Notes |
|
||||||
|
| --- | --- | --- | --- |
|
||||||
|
| ArgoCD | `argocd` | `argocd.tap-tap.win` | GitOps UI and API |
|
||||||
|
| HashiCorp Vault | `hashicorp-vault` | `hashicorp-vault.tap-tap.win` | KV v2 secret store and UI |
|
||||||
|
| Gitea | `gitea` | `gitea.tap-tap.win` | Git hosting; includes an Actions runner with Docker-in-Docker |
|
||||||
|
| Nextcloud | `nextcloud` | `nextcloud.tap-tap.win` | MariaDB 11.4.4 and Redis 7 backends |
|
||||||
|
| OnlyOffice | `onlyoffice` | `onlyoffice.tap-tap.win` | OnlyOffice Document Server |
|
||||||
|
| Plex | `plex` | `plex.tap-tap.win` | Media server using the Nextcloud media library |
|
||||||
|
| Jellyfin | `jellyfin` | `jellyfin.tap-tap.win` | Media server using the Nextcloud media library |
|
||||||
|
| Vaultwarden | `vaultwarden` | `vaultwarden.tap-tap.win` | Password manager (HTTP ingress) |
|
||||||
|
| Home Assistant | `homeassistant` | `homeassistant.tap-tap.win` | Smart-home hub |
|
||||||
|
| Mosquitto | `homeassistant` | internal | MQTT broker for Zigbee2MQTT and Home Assistant |
|
||||||
|
| Zigbee2MQTT | `homeassistant` | NodePort `30080` / `30081` | Zigbee bridge with USB stick passthrough |
|
||||||
|
| Matter Server | `homeassistant` | internal | Matter device integration |
|
||||||
|
| OpenThread Border Router | `homeassistant` | host network | Thread border router with USB radio passthrough |
|
||||||
|
| Twingate | `twingate` | — | Zero-trust connector |
|
||||||
|
| Cloudflare DDNS | `cloudflare-ddns` | — | Updates public DNS records |
|
||||||
|
| nginx demo | `nginx` | `nginx.tap-tap.win`, `nginx2.tap-tap.win` | Simple ingress demo; `nginx.tap-tap.win` is HTTP-only, `nginx2` uses TLS |
|
||||||
|
|
||||||
|
ArgoCD currently reports 14 application directories plus the root application, all `Synced` and `Healthy`.
|
||||||
|
|
||||||
|
### Present in the repository but not deployed
|
||||||
|
|
||||||
|
These directories contain manifests that are not currently running in the cluster:
|
||||||
|
|
||||||
|
| Path | What it contains |
|
||||||
|
| --- | --- |
|
||||||
|
| `monitoring/` | kube-prometheus-stack (Prometheus, Grafana, Alertmanager) |
|
||||||
|
| `grafana/` | Standalone Grafana deployment, superseded by the monitoring chart |
|
||||||
|
| `pihole/` | Pi-hole DNS and ad blocking |
|
||||||
|
| `papertrader/` | Custom PaperTrader server and browser images |
|
||||||
|
| `homarr/` | Homarr dashboard |
|
||||||
|
| `audiobookshelf/` | Audiobookshelf server |
|
||||||
|
| `navidrome/` | Navidrome music server |
|
||||||
|
| `gitlab/` | GitLab CE |
|
||||||
|
| `cloudflared/` | Cloudflare Tunnel connector |
|
||||||
|
| `metallb/` | Previous MetalLB manifests; replaced by K3s ServiceLB |
|
||||||
|
|
||||||
|
### Legacy and local working directories
|
||||||
|
|
||||||
|
| Path | Status |
|
||||||
|
| --- | --- |
|
||||||
|
| `idrive/` | Tracked but unused. Backups are handled by Borg, not IDrive. |
|
||||||
|
| `oracle/` | Local, untracked Terraform experiment for an Oracle Cloud free-tier VM; not part of the live platform. |
|
||||||
|
| `old/` | Local, untracked early prototype scripts and migration experiments. |
|
||||||
|
| `downloader/` | Local, untracked placeholder directory with no active manifests. |
|
||||||
|
|
||||||
|
## Backup and recovery
|
||||||
|
|
||||||
|
The backup design keeps multiple independent copies of the data:
|
||||||
|
|
||||||
|
| Copy | Location | Method | Frequency |
|
||||||
|
| --- | --- | --- | --- |
|
||||||
|
| Primary | TrueNAS mirrored storage | NFS export to the K3s node | Automount, 10-minute idle timeout |
|
||||||
|
| Local backup | `/mnt/backup/borg` on a 3.6 TB external USB disk | BorgBackup | Weekly |
|
||||||
|
| Offsite | Google Drive `Backup` folder | `rclone sync` of the Borg repository | Weekly, Thursday 04:30 |
|
||||||
|
|
||||||
|
### Borg
|
||||||
|
|
||||||
|
The NixOS host runs a weekly Borg job:
|
||||||
|
|
||||||
|
- **Paths:** `/root` and `/mnt/data`.
|
||||||
|
- **Repository:** `/mnt/backup/borg`.
|
||||||
|
- **Compression:** `auto,zstd`.
|
||||||
|
- **Encryption:** keyfile mode.
|
||||||
|
- **Checkpoint interval:** 300 seconds.
|
||||||
|
- **Retention:** 1 daily, 2 weekly, 2 monthly.
|
||||||
|
- **Failure handling:** restart on failure after 30 seconds.
|
||||||
|
|
||||||
|
The latest run completed successfully. Deduplication keeps incremental snapshots small: a run representing roughly 3 TB of logical data produced about 387 MB of deduplicated archive data.
|
||||||
|
|
||||||
|
### Offsite copy
|
||||||
|
|
||||||
|
A systemd timer runs every Thursday at 04:30 and synchronises the entire Borg repository to Google Drive:
|
||||||
|
|
||||||
|
```text
|
||||||
|
rclone sync /mnt/backup/borg gdrive:Backup
|
||||||
|
```
|
||||||
|
|
||||||
|
The timer is persistent, so a missed run is executed after the next boot.
|
||||||
|
|
||||||
|
### What is not backed up
|
||||||
|
|
||||||
|
There is currently **no active Proxmox VM-level backup** and no active Proxmox Backup Server datastore. The host configuration contains a prepared NFS export for a PBS datastore, but it is not in use.
|
||||||
|
|
||||||
|
### Recovery outline
|
||||||
|
|
||||||
|
1. Reinstall or rebuild the NixOS host from the flake (or restore the VM).
|
||||||
|
2. Mount the TrueNAS NFS export and the external backup disk.
|
||||||
|
3. Restore application data and configuration from the Borg repository.
|
||||||
|
4. Let ArgoCD reconcile the Kubernetes manifests from Git.
|
||||||
|
|
||||||
|
## Observability
|
||||||
|
|
||||||
|
The live cluster currently runs only `metrics-server` for resource metrics. The `monitoring/` directory contains a complete kube-prometheus-stack configuration (Prometheus, Grafana, Alertmanager, node-exporter and kube-state-metrics), but it is **not currently deployed**.
|
||||||
|
|
||||||
|
Observability is therefore a known gap and a roadmap item rather than a running feature.
|
||||||
|
|
||||||
|
## Security
|
||||||
|
|
||||||
|
- **Automated TLS:** cert-manager obtains Let's Encrypt certificates through the Cloudflare DNS-01 challenge. Nine certificates are currently issued and ready.
|
||||||
|
- **Runtime secrets:** Vault stores KV v2 secrets. External Secrets Operator authenticates using Kubernetes service-account auth and syncs only the secrets each workload needs.
|
||||||
|
- **No credentials in this README:** tokens, passwords, keys and private endpoints are intentionally excluded.
|
||||||
|
- **Zero-trust remote access:** Twingate provides identity-aware access to private services without exposing them directly.
|
||||||
|
- **Network separation:** Public ingress passes through OPNsense to Traefik; Twingate provides a separate identity-aware path for private access. The host firewall also permits cluster, Matter/Thread and NodePort ports, so isolation relies on OPNsense and the private LAN.
|
||||||
|
- **Minimal public surface:** services such as Vault, ArgoCD, Gitea and the media servers are protected by ingress and TLS; platform internals do not have public ingresses.
|
||||||
|
- **Data protection:** Borg repositories are encrypted, and the offsite copy is synchronised to Google Drive.
|
||||||
|
|
||||||
|
## Engineering highlights
|
||||||
|
|
||||||
|
- **Two-layer Infrastructure as Code:** NixOS owns the host, ArgoCD owns the cluster. Both are declarative and managed as code.
|
||||||
|
- **Immutable host configuration:** packages, services, filesystems, firewall rules and backup jobs are declared in the NixOS flake, not configured by hand.
|
||||||
|
- **GitOps delivery:** every application change is reconciled automatically; manual `kubectl apply` is only used for debugging.
|
||||||
|
- **Versioned platform components:** K3s, ArgoCD, Traefik, cert-manager, Vault and External Secrets use explicit versions.
|
||||||
|
- **Secretless application manifests:** workloads consume Kubernetes Secrets produced by External Secrets rather than embedding credentials in Git.
|
||||||
|
- **Layered storage:** TrueNAS provides mirrored network storage, local-path classes provide fast cluster-local volumes, and an external disk holds the Borg repository.
|
||||||
|
- **3-copy backup chain:** mirrored primary storage, a local Borg repository and an offsite Google Drive copy.
|
||||||
|
- **Practical home automation:** Home Assistant, MQTT, Zigbee, Matter and Thread run alongside the Kubernetes platform with USB device passthrough and host networking where required.
|
||||||
|
|
||||||
|
## Known limitations and roadmap
|
||||||
|
|
||||||
|
This is a personal homelab, not a production platform. Known limitations:
|
||||||
|
|
||||||
|
- **Single-node cluster:** no high availability; the node and its embedded etcd are a single point of failure.
|
||||||
|
- **`hostPath` coupling:** many applications bind directly to paths on the K3s VM, which makes rescheduling or multi-node operation harder.
|
||||||
|
- **Repo/live drift:** several manifests are kept in the repository but are not deployed. This README distinguishes live services from repo-only services.
|
||||||
|
- **Unpinned images:** some workloads use `:latest` tags; pinning digests or versions would make rollbacks deterministic.
|
||||||
|
- **Monitoring gap:** Prometheus, Grafana and Alertmanager are defined but not deployed.
|
||||||
|
- **No VM-level backup:** recovery relies on the NixOS flake plus data-level Borg backups.
|
||||||
|
- **Host-level secrets:** remaining host secrets should move to a declarative secret manager such as sops-nix or agenix.
|
||||||
|
- **Flake lock not committed:** the NixOS flake lock file is ignored by Git, so host builds should commit it for fully reproducible package versions.
|
||||||
|
- **MetalLB leftovers:** CRDs remain from the previous load-balancer setup and can be cleaned up.
|
||||||
|
- **No automated restore drills:** backups complete successfully, but restores should be tested on a schedule.
|
||||||
|
|
||||||
|
Planned improvements:
|
||||||
|
|
||||||
|
- Deploy the monitoring stack and configure alert routing.
|
||||||
|
- Pin all container images and add automated manifest validation.
|
||||||
|
- Move host secrets to sops-nix or agenix.
|
||||||
|
- Commit the NixOS flake lock file for reproducible host builds.
|
||||||
|
- Replace selected `hostPath` volumes with PVCs backed by network storage.
|
||||||
|
- Add restore tests and document recovery time.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
*This repository documents a personal homelab. It is shared for learning and portfolio purposes and is not intended as a production reference architecture.*
|
||||||
Reference in new issue
Block a user