Disaster Recovery - Overview¶

Scope: Complete homelab rebuild from zero. Use this guide when the physical Proxmox server (chronobyte) is destroyed, unbootable, or otherwise unrecoverable.

A reader with access to the Bitwarden vault and minimal technical knowledge can follow this guide end-to-end and restore the entire homelab stack.

What is this Homelab?¶

This homelab runs a small Kubernetes cluster on a single physical server using several layers of technology stacked on top of each other. Here is what each layer does:

graph TD
    apps["Your Applications<br/><i>Jellyfin, Uptime Kuma, Authentik, etc.</i>"]
    k3s["Kubernetes (k3s)<br/><i>orchestrates containers</i>"]
    vms["Virtual Machines<br/><i>4 VMs created by OpenTofu</i>"]
    proxmox["Proxmox VE<br/><i>hypervisor running on bare metal</i>"]
    server["Physical Server - chronobyte"]

    apps --> k3s
    k3s --> vms
    vms --> proxmox
    proxmox --> server

    subgraph cloud["Supporting services (external / cloud)"]
        ts["Tailscale - secure private networking"]
        cf["Cloudflare - public DNS records"]
        s3["AWS S3 - backups and infrastructure state"]
        gh["GitHub - all code and Kubernetes manifests"]
        bw["Bitwarden - stores all secrets/credentials"]
    end

The key insight: everything except the physical hardware is either stored in the cloud (code in GitHub, state in AWS S3, backups in S3, secrets in Bitwarden) or can be recreated automatically. This is why recovery is possible from zero.

Recovery Phases¶

Follow these phases in order. Each phase links to a dedicated document with full details.

Total estimated time: 2–3 hours (assuming Proxmox installs cleanly and VMs boot without issues)

What is NOT Lost in a Hardware Failure¶

Because critical state lives outside the physical server:

Technology Reference¶

Not sure what a technology does or how it works? These pages explain each one from scratch:

Services Running on the Cluster¶

Quick-Reference: Infrastructure Map¶

graph TD
    subgraph phys["Physical Server: chronobyte<br/>OS: Proxmox VE | LAN: <proxmox-lan-ip> | Tailscale: chronobyte.tailnet.ts.net"]
        srv["VM: k3s-server (VMID 102)<br/>LAN: <k3s-server-lan-ip> | Tailscale: <k3s-server-ts-ip><br/>Role: Kubernetes control plane"]
        ag1["VM: k3s-agent-1 (VMID 101)<br/>LAN: <k3s-agent-1-lan-ip> | Tailscale: <k3s-agent-1-ts-ip><br/>Role: Kubernetes worker node"]
        ag2["VM: k3s-agent-2 (VMID 103)<br/>LAN: <k3s-agent-2-lan-ip> | Tailscale: <k3s-agent-2-ts-ip><br/>Role: Kubernetes worker node"]
        gs["VM: game-server (VMID 104)<br/>LAN: DHCP | Tailscale: auto-assigned<br/>Role: Minecraft game server"]
    end
    srv --- ag1
    srv --- ag2

Before You Start¶

1. Read the Prerequisites page first - without the Bitwarden secrets, recovery is impossible.
2. Do not skip steps - each phase depends on the previous one completing successfully.
3. Use GitHub Actions where possible - the recommended path uses pre-built workflows that handle secrets automatically.
4. Check the Technology Reference pages if you are unfamiliar with a tool before using it.