feat(homeserver): NixOS host + ZFS disk layout to replace Proxmox - #169
Open
gapul wants to merge 11 commits into
Open
feat(homeserver): NixOS host + ZFS disk layout to replace Proxmox#169gapul wants to merge 11 commits into
gapul wants to merge 11 commits into
Conversation
First step of dissolving the single-node Proxmox box (pve) into one bare-metal NixOS host. This lands the network layer only: the reverse proxy, the subnet router, and uptime checks. - hosts/homeserver.nix: caddy (21 vhosts from one `sites` table), tailscale subnet router, gatus, podman, ssh, nix settings - hosts/homeserver-hardware-ci.nix: stub so `om ci` can build it from a Mac - flake.nix: mkHomeserver builds both the real host (guarded on the hardware file existing, same as nixos-laptop) and the CI variant Notable choices: - security.acme wildcard cert for *.gapul.net instead of Caddy's cloudflare DNS plugin: one DNS-01 order rather than 25, and no custom caddy build. reloadServices is set, or caddy would keep serving the pre-renewal cert. - The `sites` table drives both the vhosts and the gatus endpoints, so the Caddyfile / uptime-kuma / repo-copy-of-Caddyfile drift can't come back. - gatus probes upstreams directly, not https://<name>.gapul.net, so the checks work before DNS is cut over to this host. - Dropped on the way in: dockge, wud, backrest (replaced by declared restic), and pdf (stirling-pdf is already dead). uptime-kuma's monitors move to gatus. Not here yet: CT101's 37 containers, Home Assistant + matter-server, the Mullvad exit node, and the L2TP/IPsec relay. Reasons are in the host file.
Switches the plan from staging through a NixOS VM on the old Proxmox to replacing it outright, which pulls the disk layout forward: install day needs it, and there is no per-service rollback to lean on. - hosts/homeserver-disk.nix: GPT with a 1GB ESP and a single-disk zpool. ZFS is here specifically to replace vzdump's per-guest snapshots; a second NVMe can be attached later to make it a mirror without reinstalling. Datasets: root, nix (auto-snapshot off, it is reproducible), var/lib (where all service state lives, so a bad container upgrade rolls back without the OS), home. - hosts/homeserver-hardware.nix: committed by hand instead of generated. disko provides fileSystems, so only initrd modules and the CPU vendor are left, which means no uncommitted piece and CI builds what gets installed. - hosts/homeserver-hardware-ci.nix: deleted, the stub has nothing left to stub. - homeserver.nix: hostId, forceImportRoot off, autoScrub, trim, autoSnapshot, and an ARC cap of 2GB. ARC defaults to half of RAM, which would have eaten most of the ~4.7GB this migration exists to recover. - flake.nix: homeserver is unconditional now (no pathExists guard) and gains a diskoConfigurations entry for the install-time format/mount.
gapul
changed the title
…gh caddy The migration wipes Proxmox in one cut, so "it evaluates" is not enough — this is the only verification available before the old install is gone. Two things are otherwise unverifiable from a Mac: - gatus's `settings` is a freeform attrset, so a wrong key type-checks in nix and only fails when gatus parses its YAML at startup - the Caddyfile is assembled by string interpolation in hosts/homeserver.nix status.gapul.net is the one vhost whose upstream lives inside the VM (gatus on localhost), so one request through caddy exercises the generated Caddyfile, the cert wiring, the proxy, and gatus together. A second request to an off-box vhost asserts 502 rather than a connection refusal, proving the vhost matched. Also fixes an ordering bug found while writing it: the vhosts point `tls` at files on disk, so nothing stopped caddy from starting before those files existed. The preliminary self-signed cert covers first boot, but a restart could race a renewal; caddy now waits on acme-finished-gapul.net.target.
compose2nix over /opt/stacks, ten containers landed (anisette, forgejo, homepage + glances, jellyfin, navidrome, ntfy, radicale, rsshub + redis). Paths were remapped by hand: ./x became /var/lib/homelab/<stack>/x, the second LVM volume /mnt/jellyfin-media became /srv, and the docker socket became podman's. Nothing regenerates these — the compose files die with Proxmox, so the nix is the source of truth from here. They stay containers rather than moving onto native NixOS modules. Several have one, but switching means relocating data for no gain the definition in git does not already provide. Also in here, because the single-cut plan changed what they mean: - The `sites` table pointed every upstream at CT101, the HAOS VM, or the pve host, which was the staged-migration assumption. With Proxmox replaced in one go those all become 127.0.0.1; only AdGuard primary (the Pi) and the Mac mini stay remote. pve.gapul.net and files.gapul.net are dropped outright. - A /srv dataset with auto-snapshot off, for what used to be the 200GB volume. Media, the attic cache and archive dumps are huge and re-fetchable; keeping them out of the snapshot schedule is the point of splitting the dataset. Nine stacks are deliberately left on the old host for now. archivebox and samba carry a plaintext password in their compose files (samba's is in `command:`, so it is also visible in ps) and cannot be committed to a public repo. attic, dawarich, matrix, miniflux, obsidian-couchdb, paperless and vaultwarden interpolate secrets into `environment`, where compose2nix would inline an empty string or an upstream default like "password" and silently shadow the real value from environmentFiles. Both groups need per-service handling.
…containers The blanket rewrite of /var/run/docker.sock to podman's socket changed both sides of the mount. podman's socket does speak the Docker API, but homepage's config/docker.yaml and glances both look for it at /var/run/docker.sock inside the container, so the widgets would have silently shown nothing. Only the host side needed swapping.
… of the repo
Brings the declared set to 30 containers, which is the old host's 37 minus the
seven that are dropped or replaced (dockge, wud, backrest, uptime-kuma,
adguardhome-sync, and the two that become native modules).
The interesting part is what did not get converted verbatim. Where a compose
file interpolated a value from its .env, that entry is removed from
`environment` entirely and environmentFiles supplies it at runtime. This is not
tidiness: podman applies -e after --env-file, so an entry left behind wins over
the env file. compose2nix resolves ${VAR} eagerly, so a naive conversion writes
an empty string, or an upstream default like "password" or "CHANGE_ME", and the
service then starts with the wrong credential instead of failing. The first
attempt here did exactly that and was caught by grepping the output.
Two more traps found on the way:
- compose2nix only reads a .env for interpolation when -env_files is passed;
without it every ${VAR} silently became "" or its :- default.
- passing -env_files to a stack that already declares env_file: (attic) inlines
every key from that file and clobbers the literals the compose file sets, so
attic's postgres would have come up as user "postgres" instead of "attic".
samba does not become a container at all. dperson/samba takes credentials as
command-line arguments, which put a live password in a compose file and in the
output of ps. services.samba keeps the share declarative and leaves the password
in samba's own tdb, set once with smbpasswd. Its reach changes: the container
published 139/445 on the LAN, this host only trusts tailscale0.
README.md documents which keys each /var/lib/secrets/<stack>.env must define.
Several are renamed relative to the old .env (paperless's PAPERLESS_SECRET is
the container's PAPERLESS_SECRET_KEY; miniflux needs the whole DATABASE_URL
rather than the password alone), which is exactly the kind of thing that fails
quietly at 2am on install day.
Two of the passwords carried over were sitting in plaintext in compose files on
the old host and are worth rotating rather than moving.
…ively The three whose configuration lived somewhere other than a file, which is the whole reason for the migration. AdGuard's upstreams, filters and DNSSEC come from nix now, which also retires adguardhome-sync: that container existed only to copy settings from the primary on the Pi to the secondary here, and two instances generated from one definition have nothing to copy. mutableSettings stays true so the admin account, a bcrypt hash that does not belong in a public repo, survives in AdGuard's own file. Port 53 is claimed explicitly (services.resolved off) and opened on the LAN, unlike everything else here, because clients point at it directly. Syncthing's devices and folders are declared, so pairing a machine becomes a commit. Device IDs are public keys and safe to commit; the node's own identity under /var/lib/syncthing is not regenerable and has to be migrated, or the Mac sees a new device and rescans everything. restic replaces backrest. The repository and retention come from lib/restic-common.nix, which already called itself the single definition point — three hosts thinning one shared repository on different schedules is how snapshots disappear from under each other. It only had the policy as a shell snippet, so this adds retentionArgs as data and rebuilds that snippet from it; the generated string is byte-identical, so the Mac's and the laptop's scripts do not change. The VM test now waits on all three.
statix rejects an empty { ... } pattern. It is excluded from the pre-commit
hooks and only runs in ci-lint.sh, so this slipped past a local commit. The
config's drvPath is unchanged, so the fix is purely syntactic.
With thirty container units in the config, multi-user.target never settles in the test VM: every one of them tries to pull an image and there is no network in the sandbox, so the run hit the 900s timeout. The units themselves are still worth asserting on, so the test checks that one was generated and then waits only on the services that need no registry.
Home Assistant stays the official image so the config directory, HACS and the update cadence carry over untouched. Its five add-ons split: the Matter server becomes its own container, Mosquitto, ESPHome and Node-RED become native services. Supervisor does not exist outside HAOS, so the Apps store, one-click updates and the built-in backup are gone; the first two are this file, the third is backup.nix. The Matter server is deliberately not services.matter-server. nixpkgs has python-matter-server at 8.1.2 and the running add-on is 9.0.3, and the server refuses storage written by a newer schema — using the module would mean factory resetting and re-commissioning every Matter device. Both it and Home Assistant run with host networking, because discovery is mDNS and Matter mandates IPv6 even for Wi-Fi devices. A host with IPv6 off starts the server happily and leaves every device unavailable. esphome.gapul.net and nodered.gapul.net are new vhosts. Those UIs used to be reached through Home Assistant's add-on ingress, which goes away with Supervisor; both need an A record adding in Cloudflare. Also here: podman now pulls docker.io through mirror.gcr.io by default. That was a manual per-image workaround on the old host, and install day pulls about thirty images at once into an empty store, which would hit the anonymous rate limit immediately. README.md gains the Home Assistant migration notes. The one worth reading twice is trusted_proxies: it names the old Caddy container's address, and with Caddy on the same host now it has to become 127.0.0.1 or every proxied request is rejected with a 400 whose only trace is in Home Assistant's log.
The swap has no per-service rollback, so the order of operations is the safety mechanism. Written down because the parts that are hard to undo are clustered: the Matter fabric (8MB, losing it means factory resetting every device), Syncthing's cert (losing it means the Mac rescans everything), and the named podman volumes, which do not move by copying a directory. The point worth reading before anything else: the vzdump backups live on the same NVMe that gets formatted, so they are not backups until they are off the box. Also flags the two secrets to rotate on the way, the DNS records the two new vhosts need, the restic token that expires after about a week of disuse, and the ownership differences — the old host ran everything as root in docker, the native services each have their own user. HOMELAB.md gets a banner so it stops reading as current.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Proxmox の単一ノード(pve)を、ハイパーバイザ無しの NixOS 1台にそのまま置き換えるための土台。ネットワーク層(リバースプロキシ・subnet router・死活監視)とディスクレイアウトが入ります。
段階移行(pve の上に NixOS VM を立てて1サービスずつ移す)ではなく直接置き換えなので、サービスごとの切り戻しがありません。その代償として、pve を消す前に全部の宣言が揃って検証済みであることが前提になります。検証は CI(x86_64-linux で実機と同一の設定をビルド、のちに NixOS VM テストを追加)が担います。
何が入るか
nix/hosts/homeserver.nix— caddy(1つのsitesテーブルから 21 vhost を生成)、tailscale subnet router、gatus、podman、sshd、ZFS 運用設定、nix 設定nix/hosts/homeserver-disk.nix— disko でディスク全体。1GB の ESP + 単一ディスク zpoolnix/hosts/homeserver-hardware.nix— 手書きでコミット。nixos-generate-configの出力を機械に置く運用をしないnix/flake.nix—homeserverをガード無しで公開 +diskoConfigurations.homeserver設計判断
ZFS は vzdump の代替として入れています。 失う機能の中で唯一明確な劣化がゲスト単位のクラッシュ整合スナップショットなので、そこを埋めるのが目的です。単一ディスクでもスナップショットとロールバックは手に入り、後から2本目を
zpool attachしてミラーに変換できるので、ディスクの到着を待つ必要がありません。データセットは root / nix / var-lib / home の4つ。/nixは flake から再現できるのでスナップショット対象外、/var/libを分けているのは「コンテナの更新失敗を OS ごと巻き戻さずに戻す」ためです。細かい per-service データセットは後からzfs createで足せるので今は切りません。ARC を 2GB に制限しています。 既定は物理メモリの半分なので、この移行で回収する約4.7GB を ZFS が食い直してしまいます。15GB の箱で20前後のコンテナを動かす前提の初期値で、読み込みが詰まるようなら上げる調整箱です。
hardware-configuration.nix を機械に置く運用をやめました。 この箱は専用機でデュアルブートを守る必要がないため、disko にディスク全体を持たせられます。すると fileSystems も disko が生成するので、機械固有の情報は initrd モジュールと CPU ベンダーだけになり、手書きしてコミットできます。結果として homeserver には未コミットの部分が無くなり、CI が実際にインストールされる物をそのままビルドします(nixos-laptop の pathExists ガードや CI 用スタブが不要になったので削除)。
Caddy の証明書。 cloudflare DNS プラグイン入り caddy の自前ビルドをやめ、
security.acmeのワイルドカード(*.gapul.net)を1枚取って Caddy はそれを読むだけにしました。DNS-01 の発行が 25 回から 1 回になります。reloadServicesは必須で、これが無いと更新後も caddy が古い証明書を掴んで約90日後に全 vhost が静かに死にます。lego はCF_DNS_API_TOKEN、旧プラグインはCF_API_TOKENで変数名が違う点に注意。sitesテーブルが vhost と gatus の両方を生成します。 今まで Caddyfile と uptime-kuma の GUI とこのリポジトリ内の Caddyfile コピーの3箇所に同じ情報があり、実際にリポジトリのコピーは稼働版と乖離していました(スニペット整理版が未デプロイ、macmini 系4件が欠落)。gatus は upstream を直接叩くので、DNS がこのホストを指す前でも意味のある監視になります。移行に伴って落としたもの
dockge(oci-containers に吸収)、wud(Renovate へ)、backrest(宣言的な restic へ)、pdf(stirling-pdf は既に停止済み)。uptime-kuma は Raspberry Pi で稼働中なので、監視項目を gatus に移植したうえで停止します。
まだ入っていないもの
CT101 の 37 コンテナ(compose2nix)、Home Assistant と matter-server、AdGuard と Syncthing の宣言、restic、CI の NixOS VM テスト、CT106 の Mullvad exit node、VM105 の L2TP/IPsec 中継。理由はホストファイル内のコメントにあります。
検証
nix eval .#nixosConfigurations.homeserver.config.system.build.toplevel.drvPathが通ることを確認。disko が生成する fileSystems が意図通り(/=rpool/root,/nix=rpool/nix,/var/lib=rpool/var-lib,/home=rpool/home,/boot=ESP)、boot.supportedFilesystemsに zfs が入り、スナップショットのタイマー5本と scrub が生成されることも確認済み。Caddyfile の生成文字列も3パターン目視しました。