@khuedoan would it be better to use Talos instead of nixos and k3s? It makes it so you don't need to manage the underlaying OS and it's also declarative with a focus on Kubernetes and there's also PXE support. Curious on your thoughts around this, as I understand Talos can be limiting in some regards, are you worried about hardware incompatibilities, drivers or something else?

Also you don't need kube-vip with Talos as it's natively supported.

Unrelated to the above, Metallb can be replaced with Cilium (it works in L2 and/or L3) for svc LBs, it could also be used for the kube-vip once the cluster is bootstrapped.

@onedr0p hey ya, thanks for taking a look, that’s a great question! I should
probably add this to the docs as well, but here's what I currently have in
mind.

There are many things that I like in Talos that are missing in NixOS
(admittedly I haven't tried Talos again since #2 with Sidero Metal & PXE Talos
install, so I might have missed some new improvements):

• Talos is very minimal (80MB!!!) thus less attack surface and remarkably fast
  install (NixOS + k3s can be made into a small foot print with relatively fast
  install, but not as tiny and fast)
• An actual controlplane to coordinate day 2 operations, not a bunch of shell
  scripts or Ansible playbooks to do things like "upgrade master nodes first,
  drain before rebooting etc.". And holy shit controlling bare metal machines
  with Sidero Cluster API is nice once everything is set up
• They also have a separate PXE booter CLI now that can be used without
  Sidero Metal (which I haven't personally tried but it showed up on my feed
  via following you 😆)

But using NixOS I'll have to sacrifies those things (and may have to reinvent
some of them) for other advantages:

1. The first one is non-technical: with all due respect to the amazing and hard
   work of the Sidero Labs team, I have more peace of mind that NixOS will be
   around for a long time and continue to be open-source, given its 19-year
   history and the community is only getting bigger. I'm not saying Sidero Labs
   will definitely relicense or deprecate Talos, but there's a risk of that as
   demonstrated by the deprecation of the open-source Sidero Metal (MPL2) in favor
   of the source-available Omni (BSL, which unlike Sidero Metal, Omni runs on a
   separate machine). I understand that they need to balance open source with
   business revenue, which is really, really hard, but that's also why I worry
   about having a core part of my software stack depend on a single VC-funded
   company.

2. With that being said there are other technical things that I think NixOS is
   more suitable for my use case. The long term goal for this project is a "mom we
   have Heroku at home" homelab, not nessesarily a k8s homelab. I may switch from
   k8s to something else if there's a better alternative, but the OS layer should
   be very stable and rarely changes, which means I’ll lean forward a general
   purpose distro instead of ones that are built specifically for k8s - but of
   course k8s is a very high bar that I don't think there's a better alternative
   anytime soon ¹. Due to this reason some of the next points
   might be biased towards general purpose distros as well

3. NixOS unlocks some vertical integrations for Nix-based CI/CD:

   • Use the node's Nix store to speed up my CI/CD running inside Nix shell,
     which manages per-project dependencies (e.g. no need for setup-python
     like GitHub Actions). Maybe add hook to upload built artifact to a global
     Nix cache so that I never have to build something twice again (e.g. if a
     package/container was built on the PR branch, it doesn't need to be
     rebuilt after merging, and the cache works at the individual package
     level and across repos)
   • nix-snapshotter for efficient containers and the ability for transparent source-based
     deployment
   • Higher level Nix abstractions such as devenv or Nixpacks might be able to enjoy
     the same optimizations

4. NixOS can do most (all?) Talos can do regarding reproducibility, ephemerality,
   declarativity, atomic upgrade and more:

   • Talos is only declarative at runtime config, and Talos OS image builds
     are not (easily) reproducible. NixOS is declarative from source to build
     to runtime config. In Nix binary deployment is “just” a transparent
     optimization for source-based deployment ². If I have things cached in
     the Nix store of the laptop running the installation, I basically get
     air-gapped base cluster install for free, and other apps on top of it if I try
     hard enough to preload all images tgz to the configuration, as opposed to
     Talos where I need a separate private container registry
   • While NixOS is not ephemeral by default, it can additionally be configured
     in an erase your darlings style to match Talos' ephemeral design
   • Talos atomic upgrades work by using A/B partitions, NixOS atomic upgrades
     work by switching symlinks in Nix store which can have as many versions
     as your disk allows, and it can be switched without reboot to test the
     configuration (except the kernel), and if A/B partition is desired it can
     be configured with systemd-repart for OTA-style upgrade ² as well.

5. Talos mostly doesn't have the single system (node) deployment problem to
   solve since it's A/B partition-based at the cost of easily customize the OS.
   Nix/NixOS while still customizable has really good foundation and solved
   the single system deployment problem from first principles many years ago,
   which many other deployment tools (Ansible, Puppet etc.) still cannot match
   (the original paper ³ explains this much better than me - the only tool in the
   same league I can find is GNU Guix which is also based on Nix's core
   principles, but its ecosystem is not as mature compared to NixOS)

6. With NixOS I can easily spin up VMs in QEMU with the same configuration
   without applying it on real hardware ⁴, and optionally write e2e tests in
   Python using their testing framework. Talos can also do this with Talos in
   Docker but running k8s in Docker has some quirks (this was a while ago so those
   quirks might have been fixed already)

7. I use NixOS everywhere else including my laptops, work Macbooks (via
   nix-darwin), gaming PC, backup server, etc., so it's nice to have a single way
   to manage all of my computes (and also install them via PXE boot with
   nixie - WIP)

8. This is a personal thing but when I use NixOS I haven't felt the need to
   distro hop again, unlike with other distros (ok I still feel that a tiny bit
   when seeing Guix, but still)

Phew that kinda long 😆 Also I’ll try some LB combinations again, ideally fewer
tools as much as possible (the current implementation is Cilium L2 with ARP but I had
some problems bootstrapping so I still needed kube-vip but that might be fixed now)