https://github.com/mcdonc/.nixconfig/blob/master/videos/nix-serve/script.rst
Btw, why is remote caching with substituter so iffy with NixOS? I encountered the same issues demonstrated in the video. It’s not obvious why an offline substituter should block a local rebuild unless there is some nix option denoting as optional or required. Or why some matching entries in the remote nix-serve store are ignored (or untrusted?)
From the video notes, this alternative to nix-serve looks promising:
I think the current best solution to this is to add your substituter to
trusted-substituters
and then add it to thenixConfig.extra-substituters
attribute of your system config flake (or do someNIX_CONFIG
hackery in your.envrc
if you’re not using flakes). That way if that substituter dies, you can easily disable it (by removing it from your system flake) without rebuilding the configuration.Oh, and BTW, Nix has a
--fallback
option that will rebuild locally if a substituter is not available, but that will make things painfully slow, better to use the method I previously mentioned.Do you know of a complete example or live config I could read through as a reference for that first method recommended?
I’d also be interested in complete examples for a working pair of remote builder and local client (both NixOS multi user), as all the documentation I’ve come across thus far are either:
-
intent on modifying root ssh configs in spite of security practices
-
botch use of substituters on the remote builder
- causing needles recompilation, ignoring upstream cache.nix.org
-
is awkward or incompatible with non-interactive sudo sessions
-
works for nixos build but not to switch
-
https://nix.dev/manual/nix/2.30/advanced-topics/distributed-builds
-
https://discourse.nixos.org/t/nixos-rebuild-use-remote-sudo-prompts-me-for-a-password-3-times/56003
-
https://discourse.nixos.org/t/remote-nixos-rebuild-works-with-build-but-not-with-switch/34741
I don’t have an exact example to hand because I’m not using a custom substituter for now. I have a remnant of that in my config still, here: https://github.com/balsoft/nixos-config/blob/master/flake.nix#L5 . But it should be relatively straightforward: add
nix.settings.trusted-substituters = [ "http://your-substituter/" ];
to your (client device) NixOS config (e.g. inconfiguration.nix
); addnixConfig.extra-substituters = [ "http://your-substituter/" ];
to your configflake.nix
; answeryes
when prompted bynixos-rebuild
, and you should be good.As for remote builders, I don’t really dig them myself. They require fully trusting all users who wish to build on them and are finicky to set up. Instead I just
ssh
into the build machine, build whatever I need there, andnix copy
it back to my laptop. That said,intent on modifying root ssh configs in spite of security practices
You can set up your nix-daemon to run as its own user nowadays, mitigating all issues related to root entirely.
botch use of substituters on the remote builder
Never had this issue so don’t really know how to help
is awkward or incompatible with non-interactive sudo sessions
This one is pretty much unfixable due to how remote building works
works for nixos build but not to switch
You should probably use
nixos-rebuild switch --use-remote-sudo
and run it as your user rather than root.
-