Verified Commit d2522c25 authored by Matthias Adamczyk's avatar Matthias Adamczyk
Browse files


parent a65cf936
## Nix Infrastructure Deployment
> Nix is a purely functional package manager. This means that it treats packages like values in purely functional programming languages such as Haskell — they are built by functions that don’t have side-effects, and they never change after they have been built. Nix stores packages in the Nix store, usually the directory /nix/store, where each package has its own unique subdirectory such as
> `/nix/store/b6gvzjyb2pg0kjfwrjmg1vfhh54ad73z-firefox-33.1/`
> where b6gvzjyb2pg0… is a unique identifier for the package that captures all its dependencies (it’s a cryptographic hash of the package’s build dependency graph). This enables many powerful features.
## Why Nix/NixOS?
- Declarative: Almost all of the system and services configuration can be set through options. No more manually editing `/etc` files!
- Reproducible: Building the configuration leads to the exact same result no matter from where and how often you deploy the configuration.
A configuration can be built from literally anywhere where nix is installed.
- Rollbacks: When something does not work as intended, a rollback to the previous generation
of the configuration can be as simple as a `nixos-rebuild switch --rollback`.
- Steep learning curve: Learning Nix/NixOS can be hard at first, yet the pros definitely outpace the cons!
For more info on why one should consider using Nix/NixOS, I recommend reading [the Nix Pills](
## The layout
The nixfiles repository is structured as follows
├── common/ # Common configuration included on all hosts
│ ├──
│ └── ...
├── hosts/ # Per host configuration
│ ├──
│ ├── e151/ # Honeybook 01
│ │ └── ...
│ ├── e152/ # Honeybook 02
│ │ └── ...
│ └── ...
├── modules/ # Self written NixOS modules
│ └── ...
├── lib/ # Utility scripts that aid the management of this repository
│ ├──
│ └── ...
└── secrets/ # GPG Encrypted secrets
└── ...
Check the in each subdirectory for more information.
## How to...
### ...deploy a configuration change
nix build -f . deploy.HOSTNAME && ./result switch
This will generate and run a deployment script, which does the following:
- Fetch the nixpkgs checkout that was used in the last successful deployment on the target host
- Execute `nixos-rebuild` with the required parameters and any additional arguments passed to the deployment script
- This will build the new system configuration
- Copy it to the target host
- (Depending on the mode) Run the activation script, which will restart all services that have changed
- (Depending on the mode) Generate a new boot entry for the new system configuration and set it as boot default
- You can read more about the possible modes and parameters in the [nixos-rebuild manual page](
- Copy secrets from the local password store to `/var/src/secrets/` on the target
> If you need to set extra ssh parameters (eg. path to ssh-keyfile) you can specifiy them by exporting the `NIX_SSHOPTS` environment-variable with the needed ssh parameters.
> This is useful (and necessary) when you don’t have your ssh-key on a YubiKey.
> Example: `export NIX_SSHOPTS="-i ~/.ssh/id_rsa4096"`
### ...update all hosts
niv update nixpkgs
nix build -f . deploy.all && ./result switch
This will generate a script which calls all deployment script as described in the section [...deploy a configuration change to production](#deploy-a-configuration-change-to-production).
### ...unlock a virtual machine with full disk encryption
**NOTE: These instructions are a work in progress**
nix-build -A unlock.HOSTNAME && ./result
This will generate and run an unlock script, which retrieves the LUKS disk encryption passphrase from the password-store and writes it to a specific location on the target, where it will be picked up by the initramfs scripts.
The SSH server will run on a different port, because it does not have the same hostkey. The unlock script takes care of the correct parameters for SSH.
### ...set up a staging environment
**NOTE: These instructions are a work in progress**
nix build -f . vm.HOSTNAME
QEMU_NET_OPTS="hostfwd=tcp::2222-:22 ./result/bin/run-*-vm
This will set up a QEMU virtual machine with the specified configuration.
To test new changes, stop the virtual machines and re-run the command above.
Note that some services might not fully work, because DNS records are missing and the machine will not be reachable over the internet.
You can access the SSH server of the virtual machine on port 2222 of the host and access other services using SSH TCP forwarding.
### ...install a new host with NixOS
Either with the NixOS ISO or with a kexec tarball:
#### NixOS ISO
nix build -f . isoImage
This generates a NixOS install ISO with a preconfigured SSH server.
Alternatively, use the [official ISO](
#### kexec tarball
nix build -f . isoImage
Afterwards, extract the resulting .tar.xz to `/` on the target host and execute `./kexec_nixos`.
### ...add a new server to the configuration
Create a new directory with the hostname under `hosts/` and copy the configuration into it.
## Nix caveats and how we solve them
* Nix can not manage state
* Create backups of stateful data (databases, etc.) to ensure rollbacks are always possible
* Nix can not manage secrets, because all files in the nix store are world-readable
* Deploy secrets to /var/src/secrets/ from local password-store
* Services read the secrets they need from there at runtime
* Services that don't support reading the secret from a file use a ExecStartPre script to template secrets into the configuration file
## Things to note when working on NixOS systems
- Avoid stateful configuration
- Instead of creating users with `useradd`, add them to the NixOS system configuration (in [](common/users.nix)
- Don't touch files in /etc/! They are automatically generated. If you want to change an option, look for the corresponding NixOS module option on [](
- If you want use an application that is not available on the system:
- Use the package search on []( to find the application's **Attribute name**
- Use nix-shell, for example: `nix-shell -p tcpdump --run "tcpdump -i enp4s0"`
- If you use an application a lot, consider adding it to the [list here](common/default.nix)
- Avoid nix-env, because it will install applications into your user profile and these won't be updated with the system configuration.
- Be aware of the following resources
- [NixOS Manual]( (System distribution)
- [Nix Pills]( (Practical walkthrough to learn the basics of the Nix package manager)
- Ask questions
- at @matti
- Don't be afraid to take a look at the source code ^^
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment