README.md 7.35 KB
Newer Older
Matthias Adamczyk's avatar
Matthias Adamczyk committed
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
## Nix Infrastructure Deployment

> Nix is a purely functional package manager. This means that it treats packages like values in purely functional programming languages such as Haskell — they are built by functions that don’t have side-effects, and they never change after they have been built. Nix stores packages in the Nix store, usually the directory /nix/store, where each package has its own unique subdirectory such as
> `/nix/store/b6gvzjyb2pg0kjfwrjmg1vfhh54ad73z-firefox-33.1/`
> where b6gvzjyb2pg0… is a unique identifier for the package that captures all its dependencies (it’s a cryptographic hash of the package’s build dependency graph). This enables many powerful features.

## Why Nix/NixOS?

Pros:
- Declarative: Almost all of the system and services configuration can be set through options. No more manually editing `/etc` files!
- Reproducible: Building the configuration leads to the exact same result no matter from where and how often you deploy the configuration.
  A configuration can be built from literally anywhere where nix is installed.
- Rollbacks: When something does not work as intended, a rollback to the previous generation
  of the configuration can be as simple as a `nixos-rebuild switch --rollback`.

Cons:
- Steep learning curve: Learning Nix/NixOS can be hard at first, yet the pros definitely outpace the cons!

For more info on why one should consider using Nix/NixOS, I recommend reading [the Nix Pills](https://nixos.org/guides/nix-pills/)

## The layout

The nixfiles repository is structured as follows

```
├── common/        # Common configuration included on all hosts
│   ├── README.md
│   └── ...
├── hosts/         # Per host configuration
│   ├── README.md
│   ├── e151/      # Honeybook 01
│   │   └── ...
│   ├── e152/      # Honeybook 02
│   │   └── ...
│   └── ...
├── modules/       # Self written NixOS modules
│   └── ...
├── lib/           # Utility scripts that aid the management of this repository
│   ├── README.md
│   └── ...
└── secrets/       # GPG Encrypted secrets
    ├── README.md
    └── ...
```

Check the README.md in each subdirectory for more information.

## How to...
### ...deploy a configuration change

```
nix build -f . deploy.HOSTNAME && ./result switch
```

This will generate and run a deployment script, which does the following:
- Fetch the nixpkgs checkout that was used in the last successful deployment on the target host
- Execute `nixos-rebuild` with the required parameters and any additional arguments passed to the deployment script
  - This will build the new system configuration
  - Copy it to the target host
  - (Depending on the mode) Run the activation script, which will restart all services that have changed
  - (Depending on the mode) Generate a new boot entry for the new system configuration and set it as boot default
  - You can read more about the possible modes and parameters in the [nixos-rebuild manual page](https://www.mankier.com/8/nixos-rebuild)
- Copy secrets from the local password store to `/var/src/secrets/` on the target

Matthias Adamczyk's avatar
Matthias Adamczyk committed
65
66
67
68
69
70
71
**But building takes sooo long >:(**
You can offload the build to a remote host, for example krul:

```
$ nix build --builders 'ssh://USERNAME@krul.finf.uni-hannover.de' -f . deploy.HOSTNAME
```

Matthias Adamczyk's avatar
Matthias Adamczyk committed
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
> If you need to set extra ssh parameters (eg. path to ssh-keyfile) you can specifiy them by exporting the `NIX_SSHOPTS` environment-variable with the needed ssh parameters.
> This is useful (and necessary) when you don’t have your ssh-key on a YubiKey.
> Example: `export NIX_SSHOPTS="-i ~/.ssh/id_rsa4096"`

### ...update all hosts

```
niv update nixpkgs
nix build -f . deploy.all && ./result switch
```

This will generate a script which calls all deployment script as described in the section [...deploy a configuration change to production](#deploy-a-configuration-change-to-production).


### ...unlock a virtual machine with full disk encryption

**NOTE: These instructions are a work in progress**

```
nix-build -A unlock.HOSTNAME && ./result
```

This will generate and run an unlock script, which retrieves the LUKS disk encryption passphrase from the password-store and writes it to a specific location on the target, where it will be picked up by the initramfs scripts.
The SSH server will run on a different port, because it does not have the same hostkey. The unlock script takes care of the correct parameters for SSH.


### ...set up a staging environment

**NOTE: These instructions are a work in progress**

```
nix build -f . vm.HOSTNAME
QEMU_NET_OPTS="hostfwd=tcp::2222-:22 ./result/bin/run-*-vm
```

This will set up a QEMU virtual machine with the specified configuration.
To test new changes, stop the virtual machines and re-run the command above.
Note that some services might not fully work, because DNS records are missing and the machine will not be reachable over the internet.
You can access the SSH server of the virtual machine on port 2222 of the host and access other services using SSH TCP forwarding.


### ...install a new host with NixOS

Either with the NixOS ISO or with a kexec tarball:

#### NixOS ISO

```
nix build -f . isoImage
```

This generates a NixOS install ISO with a preconfigured SSH server.
Alternatively, use the [official ISO](https://nixos.org/download.html).

#### kexec tarball

```
nix build -f . isoImage
```

Afterwards, extract the resulting .tar.xz to `/` on the target host and execute `./kexec_nixos`.


### ...add a new server to the configuration

Create a new directory with the hostname under `hosts/` and copy the configuration into it.


## Nix caveats and how we solve them

* Nix can not manage state
  * Create backups of stateful data (databases, etc.) to ensure rollbacks are always possible
* Nix can not manage secrets, because all files in the nix store are world-readable
  * Deploy secrets to /var/src/secrets/ from local password-store
  * Services read the secrets they need from there at runtime
  * Services that don't support reading the secret from a file use a ExecStartPre script to template secrets into the configuration file

## Things to note when working on NixOS systems

- Avoid stateful configuration
  - Instead of creating users with `useradd`, add them to the NixOS system configuration (in [](common/users.nix)
  - Don't touch files in /etc/! They are automatically generated. If you want to change an option, look for the corresponding NixOS module option on [nixos.org/nixos/options.html](https://nixos.org/nixos/options.html)
  - If you want use an application that is not available on the system:
    - Use the package search on [nixos.org/nixos/packages.html](https://nixos.org/nixos/packages.html?channel=nixpkgs-unstable) to find the application's **Attribute name**
    - Use nix-shell, for example: `nix-shell -p tcpdump --run "tcpdump -i enp4s0"`
    - If you use an application a lot, consider adding it to the [list here](common/default.nix)
    - Avoid nix-env, because it will install applications into your user profile and these won't be updated with the system configuration.
- Be aware of the following resources
  - [NixOS Manual](https://nixos.org/nixos/manual.html) (System distribution)
  - [Nix Pills](https://nixos.org/nixos/nix-pills/index.html) (Practical walkthrough to learn the basics of the Nix package manager)
- Ask questions
  - at @matti
- Don't be afraid to take a look at the source code ^^