lelgenio

Deploying NixOS with flakes on Digital Ocean

16 de outubro de 2023

So, yesterday I spent my afternoon playing around with running a NixOS system on Digital Ocean. Since everyone agrees that lack of documentation is the #1 problem for NixOS, I feel it's appropriate to share my little adventure.

What are we doing exactly?

We will be setting up a single instance of NixOS, so no orchestration, this is not enough for any production service. It's enough to run this blog though :).
We'll be using Digital Ocean, because it's what I'm used to working with, but there are many other options, Instructions should be similar enough if you wish to use something different.
We'll be using flakes to easily pin the version of all packages, and because flake outputs have a common and organized schema for their outputs.
The idea here is that you should be able to edit you server's configuration locally and deploy it to your server over ssh; You'll never upload any nix file to your server.
I'll be introducing some less self-evident features of nix/nixos.

Basics of Flakes

Flakes are a feature of Nix which provides a common way of organizing you software repository as a function that takes other repositories as input, and produces some output; That output may be packages, NixOS modules, arbitrary Nix code, anything.

Flakes only provide a different entry-point to nix code, when migrating a codebase to flakes you will only likely add flake.nix and make very few changes to other files.

Enabling flakes

For historical reasons Flakes are marked as “experimental”.

If you are not using NixOS, you can enable flakes by editing /etc/nix/nix.conf and adding this line:

experimental-features = nix-command flakes

If you a are using NixOS, add his line to your configuration:

nix.extraOptions = ''
  experimental-features = nix-command flakes
'';

If you only wish to try Flakes temporarily, you can override the nix settings on the command line:

alias nix="nix --extra-experimental-features 'nix-command flakes'"
nix run github:NixOS/nixpkgs/release-23.05#hello

This command should work even if you don't have flakes enabled.

nixos-rebuild should be able to use flakes with no configuration.

A basic Flake

Here is a basic flake, I added some comments to quickly go over some important details.

# ./flake.nix
# A Flake file MUST be an Attribute set
{
  # inputs MUST be an Attribute set that lists the .... You know
  inputs = {
    # Here we declare we need nixpkgs, notice we only say what branch
    # not the exact commit hash, flakes will take care of that 
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-23.05";
  };
  # outputs MUST be a function that returns an Attribute set.
  # Everything else is just convention, convention that even some
  # Nix commands will assume are true.
  outputs = inputs: {
    # `nix run` and `nix build` will look for this attribute by default
    # `nix run` or `nix run .#default` should print "Hello world"
    packages.x86_64-linux.default = 
      inputs.nixpkgs.packages.x86_64-linux.hello;

    # You can switch to the system described in ./configuration with
    # `nixos-rebuild switch --flake .#hal9000`
    nixosConfigurations.hal9000 = inputs.nixpkgs.lib {
      system = "x86_64-linux";
      modules = [ ./configuration.nix ];
    };
  };
}

For Flakes to “see” a file it must be committed/added in git! If the file is not added, you will get cryptic “file not found” errors.

Whenever you use flakes, a file called flake.lock will be generated, you should commit it, it keeps track of the exact hash of the inputs and keeps your builds reproducible.

Now you can build any piece of your output attribute set:

$ nix build ".#packages.x86_64-linux.default"
$ ls -l result/bin/
.r-xr-xr-x 56k root  1 jan  1970 hello

$ nix run
Hello, world!

# nixos-rebuild internally calls this command
$ nix build ".#nixosConfigurations.hal9000.config.system.build.toplevel"
$ ls result/
activate               firmware                nixos-version
append-initrd-secrets  init                    prepare-root
bin                    init-interface-version  specialisation
boot.json              initrd                  sw
dry-activate           kernel                  system
etc                    kernel-modules          systemd
extra-dependencies     kernel-params

Introducing Digital Ocean Droplets

Droplets are Digital Ocean's Virtual Private Server (VPS), they can be rented for as low as 4 USD/month. There's really not much to say about it, it's a server, you get a full dedicated IP address, all ports are open, you have root access and can do as you wish.

Unfortunately they do not have a one-click solution for NixOS, fortunately they do allow you to upload a custom OS image, we'll go with that.

Writing a basic NixOS configuration for a VPS

This initial configuration is only supposed to get our server running and allow us to connect to it via SSH. If you don't have ssh keys, now's the time to generate them!

Warning
Be very careful of what you put in your nix configuration!
Assume anything here will be written as plain text on your server!

Here's the configuration I recommend you use, fill in your ssh public key, and optionally your hashed password:

# ./configuration.nix
{ config, pkgs, lib, ... }: {
  services.openssh = {
    enable = true;
    settings.PasswordAuthentication = false;
  };
  users.users.root = {
    openssh.authorizedKeys.keys = [
      "the contents of ~/.ssh/id_rsa.pub go here"
    ];
    # Altough optional, setting a root password allows you to
    # open a terminal interface in DO's website.
    hashedPassword = 
      "generate a hashed password with the mkpasswd command";
  };

  # You should always have some swap space,
  # This is even more important on VPSs
  # The swapfile will be created automatically.
  swapDevices = [{
    device = "/swap/swapfile";
    size = 1024 * 2; # 2 GB
  }];

  system.stateVersion = "23.05"; # Never change this
}

Customizing NixOS module inputs

Before we keep going, I want to share something important for keeping your modules organized. You might be familiar with how you can pass values to be used in nixos modules:

{ config, pkgs, lib, ... }: {
# ↑ Like this
}

These values are set by default by nixpkgs.lib, but what if you need something else... Like your Flakes' inputs? Good news, using specialArgs, you can!

# ./flake.nix
{
  # ....
  outputs = inputs: {
    nixosConfigurations.hal9000 = inputs.nixpkgs.lib {
      specialArgs = {
        # remember "inherit x;" is the same as "x = x;"
        inherit inputs; 
      };
      system = "x86_64-linux";
      modules = [ ./configuration.nix ];
    };
  };
}

Now to use it, as an example we'll tell nixos to install the nixpkgs repository under /etc/nixpkgs:

{ config, pkgs, lib, inputs, ... }: {
  environment.etc.nixpkgs.source = inputs.nixpkgs;
}

Creating a NixOS image for Digital Ocean

To get started, we'll need to create a DO Image. To do that, import the digital-ocean-image, you should also compress your image as much as possible, Digital Ocean takes a bit to process your image, so this should reduce your waiting.

You should keep the module imported even after you have generated the image, so you can update your server later.

# ./configuration.nix
{ config, pkgs, inputs, ... }: {
  imports = [
    "${inputs.nixpkgs}/nixos/modules/virtualisation/digital-ocean-image.nix"
  ];

  # Use more aggressive compression then the default.
  virtualisation.digitalOceanImage.compressionMethod = "bzip2";
  
  # ...
}

Now for let's build that image, this may take a few minutes since it temporarily spawns a virtual machine:

$ nix build .#nixosConfigurations.hal9000.config.system.build.digitalOceanImage
$ ls result/
nixos.qcow2.bz2

Uploading that image and starting our server

At least on Digital Ocean, to start a droplet with a custom image, you must first upload that image on the same region as you wish to run the droplet.

You can upload your image at https://cloud.digitalocean.com/images/custom_images. That process can take quite some time to finish (10+ minutes).

After that's done, Just click on “more” > “Start Droplet”. Fill in the form as you see fit. For reasons beyond my understanding, you must use SSH key authentication on this form.

After that's done, you'll get an IP address, since you added your public key to the configuration, try connecting to it:

$ ssh root@123.123.123.123 'echo Hi from $(hostname)!'
Hi from hal9000

How to update that server, after it's running

Now that we have a running server, how will we update it? We need a tool that can:

Connect to our server via ssh
Intelligently copy nix paths
run the activation script

What if I told you, that tool is already installed on your system?

That tool is nixos-rebuild! You just need to use the --target-host flag:

nixos-rebuild switch --flake .#hal9000 --target-host root@123.123.123.123

Closing thoughts

If you got here, I recommend you get a domain name and point it at your server, this allows you to easily setup ACME certificate generation.

Phew! Done right? Nope! Now you need to actually use your server for something. The vast collection of modules in NixOS should make this the most pleasing part of the setup. It's up to you now!

Have fun with your new server!

services.nginx = {
  enable = true;
  virtualHosts."hal9000.example" = {
    enableACME = true;
    forceSSL = true;
    root = pkgs.runCommand "www-dir" { } ''
      mkdir -p $out
      cat > $out/index.html <<EOF
        <!DOCTYPE html>
        <html lang="en">
        <body>
          <h1>
              I'm sorry Dave, I'm afraid this
              pop culture reference is overused.
          <h1>
        </body>
        </html>
      EOF
    '';
  };
};