Using Nix on macOS

I recently started using the Nix Package Manager on macOS and the process has been painful. In this post, I’m going to write down how I’m currently using Nix on macOS with the Zsh shell.

Let me start by acknowledging Ian Henry’s wonderful series of blog posts on Nix. Nix is an extremely complex system that is poorly documented. There are many blog posts by a variety of authors about using Nix, Ian’s series is the most comprehensive (and funny) that I’ve read.¹

In this post, I will not discuss NixOS at all. I have no experience with NixOS (although the idea is neat). Every use of Nix below should be understood to mean using Nix to interact with the nixpkgs set of packages running on macOS.

If you’re familiar with Nix, you may be wondering, why not use Home Manager? The short answer is Nix is already too complex. Adding additional tools just makes it harder to understand how Nix works and how to use it directly.

In this post, I’m going to stick with running standard Nix binaries and not use any additional tools (although there are two shell scripts that run Nix tools that I find useful at the end of the post). Nix has a bunch of experimental features (including the nix binary itself and flakes²). The experimental features change so I won’t discuss them at all either.

Nix overview

Nix bills itself as “a tool that takes a unique approach to package management and system configuration” that can be used “to make reproducible, declarative and reliable systems.”

In my mind, there are four main use cases for Nix.

1. User package management;
2. Consistent developer environments;
3. One-off shells with packages installed; and
4. Building software.

For the first use case, I mean package management in the style of Apt, MacPorts, or HomeBrew. The tool for this is called nix-env and it’s responsible for installing packages. It’s usually used on the command line to install packages incrementally. For example, if you want to install the excellent ripgrep tool, you’d use the following command.

$ nix-env -iA nixpkgs.ripgrep

Like any good package manager, this will install rg along with any dependencies.

Nix also supports a declarative package management system where you put the list of packages you want in a file and then you can (for example) check that file into version control.

Nix’s second use case lets developers declaratively specify a (more or less) complete environment for a project in a shell.nix file. (E.g., a Python project may require other Python packages to be installed. These dependencies can be specified in the shell.nix file. All developers can use the nix-shell command to launch a Bash shell with the appropriate dependencies installed and available.) This use case seems more aspirational than practical since it requires all developers use Nix to get this benefit.

The third use case is Nix’s killer feature in my mind. Rather than install and remove packages as necessary, Nix lets you create a one-off shell that contains a set of packages. For example, if you need a shell with socat installed and don’t want to install it in your normal environment using nix-env, you would run

$ nix-shell -p socat

which would fire up a new Bash shell with socat installed and ready to run. Neat!

The last use case is similar to the second except that rather than just have the dependencies be installed and managed by Nix, the whole project is built by Nix.

Of these, I’m interested in declarative package management, one-off shells, and developer environments, in that order. I’m not interested in building software using Nix.

Consistent, reproducable, declarative package management

My requirements for using Nix as a package management system are simple.

• It must be declarative in that I can write a simple file (I call mine env.nix) which specifies the list of packages to install;
• It must be reproducable meaning that I should be able to check my env.nix into version control, check it out later—on another machine—and get the same set of packages installed; and
• It must be consistent in the sense that if I install a package via nix-env (as I did above with Ripgrep), then that should be exactly the same as if I had specified it in my env.nix file.

One could be forgiven for thinking consistency is a natural consequence of reproducability. In Nix, consistency requires additional work.

Let’s start by making a declarative list of packages. There are at least three ways of using Nix as a declarative package manager: overlays, overrides, and for lack of a better term, a package file.

Overlays and overrides are confusing and don’t match my mental model for how declaring a set of packages to be installed should work. And I think using them is more complex than necessary. But first, we need to talk about how Nix works.

How Nix works

At a high level, you tell Nix to install a package ($ nix-env -iA nixpkgs.fd installs the excellent fd utility) and it figures out everything it needs to install, and then installs it.

In more detail, Nix reads and evaluates the “Nix expression” that says how to build software. Each expression is (usually) contained in a file somewhere that says how to build the software. This is the Nix expression for building version 8.4.0 of fd.

Nix expressions are written in the Nix language which is pure, lazy, and functional (like Haskell) and the main datatype is the “attribute set” which is just a key-value map or dictionary. The Nix expression for building software evaluates to a derivation which is an attribute set containing a key type with the value derivation. A side effect of nix-env evaluating a derivation is to download and install the package described by the derivation! This is kind of shocking in a pure language.

Once a derivation is evaluated, the package is installed under /nix somewhere and symlinks are created in the user’s profile. A profile is just a directory containing bin and share and similar directories. There’s a symlink ~/.nix-profile which points to the active profile.

Nix exposes software to a shell by creating symlinks in the profile’s bin directory. One of the shell’s start up files will contain code to put ~/.nix-profile/bin in the PATH environment variable. (See below for more details.) nix-shell, the command for getting shells with packages installed works similarly except it also adjusts the PATH before invoking Bash.

(Nix supports multiple profiles and rolling back profiles to earlier versions. This happens by updating the ~/.nix-profile symbolic link. I have found no use for using multiple profiles because my environment is specified declaratively by a file in Git, so I can use standard Git commands to roll back as needed.)

Before getting to my approach, there’s one last thing we need to discuss and that’s where does Nix get the Nix expressions that are used to build packages? The answer is complicated and is different for each Nix tool.

First, nix-env uses a “default” expression when installing, updating, and querying for available packages. (Never, ever update packages with nix-env -u. I have no idea what the use case is, but it does the wrong thing. Just don’t do it. Similarly, never use nix-env -i without also passing -A. It doesn’t do what you want otherwise.) The default expression is either a file or a directory named ~/.nix-defexpr.

If it’s a file, then the contents of a file is the default expression which must evaluate to an attribute set (dictionary). If it’s a directory, then the name of each directory inside ~/.nix-defexpr becomes a key in the default expression’s attribute set whose value is the result of evaluating the default.nix inside the directory. It’s complicated and the full details are immaterial. But as an example

~/.nix-defexpr
├── bar
│   └── default.nix
└── foo
    └── default.nix

would create an attribute set { foo = ...; bar = ...; } as the default expression where the values corresponding to foo and bar come from evaluating their default.nix files.

The traditional method of using Nix as a package manager uses Nix channels which are similar to Git branches. Each channel refers to a nixpkgs commit. Nix has a tool, nix-channel, which maintains a list of channels. It downloads a snapshot of the nixpkgs repository at the commit specified by the channel. This contains all of the Nix expressions used to build packages (as well as a bunch of helper Nix code). $ nix-env -iA can then be used to install (or update) a package.

Returning to my original example

$ nix-env -iA nixpkgs.ripgrep

this instructs nix-env to start with the “default” expression (~/.nix-defexpr) which evaluates to an attribute set as described above. Next, nix-env will look up the key nixpkgs in the default expression which returns another attribute set. Then, nix-env will look up the key ripgrep in that attribute set. Since this causes the ripgrep derivation to be evaluated, it will install the package (and dependencies).

Nix channels have a problem: they’re not reproducible. You get whatever revision of nixpkgs happens to be current when you run $ nix-channel --update. As a result, Nix channels are not useful and I won’t discuss them further, but the ~/.nix-defexpr is essential for nix-env. I’ll return to this below.

So if nix-env uses this default Nix expression, it stands to reason that other Nix tools like nix-shell do as well. Alas, no.

The second method Nix tools like nix-shell and nix-build use to look up Nix expressions is via the NIX_PATH environment variable. There are a bunch of valid formats for NIX_PATH, but the simplest is a colon-separated list of directories. NIX_PATH is used in two key places (for my purposes).

The first is when nix-shell is run on a file (shell.nix by default), the contents of the file is the entire Nix expression. Since these files need a way to refer to Nix packages, it’s common to see Nix files that contain a line like this.

import <nixpkgs> {}

Without delving too deeply into the Nix language, the import <nixpkgs> instructs nix-shell to look for a directory named nixpkgs in the NIX_PATH and load nixpkgs/default.nix as a Nix expression. The {} is an empty attribute set that is passed to the function returned by import <nixpkgs>. Why does import <nixpkgs> return a function? It does simply because the expression that was loaded from the nixpkgs/default.nix file is a function. This function will return an attribute set whose keys are the names of packages, more or less.

So we have two separate mechanisms to specify where to find a Nix expression that describes how to build a package.

The last thing we need is a way to specify a particular revision of the nixpkgs repository we want to use. There are multiple ways to do this. For NIX_PATH, we can use a URL like NIX_PATH=nixpkgs=https://github.com/NixOS/nixpkgs/archive/87d9c84817d7be81850c07e8f6a362b1dfc30feb.tar.gz which will map <nixpkgs> to nixpkgs revision 87d9c84817d7be81850c07e8f6a362b1dfc30feb.

For nix-env, there are several ways to do this, including having a single ~/.nix-defexpr file, a ~/.nix-defexpr/nixpkgs.nix file, or a ~/.nix-defexpr/nixpkgs/default.nix file. I’d prefer the first option, but as we’ll see the last option works best. The contents of ~/.nix-defexpr/nixpkgs/default.nix is a single expression that evaluates to a function which, when called, evaluates to the attribute set containing the list of packages.

import (fetchTarball "https://github.com/NixOS/nixpkgs/archive/87d9c84817d7be81850c07e8f6a362b1dfc30feb.tar.gz")

(Note the lack of {} at the end of the line. This is because the contents of the file needs to evaluate to a function which is precisely what we get from the import. If we added the {}, it’d call the function and return an attribute set.)

At the end of the day, I want the exact same packages to be installed regardless of whether I use $ nix-env -iA foo or nix-shell -p foo, or update the list of packages in my env.nix. That means we’ll need NIX_PATH and the default Nix expression to agree on the list of packages.³

Declarative package manager implementation

With all of the preliminaries out of the way, let’s see how to set up a declarative package manager that meets my three requirements.

Let’s make a list of packages that works first and pin it to a specific nixpkgs revision afterward.

As discussed above, nix-env has a default expression it uses for installing packages. This default can be overridden using the -f (or --file) argument. We can also instruct nix-env to also install every package. I have no idea why we’d want to do that in general. However, if we specify a different default expression—one that contains just a list of the packages we want—then it’ll install all of those. So without further ado, here’s my env.nix.

with import <nixpkgs> {}; [
  calc
  coreutils-full
  fd
  gdb
  jq
  nasm
  neovim
  nodejs
  pwgen
  python3
  ripgrep
  ruby_3_1
  shellcheck
  socat
  tmux
  tree
  universal-ctags
  zsh-syntax-highlighting
]

The with expr1; expr2 construct evaluates expr1 to an attribute set S and then it evaluates expr2 with the keys of S bound to their values. We can write an equivalent env.nix file as something like

let
  pkgs = import <nixpkgs> {};
in [
  pkgs.calc
  pkgs.coreutils-full
  ...
  pkgs.zsh-syntax-highlighting
]

To install all of the packages in env.nix, it’s sufficient to run $ nix-env -irf env.nix. The --remove-all (-r) flag will remove anything that was installed (e.g., by nix-env -iA) but isn’t in env.nix.

This satisfies my first requirement: a declarative package manager.

Note that this is essentially the solution Ian Henry arrived at. All credit to him for this.

Notice how $ nix-env -irf env.nix is not using the default Nix expression, ~/.nix-defexpr, but is instead using NIX_PATH by way of import <nixpkgs>. To satisfy the second requirement, reproducability, we need to pin nixpkgs to a particular revision. We can do that by setting NIX_PATH=nixpkgs=https://... as described above; however, there’s a better way.

To make this reproducable, it’s enough to check the env.nix and the shell dotfiles which set NIX_PATH into version control.

The final requirement is consistency: installing a package using any of the standard methods (nix-env -iA, nix-env -irf env.nix, nix-shell -p, etc.) should always give me the same package. If ~/.nix-defexpr and NIX_SHELL are kept in sync, pointing to the same revision of nixpkgs, this will work out.

Unfortunately, updating the Nix expressions for a package by changing the value of the NIX_PATH environment variable has a few downsides. First, changing it in a dotfile like ~/.bash_profile or ~/.zprofile has no effect on the currently running shells meaning it’d be easy to install package in a shell that still has the old NIX_PATH value.

So that suggests that NIX_PATH should remain constant and only ~/.nix-defexpr should be updated. My ~/.zprofile contains the following line (but see fixing shell integration below).

export NIX_PATH="$HOME/.nix-defexpr"

Recall that if NIX_PATH is a colon-separated list of directories, then the Nix code import <foo> is going to look for a directory named foo in each of those directories. Since I’m always using import <nixpkgs> (as is everyone else), we need a ~/.nix-defexpr/nixpkgs directory containing a default.nix file.⁴

import (fetchTarball "https://github.com/NixOS/nixpkgs/archive/87d9c84817d7be81850c07e8f6a362b1dfc30feb.tar.gz")

This is the only file that specifies which revision of nixpkgs to use and it will be used by all of the Nix tools.

Checking this file into version control (along with nix.env and the ~/.zprofile which specifies NIX_PATH) satisfies all of my requirements. Yay!

Fixing shell integration

Installing Nix on macOS is a bit unpleasant. Single-user installs are no longer supported. Multi-user installs have to be used instead.⁵ This goes through a complicated dance to create /nix and some accounts for building software not as your user.

One of the steps of the installer is to modify the system-wide /etc/bashrc and /etc/zshrc files to insert the lines

# Nix
if [ -e '/nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh' ]; then
  . '/nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh'
fi
# End Nix

The sourced nix-daemon.sh starts the Nix daemon running and sets some environment variables, including PATH.

This code doesn’t belong in either of these files for two reasons: 1. They’re system-owned files. macOS will overwrite them when updating so you need equivalent code in your local shell start up files. 2. Setting environment variables doesn’t belong in the shell’s rc files; neither in the global ones in /etc nor in the dotfile ones in $HOME. It belongs in the local “profile” files: ~/.bash_profile and ~/.zprofile.

To elaborate on the second point, we have to look at which files are sourced when shells start and that means we have to talk about types of shells (unfortunately).

Every shell is either a login shell or not. Independently, every shell is either interactive or not. All four combinations are possible, although a noninteractive, login shell is uncommon.

A login shell is created by running $ bash -l or $ zsh --login. Otherwise, it’s not a login shell.

An interactive shell is one that gives you a prompt and you, well, interact with. A noninteractive shell is what you get when you run a shell script.

Usually, your terminal emulator will create a login shell by default when you open a new terminal window. (Apple Terminal and iTerm2 both do.) And that’s generally the only time you create a login shell.

When a login shell is created, it’s usually going to create its environment from scratch rather than using the environment it inherited from its parent. In contrast, since a nonlogin shell is usually created by running bash or zsh from inside another shell, it’s going to inherit its environment. This suggests that environment variables (like PATH) should only ever be set when login shells start. Since they’re going to be the same from shell to shell, it doesn’t make sense to set them again when starting a nonlogin shell.

Interactive shells, in contrast, need to be configured every time they are started. This includes setting up shell completions and command line prompts (e.g., PS1). We have to do this every time because these settings are not inherited by child processes.

To support this, shells support a dizzying array of files that get read on start up. In the interest of length, I’m going to focus on Zsh since that’s the new default macOS shell.

Zsh has 8 different startup files it sources, depending on whether the shell is a login shell or not and whether the shell is interactive or not. Here they are, in the order they are sourced.

1. /etc/zshenv
2. ~/.zshenv
3. /etc/zprofile for a login shell
4. ~/.zprofile for a login shell
5. /etc/zshrc for an interactive shell
6. ~/.zshrc for an interactive shell
7. /etc/zlogin for a login shell
8. ~/.zlogin for a login shell

If a file doesn’t exist, it is skipped. On macOS, /etc/zshenv and /etc/zlogin don’t exist so we can ignore those. I don’t use ~/.zlogin, so we can ignore that one too.

This leaves ~/.zshenv, /etc/zprofile, ~/.zprofile, /etc/zshrc, and ~/.zshrc to deal with.

“But Steve,” I hear you object, “I don’t care about any of that. I just want to put all my shell configuration in a single file and be done with it.” That’d be nice, but macOS doesn’t care what you want.

Here’s the problem. On macOS, /etc/zprofile runs /usr/libexec/path_helper and sets PATH from its output. path_helper constructs a PATH starting with the paths from /etc/paths and /etc/paths.d and appending the other directories already on PATH. Let’s look at an example.

With an empty PATH, path_helper outputs a string (which the shell will eval) to set a default PATH.

$ PATH= /usr/libexec/path_helper
PATH="/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/Apple/usr/bin"; export PATH;

If PATH contains any other directories, such as ~/.nix-profile/bin, these will be moved to the end.

$ PATH=$HOME/.nix-profile/bin:/usr/bin /usr/libexec/path_helper
PATH="/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/Apple/usr/bin:/Users/steve/.nix-profile/bin"; export PATH;

Therefore, if a new login shell is launched, your PATH will be rearranged!

To deal with this, we’ll need to treat login shell specially and reset PATH prior to /etc/zprofile being sourced.

This is a relatively minor issue since you don’t often run a new login shell. A bigger problem comes from interactive shells.

Nix’s modifications to /etc/zshrc will source nix-daemon.sh which unconditionally prepends Nix profile directories to PATH. So invoking zsh (i.e., an interactive, nonlogin shell) causes the directories to be added to the PATH again.

To summarize, if we launch a new login shell, our PATH is rearranged. If we launch a new interactive shell, duplicate entries are added to the path. If we launch a new interactive, login shell, we get a rearranged PATH with duplicate entries. Not great.

So here’s the solution. First, after installing Nix, undo the modifications Nix made to the files in /etc. The install script creates a copy of the file before modification (e.g., /etc/zprofile.backup-before-nix). Just rename that to /etc/zprofile to restore the original.

Second, create a ~/.zshenv file with the following contents.

[[ -o login ]] && export PATH='/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin'

If Zsh is launched as a login shell, then it’ll source ~/.zshenv which will reset PATH to a sensible default. Next, it’ll source /etc/zprofile which will run path_helper and set up the default PATH according to the usual macOS rules.

Third, create a ~/.zprofile file containing all of the exported environment variables you want. You should also prepend directories to PATH in this file. Since this is sourced after path_helper is run, the directories added will not be rearranged.

This is the (partial) contents of my ~/.zprofile.

# Environment variables.
export EDITOR='nvim'
export LANG=en_US.UTF-8
export LC_CTYPE=en_US.UTF-8
export PAGER='less -R'
# Other exported environment variables here.

# Add paths to PATH.
if [[ -f '/nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh' ]]; then
  source '/nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh'
  export NIX_PATH="$HOME/.nix-defexpr"
fi
[[ -f "$HOME/.ghcup/env" ]] && source "$HOME/.ghcup/env"
[[ -f "$HOME/.cargo/env" ]] && source "$HOME/.cargo/env"

Notice that it sets NIX_PATH as described above.

It’s important that the PATH clearing happen in ~/.zshenv and not ~/.zprofile because otherwise system directories from /etc/paths won’t make it into the PATH (or work will have to be duplicated).

Finally, put the per-shell configuration in ~/.zshrc. This includes things like your prompt, shell aliases, shell completions, and so on.

In general, if it’s an exported environment variable, put it in ~/.zprofile. If it’s any other configuration, put it in ~/.zshrc.

With this setup, every new login shell will clear PATH and reset it. Nonlogin shells (interactive or not) will inherit the PATH.

Wrapping up

To summarize, to use Nix on macOS in a reliable and reproducable manner,

1. Undo the modifications the Nix installer made to the files in /etc;
2. Separate your shell configuration into ~/.zshenv, ~/.zprofile, and ~/.zshrc such that exported environment variables go in ~/.zprofile, configuration for interactive shells goes in ~/.zshrc, and ~/.zshenv resets PATH for login shells;
3. Create a ~/.nix-defexpr/nixpkgs/default.nix file with the single line

   import (fetchTarball "https://github.com/NixOS/nixpkgs/archive/87d9c84817d7be81850c07e8f6a362b1dfc30feb.tar.gz")
   

4. Modify ~/.zprofile to run nix-daemon.sh and set NIX_PATH; and
5. Create an env.nix somewhere containing your list of packages and install them with $ nix-env -irf env.nix.

The only thing I haven’t touched on is how you get the revision of nixpkgs to use. How I do it is I run a script (which I called nix-update-nixpkgs) which grabs the revision for the stable Darwin (i.e., macOS) channel and uses that.

#!/usr/bin/env nix-shell
#!nix-shell -i bash -p bash curl jq
# shellcheck shell=bash

# Just go with the revision that works for stable darwin. Might as well for now
# unless there's an issue.

set -euo pipefail

dryrun=0

usage() {
  cat <<USAGEEOF
Usage: $0 [OPTIONS]

Options:
  -h  --help      show this help
  -n  --dry-run   do not make any changes
USAGEEOF
}

for arg in "$@"; do
  case ${arg} in
    '-n' | '--dry-run')
      dryrun=1
      ;;
    '-h' | '--help')
      usage
      exit 0
      ;;
    *)
      echo "$0: Unexpected argument: ${arg}" >&2
      usage >&2
      exit 1
      ;;
  esac
done


revision=$(curl --silent --show-error 'https://monitoring.nixos.org/prometheus/api/v1/query?query=channel_revision' \
  | jq -r '.data.result[]|select(.metric.status == "stable" and .metric.variant == "darwin").metric.revision')
nixexpr="import (fetchTarball \"https://github.com/NixOS/nixpkgs/archive/${revision}.tar.gz\")"
nixpkgsfile=~/.nix-defexpr/nixpkgs/default.nix

mkdir -p "$(dirname "${nixpkgsfile}")"
if [[ -f "${nixpkgsfile}" ]] && diff -q "${nixpkgsfile}" - <<< "${nixexpr}" >/dev/null; then
  echo 'nixpkgs already up to date'
  exit 0
fi

if [[ ${dryrun} -ne 0 ]]; then
  echo "This would set nixpkgs to revision ${revision}"
else
  echo "Setting nixpkgs to revision ${revision}"
  echo "${nixexpr}" >${nixpkgsfile}
fi

"$(dirname "$0")/nix-diff" "${revision}"
# vim: set sw=2 sts=2 ts=8 et ft=bash:

The penultimate line runs my nix-diff script which shows the changes that would be made if $ nix-env -irf env.nix were run. This is based on Ian Henry’s similar script. Here it is.

#!/usr/bin/env nix-shell
#!nix-shell -i bash -p bash jq
# shellcheck shell=bash

# Usage: nix-diff [revision]
# where revision is the full nixpkgs revision.

set -euo pipefail

declare -A cur

query() {
  nix-env --query --json "$@" | jq -r '.[] | .pname + " " + .version'
}

query_args=()

if [[ $# -eq 1 ]]; then
  query_args=('--file' "https://github.com/NixOS/nixpkgs/archive/$1.tar.gz")
fi

while read -r name version; do
  cur[${name}]=${version}
done < <(query "${query_args[@]}")

ret=0
while read -r name version; do
  if [[ -n ${cur[${name}]:-} ]]; then
    if [[ "${version}" != "${cur[${name}]}" ]]; then
      echo -e "\033[33mM ${name} ${cur[${name}]} ➤ ${version}\033[0m"
      ret=1
    fi
    unset "cur[${name}]"
  else
    echo -e "\033[32mA ${name} ${version}\033[0m"
    ret=1
  fi
done < <(query -af "${HOME}/.dotfiles/env.nix")

for name in "${!cur[@]}"; do
  echo -e "\033[31mD ${name} ${cur[${name}]}\033[0m"
  ret=1
done

if [[ ${ret} -ne 0 ]]; then
  # Disable SC2016 (info): Expressions don't expand in single quotes, use double quotes for that.
  # shellcheck disable=SC2016
  echo 'Run `nix-env -irf ~/.dotfiles/env.nix` to make these changes'
fi

exit "${ret}"

# vim: set sw=2 sts=2 ts=8 et ft=bash:

Happy Nixing.