Environment generators

systemd --user does not load environment.d directories directly. In fact, it knows nothing whatsoever about those directories and does not support environment variable expansion with "+" and "-" at all. Instead, systemd calls out to a set of "environment generators" — small programs which are invoked when systemd starts and whose job is to print environment variable assignments on standard output.

Environment generators mirror unit file generators which have been a part of systemd since almost the beggining. Environment generators live in /usr/lib/systemd/system-environment-generators and /usr/lib/systemd/user-environment-generators, and unit file generators live in /usr/lib/systemd/system-generators and /usr/lib/systemd/user-generators. Unit file generators were added to move some complexity and code which requires optional dependencies from the main systemd code into separate binaries.

The man page describes the differences (environment-generators(7)). The two important ones are that environment generators are run sequentially, so that later generators can use the environment variables defined by earlier generators and can override their output, and that the user environment generators are run by the user manager, so their output can be tailored to the user (even though currently the generators themselves are installed system-wide).

Currently, only one generator is available: the systemd-environment-d-generator, which adds support for environment.d as described in the first part of this note.

The reasons

The main motivation for this blog post was provide some background for the choices made in the implementation of environment.d.

First, let's consider the question why we need to support loading environment variables, and why do we need it in systemd.

Traditionally, Linux has not had a uniformly supported mechanism to define the environment. There are at least two widely used mechanisms. The most straightforward approach is the shell profile configuration. Shells usually execute any scripts in /etc/profile.d/ and load /etc/environment. An obvious shortcoming is that this mechanism requires a shell to be used. Before systemd this wasn't that much of a problem, because shell was omnipresent and the invocation of most programs went through a shell at some point. In particular, graphical sessions invoked a bunch of shell scripts in /etc/X11/xinit/xinitrc.d/, so starting one more shell to load the user environment wasn't an issue. With systemd things are moving towards a shell-free environment, and Wayland sessions could do without a shell at all. In fact this was the case with gnome-wayland, until the need to load the environment forced the gnome developers to undo that (gnome bug 736660).

Another mechanism is pam_env. This PAM module supports reading of system-wide configuration from /etc/environment and (optionally) user-specific configuration from ~/.pam_environment. It is fairly widely supported, but cannot be considered a solution. The syntax is a bit quirky:

VARIABLE [DEFAULT=[value]] [OVERRIDE=[value]]

with @{VAR} syntax for PAM variables, and ${VAR} for shell variables. This is obviously not shell-compatible, and fairly easy to get wrong (for example the man page warns that $HOME may not be set). The real deal breaker is the fact that the loading of the environment happens in privileged mode. Setting arbitrary variables is very tricky to do safely (think not just $PATH, but also $LD_PRELOAD, $LD_LIBRARY_PATH, and various other variables which, if set arbitrarily by the user, might derail the code that is running as root). pam_env(8) suggests that this module should be last one the in the PAM stack, but (even assuming that this is enough to close the hole), it's too easy to get this wrong. Thus, the loading of user configuration is usually disabled (at least in the redhattish and debianesque distros). We need a solution which is more flexible and which does not require a parser running as root and which allows us to safely define arbitrary variables.

In the new scheme of things, systemd (the system instance), starts user@.service for the user, and the systemd running in user@.service directly forks various processes. Systemd is already in the business of defining the environment for child processes, and the systemd --user instance is pretty much the only place where we can add environment variables. It has to be done by the manager itself, because we want all processes to inherit the same environment. It is possible to upload new environment settings after the manager is already running (e.g. systemctl set-environment), but this would only apply to services started later. Thus if we want to influence all services started by systemd --user, the manager itself should support doing this during startup. Since the user manager runs with the privileges of the user, no privilege boundaries are crossed. (In fact the environment defined by generators is not used by the manager itself, but that's because there doesn't seem to be any need, and not because it would be insecure).

Second, let's consider the question whether the extended substitution syntax with ${..:-...} and ${..:+...} and arbitrary generators is needed. I think that the example with $PATH and $PYTHONPATH above shows why some kind of conditional logic is needed. Without it various variables could not be set in a useful way (also $XDG_DATA_DIRS, $LD_LIBRARY_PATH, etc.). The extended substitution syntax solves our immediate needs, but think it would be too constraining if it was the only option. Even relatively simple things like iterating over a set of directories and adding them to a list if they exist, or setting some variable only for specific users, or defining a variable if some other variable has a specific value, would be impossible. I felt that we'd face immediate pressure to add more logic, until Turing-completeness would be reached. This would be implemented in the systemd manager itself, and we'd end up either defining a custom DSL or executing shell code ourselves. It seems much better to sidestep this problem, and allow arbitrary executables to be used as generators. The manager then only needs to fork off the generators (and forking off processes is something that systemd does very well), and parse very simple pairs of VAR=value assignments. On the other hand, generators can be very flexible and implement arbitrary logic. The way that generators are ordered and their input and output are clearly defined and hopefully we'll end up with an ecosystem of generators that do their job without stepping on each others' toes.

Third, note that keeping the environment in systemd --user is quite efficient, because parsing of the environment is done just once. systemd-environment-d-generator is a written in C and is quite lean, but a custom generator in bash or Python is possible in one line. If a generator proves generally useful it can always be rewritten in a compiled language for efficency.

Support for /etc/environment

/etc/environment is pulled into environment.d by a symlink, so it is read by systemd-environment-d-generator.

Current status and further work

The idea and implementation of environment.d directories came from Ray Strode (a.k.a. halfline). I added the support for generators and repurposed Ray's code for systemd-environment-d-generator. Following Lennart Poettering's suggestion generators are run in a pipeline, the output of one forming the input for the next one. If any of that turns out to be a bad idea, blame me ;)

Current code loads environment.d into the user manager instance. This defines the environment of processes started by systemd. It can be easily exported (systemctl --user show-environment), but to load it into other login sessions, support from their side is needed. gdm does that already, and login might in the future.