Privilege drop, privilege separation, and restricted-service operating mode in OpenBSD

My main focus in OpenBSD are privilege separated network daemons running in restricted-service operation mode. I gave talks at BSDCan and FOSDEM in the past about how I used these techniques to write slaacd(8) and unwind(8). While I do not think of myself as a one-trick pony, I have written some more: slowcgi(8), rad(8), dhcpleased(8), and gelatod(8). I also wrote the first version of what later turned into resolvd(8).

At one point I claimed that it would take me about a week to transmogrify one daemon into a new one.

Privilege drop, privilege separation, and restricted-service operating mode are exploit mitigations. When¹ an attacker finds a bug we try to stop them from causing damage. The mitigations we are talking about here are aimed at attackers that achieved arbitrary code execution. Due to other mitigations that is quite difficult to pull off. These are the last line of defence. We try to remove as many resources from the attacker to play with and try to crash the program as quickly as possible if an attacker touches something they are not supposed to.

Privilege drop is probably the weakest mitigation discussed in this article. It is a very old technique, but still important for set-user-ID root binaries.

Theo de Raadt refactored ping(8) over 26 years ago to open a raw socket early and then drop root privileges. This prevents a local user from elevating their privileges when finding a bug in ping(8):

@@ -191,6 +191,14 @@
        char rspace[3 + 4 * NROUTES + 1];       /* record route space */
 #endif

+       if (!(proto = getprotobyname("icmp")))
+               errx(1, "unknown protocol icmp");
+       if ((s = socket(AF_INET, SOCK_RAW, proto->p_proto)) < 0)
+               err(1, "socket");
+
+       /* revoke privs */
+       setuid(getuid());
+
        preload = 0;
        datap = &outpack[8 + sizeof(struct timeval)];
        while ((ch = getopt(argc, argv, "DI:LRS:c:dfh:i:l:np:qrs:T:t:vw:")) != EOF)
@@ -235,6 +243,8 @@
                        loop = 0;
                        break;
                case 'l':
+                       if (getuid() != 0)
+                               errx(1, "must be root to specify preload");
                        preload = strtol(optarg, NULL, 0);
                        if (preload < 0)
                                errx(1, "bad preload value: %s", optarg);
@@ -323,12 +333,6 @@
                        *datap++ = i;

        ident = getpid() & 0xFFFF;
-
-       if (!(proto = getprotobyname("icmp")))
-               errx(1, "unknown protocol icmp");
-       if ((s = socket(AF_INET, SOCK_RAW, proto->p_proto)) < 0)
-               err(1, "socket");
-       hold = 1;

        if (options & F_SADDR) {
                if (IN_MULTICAST(ntohl(to->sin_addr.s_addr)))

20 years later we realized that we would not drop privileges when ping(8) is invoked as root. We would just "drop" from root to root. We can protect ourselves from a malicious ping target by dropping to a dedicated user:

@@ -272,8 +275,12 @@

        /* revoke privs */
        uid = getuid();
-       if (setresuid(uid, uid, uid) == -1)
-               err(1, "setresuid");
+       if ((pw = getpwnam(PING_USER)) == NULL)
+               errx(1, "no %s user", PING_USER);
+       if (setgroups(1, &pw->pw_gid) ||
+           setresgid(pw->pw_gid, pw->pw_gid, pw->pw_gid) ||
+           setresuid(pw->pw_uid, pw->pw_uid, pw->pw_uid))
+               err(1, "unable to revoke privs");

        preload = 0;
        datap = &outpack[ECHOLEN + ECHOTMLEN];

ping(8) needs a raw socket to be able to send ICMP echo request packets. This is an operation that only root is allowed to do². Once that socket is open though, ping(8) no longer needs to do any other privileged operation. It can hold on to the socket for later use and drop root privileges.

Another use for privilege drop is in daemons to restrict file-system access by chroot(2)'ing to /var/empty. The daemon needs root privileges to call chroot(2), but afterwards it can run without elevated permissions. To the process it looks like there is only the / directory where it does not have any permissions.

The standard pattern can be seen in frontend.c of rad(8):

if ((pw = getpwnam(RAD_USER)) == NULL)
        fatal("getpwnam");

if (chroot(pw->pw_dir) == -1)
        fatal("chroot");
if (chdir("/") == -1)
        fatal("chdir(\"/\")");

if (setgroups(1, &pw->pw_gid) ||
    setresgid(pw->pw_gid, pw->pw_gid, pw->pw_gid) ||
    setresuid(pw->pw_uid, pw->pw_uid, pw->pw_uid))
        fatal("can't drop privileges");

We first get a user with getpwnam(3) to drop to. The user has /var/empty configured as its home directory so we can use that in chroot(2). Next we chdir(2) to the new file-system root to have a valid current working directory. This prevents us accidentally marking a file-system as busy depending on from where the daemon was started, preventing unmounting file-systems while the daemon is running.

We then drop privileges by putting the user into a single group using setgroups(2). The calls to setresgid(2) and setresuid(2) set the real, effective and saved group and user IDs. This safely drops from root:wheel to (in this case) _rad:_rad with no way to escalate back to root.

This technique is probably used in most, if not all, OpenBSD's privilege separated daemons.

With privilege drop in ping(8) we prevent a local unprivileged user to gain superuser or root privileges. If there were a bug in message parsing³, a malicious ping target, or even host in the middle, could still read and exfiltrate ssh private keys. ping(8) runs as my user-id. It can read all files my user can read, it can open network connections to any host on the internet, it can execute arbitrary programs, heck it can talk to my GPU. That is a lot of power that it does not need. It only needs to write to stdout and stderr⁴, and send and receive ICMP packets.

We could lock ping(8) away using chroot(2), that at least takes away file-system access. But what can we do about programs that need file-system access but should not execute other programs or talk to the Internet? Like file(1) for example.

Decades ago Niels Provos developed systrace(4) but it turned out that it was difficult to use. The only user in OpenBSD base was sshd(8).

In 2015 Theo de Raadt tricked Nicholas Marriott into privilege separating and sand-boxing file(1) using systrace. It lasted for half a year, it was that painful.

One problem with systrace(4) was that it worked on the level of syscalls and their arguments. This is not something user-land developers are intimately familiar with. We are interacting with libc and do not know what kind of syscalls libc does on our behalf. Another issue is that a program might need some ioctl(2)s or sysctl(2)s but it should not be able to do all of them. So we need to encode restrictions on arguments of syscalls. This gets unwieldy, fast.

There was also systrace(1) to define a policy outside of the program. It turns out that most programs need to do some sort of initialization where they need wide access to the system. This is before they touch untrusted data. Once the initialization is done we can restrict access. How much we can restrict access can depend on command line flags. systrace(1) could not help with this, the program would retain all the privileges it needed for initialization. They would be fewer than all privileges, but still way to many.

As far as I know, the experience with file(1) was the last straw. Theo set out to improve on this situation by developing tame(2), which was later renamed to pledge(2)⁵.

pledge(2) was developed by studying all programs in OpenBSD base and putting their needed services into categories using broad strokes like memory management, read-write on open file descriptors, opening of files, or networking. If a program violates what it pledged to do, for example trying to open a file when it did not pledge rpath, it will be terminated with an uncatchable SIGABRT.

It is worth repeating that: If a program violates what it pledged to do it will be terminated by the kernel. An attacker does not get to play again and try something else⁶.

This was an iterative process with patches floating around. A few co-conspirators, including myself, joined the effort a bit later to add pledge(2) to more programs. Once we hit 50 or so pledged programs, Theo considered it mature enough for commit and work continued in tree. Soon after, the list of programs not pledged at all was shorter than the list of pledged programs. This is a huge success and speaks to the usability of pledge(2). In decades we only had one program using systrace(4) in OpenBSD base, then pledge(2) shows up and in less than a year nearly all of OpenBSD base uses it.

To add pledge(2) to a program we need to know what it does and potentially re-factor it to pull (hoist) one-time initialization up before pledge(2) is called for the first time. Some programs are sloppy in the sense that they open a certain resource the moment they need it, this means that they retain more access than they need. As we have seen with ping(8), if we pull opening of the raw socket before option parsing we can drop root privileges before touching untrusted data⁷.

Since pledge(2) is internal to the program we can call it once we are done with option parsing and pledge different things depending on given options. For example, ping(8) retains the ability to do DNS lookups depending on the -n flag:

if (options & F_HOSTNAME) {
        if (pledge("stdio inet dns", NULL) == -1)
                err(1, "pledge");
} else {
        if (pledge("stdio inet", NULL) == -1)
                err(1, "pledge");
}

pledge(2) is not fine-grained. It turns out that programs fall into broad categories of what they want to do after initialization. There are not hundreds of different promises for every obscure program, it is not needed. To add a new promise, a rule of thumb is: At least two programs have been identified that need a new promise. To add a syscall to an existing promise, that is, to give more power to an existing promise, needs careful evaluation of what all the other programs already using the promise, gain. It is not enough to show that it is fine for the new program, existing programs are much more important. Another question is how much additional kernel attack surface this exposes.

pledge(2) does not only protect the user of the system or systems on the Internet from harm when a bug is found, it also protects the kernel from user-land. Checking if a syscall is allowed happens early, before a lot of kernel code runs.

pledge(2) can be used to gain understanding on what a program does. We see the following pledges in file(1):

$ cat -n file.c | fgrep 'pledge("'
 171 if (pledge("stdio rpath getpw recvfd sendfd id proc", NULL) == -1)
 210 if (pledge("stdio rpath sendfd", NULL) == -1)
 374 if (pledge("stdio getpw recvfd id", NULL) == -1)
 389 if (pledge("stdio recvfd", NULL) == -1)

The reader is encouraged to stop here and read the code around those pledge(2) calls to figure out what file does and why it pledges those things.

On line 171, file(1) pledges all the things it needs. It can read-write already open file descriptors⁸, open files for reading, pass file descriptors around, figure out under which user it runs and lookup another user. It can also fork itself.

On line 210 it forked itself and we are in the parent process. It can shed all pledges that the forked child needs for its initialization as well as the ability to fork another instance. The parent process can now only open files for reading, pass those file descriptors to the child process and read-write already open file descriptors.

On line 374 we are in the child process and we shed all pledges that the parent needed. We need to be able to read-write open file descriptors and receive file descriptors from the parent process. During initialization we need to know if we are running as root and be able to look up a user. file(1) can privilege drop to a dedicated user when invoked as root! Once we are done with that, the child process, which does all the magic⁹, sheds all those additional privileges and on 389 it can only read-write existing file descriptors and receive new file descriptors from the parent process.

We see file(1) is privilege separated, running two processes. One process opens files but does not look at the contents. The other process, which is completely locked away, parses untrusted data and informs the parent process about the result.

Neither process can talk to the Internet or write to disk. They cannot create new files or open existing files for writing. While it can read ssh private keys, it cannot exfiltrate them.

We can find out what file(1) can and cannot do from just those four pledge lines, that is very powerful when starting a code review. They can also be used to get a quick overview how file(1) operates by reading the code adjacent to the pledge(2) calls, that is where interesting stuff happens.

A single process design that pledges "stdio inet rpath" still has a lot of attack surface. This is not good if it is a network daemon running as root and enabled per default on all installations. Like dhcpleased(8) for example. For starters it can read and exfiltrate ssh private keys.

As we have seen with file(1), we can split up a program into multiple communicating processes that each pledge less operations than the sum of all pledges. We can move the risky operations of parsing untrusted data to a process that does not have access to the Internet, nor the file-system. That process will also not have any elevated privileges to change the system configuration like configuring network addresses or changing the routing table.

An attacker who finds a loophole into this least privileged process will have a hard time creating havoc. They can only talk to more privileged processes using a very narrow communication channel with easy to parse¹⁰ messages.

We will look in detail at how dhcpleased(8) uses privilege separation and pledge(2) to implement a DHCP client in a safe way. Other network daemons follow a similar pattern and the reader should be able to study the source code of rad(8), slaacd(8), and unwind(8) to find out how those daemons work since they share a common ancestry.

Writing software in C with security in mind can be a lot of fun when standing on the shoulders of giants and having things like privilege separation and restricted-service operating mode in your toolbox.

I wrote two daemons, dhcpleased(8) and slaacd(8), that are enabled by default on every OpenBSD installation. We might eventually add a third one. Having the mitigations shown here as well as all the other mitigations constantly being added and enabled per default is what lets me sleep at night. I am cautiously optimistic that when a bug is found in dhcpleased(8) or slaacd(8) an attacker will have a hard time pivoting to arbitrary code execution as root.