Avoiding Complexity with Systemd

Unix machines, since the early days of the operating system, have been designed for multiple users to use concurrently. Traditionally there is a set of “unprivileged” users used by people and system services, and the root account which can generally do anything. Because of the concept that most things in Unix are represented by a file, users could be allowed to perform various operations by adding them to groups and using filesystem permissions. There were also other functions which could not be delegated in this way–notably, binding to certain IP ports. Various operating systems developed over the years have blurred these lines a little, particularly on Linux which has features like capabilities and ACLs that allow more control than the standard Unix permission model provides.

Linux goes much further than this. There are mandatory access control systems like SELinux or AppArmor that let you apply restrictions at the kernel level outside of the software you're running. Features like `cgroups` and namespaces combine to provide what we now call containers. Other features like `seccomp` allow software to opt-in to limits on its own ability to use various system calls.

BSD operating systems have similar features, notably pledge and unveil on OpenBSD.

Much of the original permission model remains to this day. If you want to run a service that listens on a “well-known” port, those numbered less than 1024, you generally need to be root to bind to the port. There are other ways to allow this on Linux, such as applying a capability called CAP_NET_BIND_SERVICE to the program you want to run, but most server software that is designed to be portable among Unix systems implements a feature called privilege dropping. The service initially starts as root, binds to the ports that it requires, and then calls some functions to set its own user and group IDs to an unprivileged user. Ideally it does this before doing any significant work in order to minimise the potential for an exploit to occur while running as root.

Even if the service isn’t going to bind to a privileged port, sometimes there are files only readable by root that are required for the service to operate. A good example of these are the private keys for TLS certificates, which often live in a root-owned directory under /etc. It’s common to do a similar trick to access these–start the service as root, open the files, and then drop the privileges once they’ve been read. Ideally the service will drop the privileges before parsing anything. Parser code needs to be written very carefully, especially in languages with manual memory management, and it has historically been a source of security vulnerabilities.

All of this adds a bit of complexity to the services we write, which it would be nice to avoid. It also adds to the attack surface: privilege dropping code has been a source of vulnerabilities, notably on a couple of occasions in Bash. Avoiding writing it at all, or at least delegating it to other software with more testing than our own, would be good.

It’s fairly common these days to write a service that runs on an unprivileged port and then run some other software in front of it as a reverse proxy–often nginx or Apache are used for this purpose. Depending on the use case these may provide some advantages, but they do require additional configuration and will use some resources on the machines operating the service.

An example

I’m working on my new startup idea–Lunch as a Service, a new SaaS application to solve the complex problem of deciding what to have for lunch. While I’m working on getting the VC funding together, I’m not going to be deploying it on a huge Kubernetes setup with a service mesh and a GitOps CI/CD platform. While that would be nice one day, for now I’m going to make do with a VM.

The core of the new service is lunchd. It looks like this.

package main

import (
        "fmt"
        "math/rand"
        "net/http"
        "time"
)

var lunchOptions = []string{
        "Sandwich",
        "Soup",
        "Salad",
        "Burger",
        "Sushi",
}

func getRandomLunch() string {
        return lunchOptions[rand.Intn(len(lunchOptions))]
}

func main() {
        rand.Seed(time.Now().UnixNano())

        http.HandleFunc("/", func(w http.ResponseWriter, req *http.Request) {
                fmt.Fprintf(w, "<h1>%s</h1>", getRandomLunch())
        })

        fmt.Println("Starting web server on port 8080")
        http.ListenAndServe(":8080", nil)
}

I’m using a reasonably recent Linux distribution with systemd, so I’m going to use that to start the service when the machine boots. The basic unit file looks like this.

# This goes in /etc/systemd/system/lunchd.service
[Unit]
Description=Lunch as a Service

[Service]
ExecStart=/usr/local/bin/lunchd
User=lunchd

[Install]
WantedBy=multi-user.target

I’ve added a user called lunchd for the the service. After running sudo systemctl daemon-reload, I can then do sudo systemctl start lunchd. The service is hard coded to run on port 8080 right now, which is not a privileged port so it’ll start quite happily and run as a standard user. The final line makes sure it starts when the system boots normally.

Getting it ready for production

I can boot the application and have it run as a user other than root, which is a good start. However, I want it to be able to run on port 443 and use a TLS certificate. I would also like to use some of the filesystem protection features that systemd advertises. Can I use all of these to avoid writing code to drop privileges?

Filesystem sandboxing

Our service doesn’t require much in the way of filesystem access. systemd provides ways to restrict the parts of the filesystem the service can see. By doing so, we can limit some of the opportunities for an attacker if the service is compromised.

• ProtectSystem can be set to full to make /usr, /boot read-only for this process. If set to strict, /etc is read-only too. This is fine for this service as it doesn’t read anything, so we’ll enable that.
• ProtectHome can be set to true to make /home, /root and /run/user empty and inaccessible from the point of view of the service.
• PrivateTmp makes sure that the process’s temp directories are only visible to itself, and not another process. Additionally, they’ll be emptied once the process finishes.

The complete list of options can be found in the systemd documentation.

Let’s add some of these features to our unit file.

[Unit]
Description=Lunch as a Service

[Service]
ExecStart=/usr/local/bin/lunchd
ProtectSystem=strict
ProtectHome=true
PrivateUsers=true
PrivateTmp=true

[Install]
WantedBy=multi-user.target

Dynamic users

Traditionally, many services have expected to run as a defined user and group, usually specified in their configuration file. Some will start and run as that user if they don’t need to do anything privileged, but others will start as root and switch. Many common internet services such as web and mail servers operate in this way.

With systemd we can have a brand new user and group allocated for us when the service starts. These users are prevented from changing any state in the system except in the directories detailed above. Once the service exits, the user is removed–it never actually exists in /etc/passwd. This makes it harder to create a persistent route in to a system after an exploit.

[Unit]
Description=Lunch as a Service

[Service]
ExecStart=/usr/local/bin/lunchd
ProtectSystem=strict
ProtectHome=true
PrivateUsers=true
PrivateTmp=true

DynamicUser=yes

[Install]
WantedBy=multi-user.target

For this I’ve removed the system user I added earlier. After another daemon-reload and restarting the service, we can see that the service is still running as a user called lunchd, but it has a very high user ID:

mgdm@lunchbox:~$ ps aux | grep lunchd
lunchd     33146  0.2  0.4 1002788 4312 ?        Ssl  20:28   0:00 /usr/local/bin/lunchd
mgdm@lunchbox:~$ id lunchd
uid=62840(lunchd) gid=62840(lunchd) groups=62840(lunchd)

None of these changes have, so far, required much in the way of modification to our source code. At the most, we just need to make sure our service can accept some configuration for the directories it writes to.

Using port 80

Although we’ve added some security features, our service is still running on port 8080. That’s fine for development purposes, but we’ve run into the first of the problems I described at the start. We want to bind to a privileged port, but the service is now definitely running as an unprivileged user.

systemd has a feature called socket activation, which allows it to bind to a port and then hand a file descriptor for that port to a process it launches. It can either launch the process once per connection, in a similar fashion to the old inetd daemon, or it can bind to the port once and hand it to a persistent process. This latter option is what we’ll do. It does require some modification to the code, but none of it will ever run as root which further reduces the attack surface.

It’s certainly possible to hand-write this code but in this case I’m going to use a package from CoreOS. Here’s the modified version.

package main

import (
        "fmt"
        "log"
        "math/rand"
        "net"
        "net/http"
        "time"

        "github.com/coreos/go-systemd/activation"
)

var lunchOptions = []string{
        "Sandwich",
        "Soup",
        "Salad",
        "Burger",
        "Sushi",
}

func getRandomLunch() string {
        return lunchOptions[rand.Intn(len(lunchOptions))]
}

func getListeners() (listeners []net.Listener, err error) {
        listeners, err = activation.Listeners()

        if err != nil || len(listeners) != 1 {
                log.Printf("no listeners supplied: %s", err)

                listeners = make([]net.Listener, 1)
                listeners[0], err = net.Listen("tcp", ":8080")
        }

        return
}

func main() {
        rand.Seed(time.Now().UnixNano())

        listeners, err := getListeners()

        if err != nil {
                log.Panicf("Could not set up listener: %s", err)
        }

        http.HandleFunc("/", func(w http.ResponseWriter, req *http.Request) {
                fmt.Fprintf(w, "<h1>%s</h1>", getRandomLunch())
        })

        log.Printf("Starting web server on port %s", listeners[0].Addr().String())
        http.Serve(listeners[0], nil)
}

Once we’ve built the new version, we can test it:

mgdm@lunchbox:~$ systemd-socket-activate -l 8080
Listening on [::]:8080 as 3.
Communication attempt on fd 3.
Execing ./main (./main)
Starting web server on port 8080

This runs fine. To make it work this way permanently, we need to create a socket file for systemd in addition to the service file. It describes the port you want the systemd to bind to and pass to the service. This can go in /etc/systemd/system/lunchd.socket.

[Socket]
ListenStream = 80
BindIPv6Only = both

[Install]
WantedBy = sockets.target

After another systemctl daemon-reload, you can then type systemctl start lunchd.socket which will make it listen to the port. It can then boot your service on the first request, or you can do systemctl start lunchd yourself if you think that would take a while.

The advantage of this is that if the service crashes, requests can still come in on port 80 and systemd will take care of starting a new one for you. Theoretically at least, this means you shouldn’t drop those requests.

Adding HTTPS

For this we’ll need a certificate, which most of the time we can get from LetsEncrypt. However, most LetsEncrypt clients will place the private key somewhere in /etc in a directory only accessible by root. This is fine, as having the private key owned by another user makes it harder to steal in the event our service is compromised. (We’ll just have to hope we don’t have some kind of memory leak attack). We want to be able to use the certificate, but we don’t want to use privilege dropping here.

Normally for this, we’d use code something like the following, taken from the Go documentation:

func ExampleLoadX509KeyPair() {
        cert, err := tls.LoadX509KeyPair("example-cert.pem", "example-key.pem")

        if err != nil {
                log.Fatal(err)
        }

        cfg := &tls.Config{Certificates: []tls.Certificate{cert}}
        listener, err := tls.Listen("tcp", ":2000", cfg)

        if err != nil {
                log.Fatal(err)
        }

        _ = listener

}

This assumes the process is going to be able to open the files from whatever user it’s running as. I’ve used certbot to create a LetsEncrypt certificate, which is stored in /etc/letsencrypt/live/lunch.mgdm.net and is only readable by root.

We can work around this using systemd. It can open the files and present them in a modified filesystem view to the service. We’ll need to make some more changes to our code, but we were going to have to do this anyway to add the TLS. In this case, I’ve added flags called key and certificate which provide the paths to the TLS private key and the certificate with its chain.

The key thing here is the LoadCredential setting. It takes both a key and a path, separated by :. systemd will load the contents of the file at the specified path, and expose it in a directory as a file named after the key. This directory is given to the process as an environment variable called CREDENTIALS_DIRECTORY, which can also be used in the ExecStart line of the unit file. You can see this in action in the modified unit file below. We’ve also made a change to the socket file in order to make it listen on port 443 instead.

lunchd.service

[Unit]
Description=Lunch as a Service

[Service]
ExecStart=/usr/local/bin/lunchd -key=${CREDENTIALS_DIRECTORY}/key.pem -certificate=${CREDENTIALS_DIRECTORY}/chain.pem

LoadCredential=key.pem:/etc/letsencrypt/live/lunch.mgdm.net/privkey.pem
LoadCredential=chain.pem:/etc/letsencrypt/live/lunch.mgdm.net/fullchain.pem

ProtectSystem=strict
ProtectHome=true
PrivateUsers=true
PrivateTmp=true
DynamicUser=yes

[Install]
WantedBy=multi-user.target 

lunchd.socket

[Socket]
ListenStream = 443
BindIPv6Only = both

[Install]
WantedBy = sockets.target

main.go

# main.go
package main

import (
        "crypto/tls"
        "flag"
        "fmt"
        "log"
        "math/rand"
        "net"
        "net/http"
        "time"

        "github.com/coreos/go-systemd/activation"
)

var lunchOptions = []string{
        "Sandwich",
        "Soup",
        "Salad",
        "Burger",
        "Sushi",
}

func getRandomLunch() string {
        return lunchOptions[rand.Intn(len(lunchOptions))]
}

func getListeners() (listeners []net.Listener, err error) {
        listeners, err = activation.Listeners()

        if err != nil || len(listeners) != 1 {
                log.Printf("no listeners supplied: %s", err)

                listeners = make([]net.Listener, 1)
                listeners[0], err = net.Listen("tcp", ":8080")
        }

        return
}

func getCertificates() (key string, certificate string) {
        keyPath := flag.String("key", "", "The path to the private key")
        certPath := flag.String("certificate", "", "The path to the certificate")
        flag.Parse()

        return *keyPath, *certPath
}

func tlsSetup(keyPath string, certPath string, listener net.Listener) (tlsListener net.Listener, err error) {
        config := &tls.Config{
                Certificates:             make([]tls.Certificate, 1),
                NextProtos:               []string{"h2", "http/1.1"},
                PreferServerCipherSuites: true,
        }

        log.Printf("Loading certs from key: %s and cert: %s", keyPath, certPath)
        config.Certificates[0], err = tls.LoadX509KeyPair(
                certPath,
                keyPath,
        )

        if err != nil {
                log.Printf("Failed to configure TLS: %s", err)
                return nil, err
        }

        tlsListener = tls.NewListener(listener, config)
        return
}

func main() {
        rand.Seed(time.Now().UnixNano())

        listeners, err := getListeners()

        if err != nil {
                log.Panicf("Could not set up listener: %s", err)
        }

        keyPath, certPath := getCertificates()
        tlsListener, err := tlsSetup(keyPath, certPath, listeners[0])

        http.HandleFunc("/", func(w http.ResponseWriter, req *http.Request) {
                fmt.Fprintf(w, "<h1>%s</h1>", getRandomLunch())
        })

        fmt.Printf("Starting web server on port %s", tlsListener.Addr().String())
        http.Serve(tlsListener, nil)
}

I could modify the service further so that systemd sets up listeners on both ports 80 and 443, so the service deals with both HTTP and HTTPS itself, but I think I’ll leave that as an exercise for the reader.

References

I found the following articles helpful.

1. The documentation for systemd.exec
2. Options for hardening systemd service units
3. Integration of a Go service with systemd: socket activation by Vincent Bernat

from Hacker News https://ift.tt/3A4PcXK