Notices by Lennart Poettering (pid_eins@mastodon.social), page 6

It's a boolean option: if enabled the coredump processing on the host would forward the coredumps to the unit's code. The idea is that a container manager enables this on the container's unit, and this magically ensures that coredumps that happen inside the container are delivered to the container itself, and are then processed inside of it, with the container's own coredumping logic.

Security-wise this is really nice behaviour I think: to a large degree coredump handling inside…

…logged events it all stopped on the container boundary: only with luck you'd get a proper backtrace, but you usually didn't because the coredump processor on the host couldn't deal with the different compiler/debug situation inside the container. Given that containers are mildly successful these days this of course is a big problem.

Back in v255 we added a new unit file setting CoredumpReceive= to unit files (services and scopes in particular), to address this issue.

Except of course, that until recently it all fell apart once containers came into the mix: containers typically indicate a "binary boundary" when it comes to coredump processing: the code running inside the container and the code running on the host typically do not originate from the same source, they are built differently, with different compilers, compiler settings, debug symbols, optimization levels and so on.

And that showed: while coredumps of the system itself were now nicely…

With v257 there's now a knob to address this situation too. systemd-coredump can now be configured (opt-in!) so that it will try to process coredumps of containers *on the host*. If you set EnterNamespace=yes in coredump.conf it will acquire access to the container's mount tree, mount it within its own private mount namespace to some special location, and then run coredump processing on that – while being part of the host runtime in almost all ways.

…in systemd-nspawn: they have a proper init system as PID 1, as well as service management, so that they can actually reasonably process coredumps inside in parallel to whatever else they are supposed to be doing.

Of course, containers in the Docker sense are not like that: they typically run in some weird mixture of "i am an independent system" and "i am part of another system", and the payload is run as PID 1 without any further service management available.

Bummer!

the usual systemd socket activation protocol) that it shall respond to Varlink queries via a passed in socket fd.

The path may also be prefixed with "ssh-unix:" in which case an SSH connection is made to some remote Varlink service.

New with v257 is that "ssh-exec:" is now also available which also sets up an SSH connection, but invokes a specified binary on the remote side, talking to it via standard input/output.

All four ways to communicate (connect to AF_UNIX socket, execute binary, …

Here are two examples using that:

varlinkctl introspect /run/systemd/io.systemd.Credentials

and

varlinkctl call /run/systemd/io.systemd.Credentials io.systemd.Credentials.Encrypt '{"text":"foobar"}'

The 2nd argument always specifies the entrypoint socket to talk to. In most cases where you call this locally that's the file system path of an AF_UNIX socket. However this can also be the path to an executable, in which case the executable is invoked and told via $LISTEN_FDNAMES (i.e.

2️⃣3️⃣ Here's the 23rd post highlighting key new features of the upcoming v257 release of systemd. #systemd257

In systemd, as mentioned in one of the previous installments, we are adopting the Varlink IPC at more and more places.

To interface with Varlink IPC from the command line we provide the "varlinkctl" tool. "varlinkctl introspect <socket>" for example introspects which method calls, types, and errors a service provides. Similar, "varlinkctl call <socket> <method> <json>" calls a method.

… connect to remote AF_UNIX socket, execute remote binary), are also accessible via sd-varlink.

Or in other words: it's now almost as easy to talk to a local Varlink socket as it is to talk to a remote one.

(This is super handy in combination with the AF_VSOCK support we provide for SSH'ing into local VMs, added in v256).

And that's it for today!

1️⃣9️⃣ Here's the 19th post highlighting key new features of the upcoming v257 release of systemd. #systemd257

A relatively basic feature of systemd's service management is the ability to automatically restart a service in case it terminates unexpectedly, configurable via the Restart= setting.

In v254 we added the RestartMode= setting that allows to fine tune the mechanism to use for restarting the service, i.e. it adds a logic to optionally avoid marking the service as failed between…

…named the same way for the lifetime of the system. Different definitions of "same" exist, i.e. some people prefer if the very same physical device always carries the same name (in which case the MAC address is a good identifier, if the device has one), others prefer if the slot the devices are plugged into gets the fixed name so that the devices can be replaced if broken.

The policy which naming scheme to use can be configured, see the systemd.net-naming-scheme(7) man page for details.

1️⃣7️⃣ Here's the 17th post highlighting key new features of the upcoming v257 release of systemd. #systemd257

Linux network interfaces since a longer time have not just a primary name, but any number of additional "alternative" names. Similar semantics apply, but they can be longer and there can be many.

You might know the naming policies that systemd by default applies when picking the (primary) name of hardware network interfaces, to ensure they remain stable, so that the same hw iface is...

But that got fixed in the kernel a while back. And thus with v257 we decided to drop the special casing again. So from now on all applicable predictable network interface names are applied by default to network interfaces, and the naming policy simply controls which one of them to use as primary name.

And that's all for today.

That was because in some situations MAC addresses can get "inherited" from one device to another. (Thus allowing multiple interfaces with the same mac.) Now, the kernel actually allowed to mark these cases so that we can recognize them and simply not apply alternative names bases on "inherited" MAC addresses.

Except of course that various drivers were broken in this regard, most recently some laptop docking stations which "inherited" their MAC address from the laptop's own ethernet port.

Since a while systemd would always follow that policy for the primary name, and then add all other candidate names as alternative names to the interfaces too.

That way, you can use all applicable names for the interfaces in tools that understand the alternative names concept of Linux, and for older tools you can use the primary name as per policy.

We only made one exception from this logic so far: the MAC-address based names (i.e. "enx…" and similar) would not be applied as alternative names.

8️⃣ Here's the 8th post highlighting key new features of the upcoming v257 release of systemd.

A longer time ago systemd introduced JSON based user records as an extension of classic UNIX `struct passwd`. These records can be provided via Varlink IPC or via drop-in files. The much richer set of account settings is documented here:

https://systemd.io/USER_RECORD/

…, in order to avoid a chicken/egg problem, a cyclic dependency, and deadlocks. (Example: journald provides logging to the D-Bus broker, and hence cannot provide APIs via D-Bus. Similar PID 1 itself, or systemd-userdb/systemd-homed which provide user record resolution which D-Bus needs for its policies, and so on and so on).

These problems are very hard to tackle. For example in PID 1 itself we provide our D-Bus APIs not just via the broker, but also via another local "direct" socket.

D-Bus' model is built around a central broker daemon, which is started during boot, but unfortunately relatively late (i.e. together with other, regular daemons instead of early boot or the initrd). However, systemd brings up the system as a whole and hence needs IPC from earliest moment on.

And then there are various components of systemd that the D-Bus broker relies on (i.e. consumes functionality of) and hence cannot themselves provide their services on D-Bus, …

5️⃣ Here's the 5th installment of posts highlighting key new features of the upcoming v257 release of systemd.

Since its beginnings systemd has been a heavy user of the D-Bus IPC system. It provides D-Bus APIs, it calls D-Bus APIs, it schedules activation of the D-Bus broker, and even provides its own C D-Bus client library (sd-bus).

However, since early on our use of D-Bus was not without various major problems. One of the biggest goes something like this:

That alternative IPC is called Varlink (https://varlink.org/). It has been around for a while, and initially we only adopted it where D-Bus was just too bad to use, and only internally. Over the last couple of releases that changed however: we started to make heavier use of it and provide public interfaces via Varlink in addition or instead of D-Bus.

In many ways Varlink is much nicer to work with than D-Bus: it's a lot simpler, it's brokerless design make it a ton faster, …