Notices by Lennart Poettering (pid_eins@mastodon.social), page 4

Here's a blog story with links and very brief summaries of all those #systemd257 stories on Mastodon:

https://0pointer.net/blog/announcing-systemd-v257.html

Enjoy! And stay tuned for #systemd258!

All in all I would say systems became a lot more debuggable out of the box this way, which is not just good for quickly tracking down issues in production environments, but I also see as a relevant in context of the open source philosophy: since the whole OS is typically open source, it also means coredumps are comprehensively useful, since you can always cross-link the stackframes to the sources: the pathway from execution to the sources behind it is now nicely paved.

…are in a pretty good position: we use libdw to generate the stack trace, running inside a local sandbox (since generating stacktraces means analyzing frequently corrupted, possibly differently privileged, complex data, which is hence security sensitive par excellence), and since the relevant distributions now ship minidebuginfo packages and are built with frame pointers enabled the default stacktraces you get this way are typically quite useful – without having to bother with gdb or so.

3️⃣2️⃣ Here's the 32nd post highlighting key new features of the current v257 release of systemd. #systemd257

One of the features we added on early to systemd was coredump processing. We wanted that crashing processes on one hand could be treated very much like any loggable event, and on the other hand be truly and immediately useful, i.e. the log messages generated should already carry a fully symbolized backtrace.

The path towards that goal was rocky, but today I think we…

…the container is just like handling on the host, and the processing of the dump data is done within the immediate sandbox and context of the code that owns it. Great!

Except of course that this is only a full solution if the container actually is able to do all that, i.e. is complete enough to actually do this kind of processing on its own. Effectively this means that the CoredumpReceive= logic only really works for "full-OS" containers, i.e. containers how they are typically run…

It's a boolean option: if enabled the coredump processing on the host would forward the coredumps to the unit's code. The idea is that a container manager enables this on the container's unit, and this magically ensures that coredumps that happen inside the container are delivered to the container itself, and are then processed inside of it, with the container's own coredumping logic.

Security-wise this is really nice behaviour I think: to a large degree coredump handling inside…

…logged events it all stopped on the container boundary: only with luck you'd get a proper backtrace, but you usually didn't because the coredump processor on the host couldn't deal with the different compiler/debug situation inside the container. Given that containers are mildly successful these days this of course is a big problem.

Back in v255 we added a new unit file setting CoredumpReceive= to unit files (services and scopes in particular), to address this issue.

Except of course, that until recently it all fell apart once containers came into the mix: containers typically indicate a "binary boundary" when it comes to coredump processing: the code running inside the container and the code running on the host typically do not originate from the same source, they are built differently, with different compilers, compiler settings, debug symbols, optimization levels and so on.

And that showed: while coredumps of the system itself were now nicely…

With v257 there's now a knob to address this situation too. systemd-coredump can now be configured (opt-in!) so that it will try to process coredumps of containers *on the host*. If you set EnterNamespace=yes in coredump.conf it will acquire access to the container's mount tree, mount it within its own private mount namespace to some special location, and then run coredump processing on that – while being part of the host runtime in almost all ways.

…in systemd-nspawn: they have a proper init system as PID 1, as well as service management, so that they can actually reasonably process coredumps inside in parallel to whatever else they are supposed to be doing.

Of course, containers in the Docker sense are not like that: they typically run in some weird mixture of "i am an independent system" and "i am part of another system", and the payload is run as PID 1 without any further service management available.

Bummer!

the usual systemd socket activation protocol) that it shall respond to Varlink queries via a passed in socket fd.

The path may also be prefixed with "ssh-unix:" in which case an SSH connection is made to some remote Varlink service.

New with v257 is that "ssh-exec:" is now also available which also sets up an SSH connection, but invokes a specified binary on the remote side, talking to it via standard input/output.

All four ways to communicate (connect to AF_UNIX socket, execute binary, …

Here are two examples using that:

varlinkctl introspect /run/systemd/io.systemd.Credentials

and

varlinkctl call /run/systemd/io.systemd.Credentials io.systemd.Credentials.Encrypt '{"text":"foobar"}'

The 2nd argument always specifies the entrypoint socket to talk to. In most cases where you call this locally that's the file system path of an AF_UNIX socket. However this can also be the path to an executable, in which case the executable is invoked and told via $LISTEN_FDNAMES (i.e.

2️⃣3️⃣ Here's the 23rd post highlighting key new features of the upcoming v257 release of systemd. #systemd257

In systemd, as mentioned in one of the previous installments, we are adopting the Varlink IPC at more and more places.

To interface with Varlink IPC from the command line we provide the "varlinkctl" tool. "varlinkctl introspect <socket>" for example introspects which method calls, types, and errors a service provides. Similar, "varlinkctl call <socket> <method> <json>" calls a method.

… connect to remote AF_UNIX socket, execute remote binary), are also accessible via sd-varlink.

Or in other words: it's now almost as easy to talk to a local Varlink socket as it is to talk to a remote one.

(This is super handy in combination with the AF_VSOCK support we provide for SSH'ing into local VMs, added in v256).

And that's it for today!

1️⃣9️⃣ Here's the 19th post highlighting key new features of the upcoming v257 release of systemd. #systemd257

A relatively basic feature of systemd's service management is the ability to automatically restart a service in case it terminates unexpectedly, configurable via the Restart= setting.

In v254 we added the RestartMode= setting that allows to fine tune the mechanism to use for restarting the service, i.e. it adds a logic to optionally avoid marking the service as failed between…

…named the same way for the lifetime of the system. Different definitions of "same" exist, i.e. some people prefer if the very same physical device always carries the same name (in which case the MAC address is a good identifier, if the device has one), others prefer if the slot the devices are plugged into gets the fixed name so that the devices can be replaced if broken.

The policy which naming scheme to use can be configured, see the systemd.net-naming-scheme(7) man page for details.

1️⃣7️⃣ Here's the 17th post highlighting key new features of the upcoming v257 release of systemd. #systemd257

Linux network interfaces since a longer time have not just a primary name, but any number of additional "alternative" names. Similar semantics apply, but they can be longer and there can be many.

You might know the naming policies that systemd by default applies when picking the (primary) name of hardware network interfaces, to ensure they remain stable, so that the same hw iface is...

But that got fixed in the kernel a while back. And thus with v257 we decided to drop the special casing again. So from now on all applicable predictable network interface names are applied by default to network interfaces, and the naming policy simply controls which one of them to use as primary name.

And that's all for today.

That was because in some situations MAC addresses can get "inherited" from one device to another. (Thus allowing multiple interfaces with the same mac.) Now, the kernel actually allowed to mark these cases so that we can recognize them and simply not apply alternative names bases on "inherited" MAC addresses.

Except of course that various drivers were broken in this regard, most recently some laptop docking stations which "inherited" their MAC address from the laptop's own ethernet port.

Since a while systemd would always follow that policy for the primary name, and then add all other candidate names as alternative names to the interfaces too.

That way, you can use all applicable names for the interfaces in tools that understand the alternative names concept of Linux, and for older tools you can use the primary name as per policy.

We only made one exception from this logic so far: the MAC-address based names (i.e. "enx…" and similar) would not be applied as alternative names.