Notices by Zack Weinberg (zwol@masto.hackers.town)

@lanodan @alanc @jmc Yeah, my brief time teaching was much later and all the students were on regular Linux (glibc, not musl). Though, if the Sun clade has had "Arithmetic exception" forever and musl uses it too, that's all the excuse I need to go change glibc myself :)

A tiny hill I will die on: It is *wrong* to print out numeric errno codes. If you feel it is necessary to print a mysterious code in addition to the human-readable message you get from strerror(), print the Exxx name. Yeah, C doesn't give you any good way to do that. Maybe you don't need to print a mysterious code at all?

You know what you *should* print, though? The name! of the thing! you couldn't do something with! (usually but not always a filename)

@dalias No excuse will be accepted. Databases have done fully arbitrary transactions since the 1960s. It _cannot_ be as hard as everyone on the filesystem side of the fence keeps insisting.

there was a big long thread about shells and filesystems that @hipsterelectron tagged me on a few days ago and i don't have useful thoughts on most of it but:

there's two things that are unambiguously bad about file systems as we know them. well, more than two, but these are the ones that seem most directly relevant.

* no transactions
* no scope

(1/n)

"no transactions": I mean "transactions" exactly the way database people mean it. you should be able to stack up an arbitrarily large set of changes to files, both data and metadata, and then tell the OS "either make _all_ of this happen permanently, or _none_ of it."

i'm not the first person to say this, people have been saying it for _decades_. the way it would have to work is well understood. FS devs who say it's too hard to implement need to git gud.

(2/n)

@elilla this makes me wonder if maybe people *think* second language acquisition should work by drilling and formal grammar analysis because that's the part of their education in their first language that happened after they were old enough to remember it consciously.

@lanodan zpool scrub *does not* repair, or even validate, the on-disk data structure. It only checks the block checksums. I have actually been burnt by this.

@lanodan see take #29

@ska @JdeBP I never bothered looking at sudo-rs because there just isn't any point unless you're going to replace the config file format. I'd bet actual money that at least an order of magnitude more servers have been compromised because of mistakes writing /etc/sudoers than because of any problem with sudo's *code*.

As for ssh, this conversation is reinforcing my desire to do some actual experiments. We've all seen what we've seen, but that's only good enough for hypothesis generation.

28. Use a boring, reliable file system. Ideally, use a boring, reliable file system written by people who understand that file systems exist to serve the needs of applications -- but I'm not sure there *is* any such file system! (see https://wiki.postgresql.org/wiki/Fsync_Errors )

29. Do not use any file system that doesn't have an offline consistency check and repair tool (fsck), no matter how boring and reliable it otherwise appears to be (yeah, ZFS, I'm talking about you)

23. You should _not_ be backing up the OS. Instead, you should be prepared to recreate the server from scratch at any moment, and then restore the latest data backup onto the new machine. That's what the version controlled system configuration is for.

24. It's an incredible pain in the ass to set up, but consider setting up block-level integrity protection (dm_integrity on Linux) as your lowest layer of storage management. This is especially valuable with any form of RAID, because it fixes the problem where, if a RAID stripe is self-inconsistent, you don't know which copy of the stripe is the good one.

(Disk encryption includes integrity protection; you don't need both. Just make sure the encryption is under the RAID.)

25. If you're building or specifying a server from the hardware on up: get the ECC RAM. Actually do this no matter what kind of computer you're building.

26. Some security hardening tips that don't appear in most security hardening guides:

* Disable core dumps.
* On 64-bit machines, disable 32-bit executable compatibility.
* Enlarge the size of the NULL pointer guard region at the bottom of the address space. 4194304 (4MiB) is safe. Higher is more effective as a security measure (I'd *like* to set this to 2**32!) but higher breaks non-PIE executables.

27. Linux's defaults for how much RAM can get filled up with "dirty pages" (data that needs to get written to persistent storage Real Soon Now), before it actually starts doing writeback, are *way* too high for how much RAM modern computers have. Set both `vm.dirty_background_bytes` and `vm.dirty_bytes` to no more than a few tens of megabytes.

20. However, you should make sure you _do_ have 'dig', 'ping', 'traceroute', 'tcpdump', 'lsof', and 'strace' installed. These are essential troubleshooting tools, and some of them are very useful in situations where the network is busted and you _can't_ download them on the fly.

21. This is more of a "Unix shell usability" hot take than a "sysadmin" hot take, but did you know you can embed newlines in your shell prompt? PS1='\u@\h\n\w\n\$ ' is significantly more ergonomic than the usual PS1='\u@\h \w \$ '.

22. You gotta have automated data backups. You knew that already. But: those data backups should be going to a machine in a *different data center* than the one where the server itself is. Preferably, one run by a different organization.

18. You need an NTP daemon. For people who aren't running a time _server_, 'chrony' currently appears to be the least bad NTP daemon on the market.