Jan Schaumann
Advanced #Programming in the #UNIX Environment
Week 6, The Environment
We're going to take a look at the process environment, using what we learned about the process layout in memory to understand how the environment variables are stored and, if necessary, moved around. We'll also get a quick look at what malloc(3) does.
Jan Schaumann
Advanced #Programming in the #UNIX Environment
Week 7, Reentrant and Interrupted Functions
We learn that only functions guaranteed to be async-signal-safe can safely be used in signal handlers (see sigaction(2)), as well as what happens when we are interrupted while blocking on certain I/O. (Note: some Unix versions provide reentrant "_r" versions of otherwise non-reentrant functions (see e.g., ctime_r(3)). Both are distinct from thread-safe functions.)
Jan Schaumann
Advanced #Programming in the #UNIX Environment
Some general advice on good coding style and readability:
The happy path is left-aligned:
https://medium.com/@matryer/line-of-sight-in-code-186dd7cdea88
Return Early Pattern:
https://medium.com/swlh/return-early-pattern-3d18a41bba8
Stanford CS106 Coding Style
https://web.stanford.edu/class/cs106b/resources/style_guide.html
I'm a Never-Nester and you should, too"
https://mstdn.social/@jschauma/109559000445536685
Jan Schaumann
Advanced #Programming in the #UNIX Environment
Week 7, Signals
Having seen how job control in the #shell often uses signals to communicate between process groups, let's take a closer look at these simple, asynchronous event notifications. We run through a number of examples to illustrate how signals are delivered, can be ignored, caught, handled, or blocked.
This video in a single slide:
Jan Schaumann
Advanced #Programming in the #UNIX Environment
Week 7, Job Control
We illustrate the concept of job control in the shell, first introduced in the C #shell, and allowing you to run multiple tasks from within the same terminal, switching back and forth between them by placing them into the background, suspending them, or bringing them to the foreground. It's one of my all-time favorite productivity hacks - Ctrl+Z FTW!
Jan Schaumann
Advanced #Programming in the #UNIX Environment
Week 7, Process Groups and Sessions
We look at how processes are grouped together and begin to develop an understanding of how a login session relates to the controlling terminal:
• each process belongs to a process group
• a session is a collection of one or more process groups
• process groups are used for distribution of (keyboard generated) signals and to implement e.g., job control in a shell (see next video)
Jan Schaumann
Advanced #Programming in the #UNIX Environment
Week 7, The Login Process
In this week, we begin our discussion of process relationships, including process groups, sessions, and our first, asynchronous type of interprocess communication in the form of signals. To get us started, we first look at how processes created during the normal Unix boot process relate to one another. (And no, we will _not_ discuss #systemd. I said *normal* boot process.)
Jan Schaumann
Advanced #Programming in the #UNIX Environment
Week 6 related link: #Linux x86 Program Start Up or - How the heck do we get to main()?
Compare to what we covered in Segment 2 of this week.
http://dbp-consulting.com/tutorials/debugging/linuxProgramStartup.html
Jan Schaumann
Advanced #Programming in the #UNIX Environment
Week 6, Process Control
In the last video lecture for this week, we look at process control: how new processes are started from an executable, what resources are shared between parent and child, and what happens when they terminate. In particular, we will look at the fork(2), exec(3), and wait(2) system calls. Be warned, though: there will be #zombies, so limber up (Rule #7).
Jan Schaumann
Advanced #Programming in the #UNIX Environment
Week 6, Process Limits and Identifiers
Having looked at the process layout in memory, how it starts and is terminated, we now learn that certain properties are restricted via resource limits, specified as a "soft" and a "hard" limit, with only the superuser being able to raise the latter.
A process also has a process ID (PID) and a parent process ID (PPID). More
on these process relationships in our next videos.
Jan Schaumann
Advanced #Programming in the #UNIX Environment
Week 6, The Environment (a tangent from 2011)
How large can the environment get? Is there a limitation on the size of a single environment variable? And why did I sometimes encounter this error message:
sudo: unable to execute <command>: success
Jan Schaumann
Advanced #Programming in the #UNIX Environment
The #NetBSD operating system comes with a number of historical Unix research papers found in /usr/share/doc: Marshall Kirk McKusick on the Fast File System, Robert Morris and Ken Thompson on Password Security, a shell tutorial by Stephen R. Bourne, a guide to using vi(1) by Bill Joy, and the well known BSD IPC tutorials (linked below) that we used in our last videos!
https://stevens.netmeister.org/631/ipc.pdf
https://stevens.netmeister.org/631/ipctut.pdf
Jan Schaumann
Advanced #Programming in the #UNIX Environment
Week 9, I/O Multiplexing using select(2)
We complete our discussion of interprocess communication and move from one-to-one communications towards a more typical client-server model with a server side process capable of handling multiple simultaneous clients. We accomplish this by using the select(2) syscall to handle synchronous I/O multiplexing.
Jan Schaumann
Advanced #Programming in the #UNIX Environment
Week 9, socket(PF_INET6, SOCK_STREAM, 0)
In this video, we demonstrate the use of STREAM sockets in the INET6 domain, meaning we'll use TCP to establish a sequenced, reliable, two-way byte stream over an #IPv6 network. We use our good friend #tcpdump to inspect the packets on the wire, observing the TCP three-way handshake, data being pushed, the connection tear down or reset (RST) packets from the server.
Jan Schaumann
Advanced #Programming in the #UNIX Environment
Week 9, socket(PF_INET, SOCK_DGRAM, 0)
In this video lecture, we show how to communicate between hosts across the internet using datagram sockets in the PF_INET domain (i.e., #UDP). We also observe the packets as they are sent by using the #tcpdump utility.
Jan Schaumann
Advanced #Programming in the #UNIX Environment
Week 9, socket(PF_LOCAL, SOCK_DGRAM, 0)
In this segment, we introduce the #sockets API by an example of datagram connections in the PF_LOCAL (formerly AF_UNIX) domain.
These “Unix domain sockets” appear in the filesystem (S_ISSOCK), but serve only as rendezvous point. I/O can be done using read(2)/write(2) or recv(2)/send(2).
Jan Schaumann
Advanced #Programming in the #UNIX Environment
Week 9, socketpair(2)
We continue our discussion of IPC with a look at the socketpair(2) system call and compare its functionality with that of the pipe(2) call. We also introduce the concept of sockets, domains, and protocols.
Jan Schaumann
Advanced #Programming in the #UNIX Environment
Week 8, Pipes and FIFOs
With this video lecture, we continue our discussion of Interprocess Communication and dive into two of the oldest and most ubiquitous forms of Unix #IPC: pipes (the basis of the Unix Philosophy) and FIFOs. We also discuss popen(3) and the dangers of shelling out to unverified commands.
Jan Schaumann
Advanced #Programming in the #UNIX Environment
Week 8, System V Interprocess Communication
In this video lecture, we cover traditional #SystemV #IPC: semaphores, shared memory, and message queues. We also look at the POSIX message queues implementation and how this model forms the basis of many popular "pub sub" services like e.g., Amazon Simple Queue Service, Apache Kafka, or RabbitMQ.
Jan Schaumann
Advanced #Programming in the #UNIX Environment
Week 8, Interprocess Communications Intro
We begin our discussion of Interprocess Communication by providing a quick overview of the different properties of #IPC mechanisms as well as which mechanisms we will cover:
- Signals (see previous week)
- System V IPC (Semaphores, Shared Memory, Message Queues)
- Pipes and FIFOs
- socketpairs and sockets
Jan Schaumann
Advanced #Programming in the #UNIX Environment
Week 12, Resource Locking
In this video lecture, we discuss "advisory" resource- and record locking between cooperating processes using flock(2), lockf(3), and fcntl(2).
(We will look at many other methods of restricting (non-cooperating) processes in more general terms in the next week.)
Jan Schaumann
Advanced #Programming in the #UNIX Environment
Week 12, Non-blocking I/O
In this video lecture, we briefly cover non-blocking I/O, whereby a system call that would normally block, waiting for I/O, returns immediately with an errno of EAGAIN or EWOULDBLOCK. This reaches back into some previous lectures, such as Week 7's discussion of interrupted and reentrant functions.
Jan Schaumann
Advanced #Programming in the #UNIX Environment
Week 12, syslogd(8)
This week we are covering a number of small, miscellaneous topics. We start with a look at the standard Unix central logging facility, syslogd(8), illustrated below, and the syslog(3) library function. We show examples of system logs and how to separate messages by priority and level.
Jan Schaumann
Advanced #Programming in the #UNIX Environment
Additional reading relating to shared libraries and ELF:
Tool Interface Standard (TIS) Executable and Linking Format (ELF) Specification:
https://refspecs.linuxbase.org/elf/elf.pdf
How to write shared libraries:
https://www.akkadia.org/drepper/dsohowto.pdf
Ian Lance Taylor's 20 part linker posts:
https://lwn.net/Articles/276782/
Jan Schaumann
Advanced #Programming in the #UNIX Environment
Week 11, Shared Libraries
In this video, we cover shared libraries themselves. We'll discuss just what exactly a shared library is, how you create one, and how we can influence the behavior of dynamically linked executables due to the way the link-editor works by, e.g., setting the LD_LIBRARY_PATH variable or using the "-rpath" flag for the linker.
Jan Schaumann
Advanced #Programming in the #UNIX Environment
Week 11, Of Linkers and Loaders
Building on top of what we discussed in Week 5 in the context of the compiler chain, we take a look at how a relocatable object file is turned into an executable by the dynamic linker (ld) and how an executable is loaded into memory by the run-time link-editor (ld.so or ld.elf_so).
Jan Schaumann
Advanced #Programming in the #UNIX Environment
Week 11, The Executable and Linkable Format
We begin our exploration of shared libraries. We start with a look at the Executable and Linkable Format (#ELF) for binary files such as executables, object files, core files and shared libraries. We'll use the hexdump(1) and readelf(1) utilities to better understand the format.
Jan Schaumann
Advanced #Programming in the #UNIX Environment
Week 10, Daemon processes
We take a quick look at processes intended to run continuously and in the background: daemon processes. We'll also brush upon system start scripts and daemon/service conventions.
Jan Schaumann
Advanced #Programming in the #UNIX Environment
Side-quest: IPC Buffer Sizes
In our discussion of IPC, we noticed that each form must use some sort of buffer. But what is the actual size of those buffers? Is it the same across #NetBSD, #Linux, #FreeBSD and other Unix versions?
Jan Schaumann
Advanced #Programming in the #UNIX Environment
Week 13, eUIDs, file flags, mount options, securelevels
We look at new ways to restrict processes and even root itself. We revisit how we change effective UIDs (see week 3) using su(1) and sudo(8), and then cover file flags, certain mount options, and BSD securelevels.
Jan Schaumann
Advanced #Programming in the #UNIX Environment
Week 13, POSIX.1e ACLs
In this week, we look at the various ways in which processes can be restricted from impacting one another, beginning with methods we've already discussed to some degree (e.g., unix file access semantics, resource limits) and ultimately leading up to #containers.
In the first video, we show how POSIX.1e Access Control Lists (ACLs) can be used for more fine-grained file system access control.
Jan Schaumann
Advanced #Programming in the #UNIX Environment
Coding mistakes using strncat(3) are common - better: use strlcat(3).
Jan Schaumann
Advanced #Programming in the #UNIX Environment
Week 12, Asynchronous and Memory Mapped I/O
We conclude our series on a set of mixed advanced I/O topics with a quick summary of asynchronous (aio(7)) and memory mapped I/O (mmap(2)).
GNU social JP is a social network, courtesy of GNU social JP管理人. It runs on GNU social, version 2.0.2-dev, available under the GNU Affero General Public License.
All GNU social JP content and data are available under the Creative Commons Attribution 3.0 license.