Paul Cantrell
Still thinking about this post from @tess. (See the threaded follow-up for explanation!)
The One Central Thing that makes building software uniquely difficult, the thing that clears up so many “Can’t you just…” misunderstandings from nontechnical software stakeholders, is this:
Humans •interpret• instructions. Software •follows• them. 🧵 https://mastodon.social/@tess/113280252872087002
Paul Cantrell
Human communication is chock full of ambiguity. We rely on shared assumptions, on the listener filling in blanks. •Rely• on it! Feature? Flaw? Both, and regardless, it’s the nature of the beast.
When we humans hear instructions, when we do something, humans can use creativity, common sense, maybe even a moral compass and a capacity to imagine consequences. And mess up, and misunderstand. And misunderstand for the better, sometimes!
2/
Paul Cantrell
Programming is necessarily •unambiguous•, because it’s interpreted by a machine. When we write code, it means something — even if that meaning is “error” or “illegal input” — and that meaning is created mechanically.
(NB: Code might have “undefined behavior,” but that’s a judgement about human specification; the machine •will• do something with that code.)
3/
Paul Cantrell
Non-programmers tend to think that •syntax• is the hard part of programming, but it’s not. The hard part is dealing with unambiguous communication, watching a machine do •exactly what you told it to do• — no matter how wrong you were, or how little sense it makes, or how small the mistake. Nothing makes one feel as stupid as writing code! It’s sorcerer’s apprentice all day every day: fallibility come to life.
Computers make our imaginary ideas talk back to us, and our ideas surprise us.
4/
Paul Cantrell
We programmers, I think, tend to misunderstand the legal system, imagining that laws are like code — or that they •should• be like code, and the problem with them is that they’re insufficiently precise.
That’s wildly incorrect.
The whole •point• of the law is that it’s ambiguous and interpreted by humans. Justice Oliver Wendell Holmes: “The life of the law has not been logic: it has been experience.”
5/
Paul Cantrell
As much as laws have unintended consequences — sometimes that were contrary to intent, even consequences that •nobody• wanted! — we do have room for humans to wiggle within and around the law, to argue its meaning, to refine and sharpen its interpretation. That means courts reading the law, yes — and it also includes the creative workaround and/or malicious compliance of these city workers repaving a sidewalk.
Interpretive latitude in the law is a feature, not a bug.
6/
Paul Cantrell
(If you are still on a thing about how the code really should be like law, please, please imagine: would you trust the people who serve in Congress to write actual code and put it in production without being able to test it? Really?? “Live debugging” of the law is the only thing that keeps society based in the rule of law from collapsing!)
7/
Paul Cantrell
Part of the appeal of LLMs, I think, is that they may the tantalizing false promise of allowing humans to give instructions to computers expecting the same kind of common-sense interpretation we expect of humans.
Bad news, kids: it’s still a machine. •Somewhere• in the chain, it becomes unambiguous. And that moment where we cross the threshold from ambiguous, contextualized human interpretation to unambiguous machine instructions — •that• is the moment of programming. There live the demons.
8/
RommelRico
@inthehands
Computer: Your code is unambiguous
Also computer: NullPointerException
😜
Paul Cantrell
In code, it’s the programmer who crosses that threshold. We straddle two worlds: the messy world of human communication and human context, and the alien world of machine logic.
If you turn code into LLM prompting, who crosses that threshold? It’s not the prompt-writer, not really; they only shape it, like a middle manager talking to a programmer. It’s not the author of the LLM’s code either.
It’s a random number generator.
9/
Paul Cantrell
An LLM will •randomly• turn a prompt into some unambiguous machine behavior that has some probability of fitting the prompt. And — worse!— it reapplies that randomness afresh every time.
That’s not necessarily a bad thing. Randomness can be useful. Generating random possibilities can be useful.
But I don’t think that’s what people have in mind when they imagine replacing programming with generative AI.
10/
Paul Cantrell
Say what you will about software, but at least it’s consistent: right or wrong, at least it will follow the same instructions in the same unambiguous way.
Say what you will about the law, but at least it’s human: we constantly renegotiate it, reshape it as we live it, hot patch it as we follow it.
LLMs are neither of these things. I don’t think we have good intuitions about what they are. A lot of sloppy thinking is filling in the gaps, and investors are capitalizing on that.
/end
Paul Cantrell
@emenel
Indeed. Related: https://hachyderm.io/@inthehands/113227415258787221
emenel
@inthehands and data != context or understanding … something people seem to misunderstand these days.
ShadSterling
@inthehands I remember Amelia Bedelia being surprised that some people would “dust” their furniture - she would un-dust her furniture!
Paul Cantrell
@ShadSterling
Loved those books.
David Nash
@inthehands This reminds me of the (fortunately) brief time during the cryptocurrency bubble when a few techbros thought that “code as law” would be a workable concept. Even without the instances of “code as law” losing badly to “existing law as law”, there were way too many cases of the code doing exactly what code will do, and screwing its own designers over in ways that were completely predictable to non-techbros.
Paul Cantrell
@RommelRico
You jest, but NullPointerException was a major safety advancement over dereferencing null and getting whatever’s at memory address zero! Why “safety?” Because it made the machine’s rule (“you just can’t dereference null, period”) something that humans could mentally model and predict.
Sheldon
@inthehands every time I get asked if I'm nervous about some random technology replacing my skills as a developer, I laugh. The tech isn't the roadblock. The humans are. Until humans are better at humaning, I'll always have a job understanding what the humans really mean when they say they want this vs that.
I got into doing tech work an unusual way. I was a physical therapist before and I spent my days hearing people's theories on why their knee hurt. It was the weather. They sat in the wrong chair 10 years ago. They took the wrong supplement. It's because they wore shorts on a cold day... and then after all that I'd conclude they actually had a hip or a back problem or needed new shoes.
Figuring out what humans want with technology has actually been a lot easier than understanding humans well enough to help them understand how their own bodies work. I had no and still have no certifications, but leaned hard into my interviewing skills to gain an edge as a late starter.
Paul Cantrell
@sysop408 @tess
Yeah, bodies are absurdly complicated! Our computers are just little toys next to them.
Paul Cantrell
@partnumber2 @tess
Yes, and not just technical either. ALL jobs!
𝓓 𝓑 𝓒𝓸𝓸𝓹𝓮𝓻
The “Can’t you just…” mantra is NOT limited to SWE; it applies to all technical jobs. 👀
Paul Cantrell
@zachnfine
Richard Cook phrased this as “Actions at the sharp end resolve all ambiguity:” https://how.complexsystems.fail
If you think that insight means there’s no difference between what the average human does and what the best LLM does, I have some discounted swampland to sell you.
Zach Fine
@inthehands It's also the case that, if I ask someone to do something, at some point any ambiguity in my request is turned into non-ambiguous actions. It's no machine-human threshold. Differences of interpretation are normal in human communication.
What you've called a "tantalizing false promise" in the thread of expecting from LLMs "the same kind of common-sense interpretation we expect of humans", is a *true* promise. I expect and see LLMs making human-like coding mistakes all the time.
Paul Cantrell
@phaedral
As quoted upthread!
Robert Link
@inthehands “The life of the law has not been logic; it has been experience.”
Paul Cantrell
@Tak
I’m old enough to still think DAO means “data access object.”
Tak!
@inthehands DAOs were a great object lesson in "what if the laws actually were code‽"
(When looking for the wikipedia link, I was pleasantly surprised to find nothing web3-related in the first page of search results for "dao" - we did it, y'all!)
Paul Cantrell
@phaedral
Great minds think alike! (Other kinds too, but I’m it’s greatness at work here, for sure)
Robert Link
@inthehands Sorry I missed it!
Paul Cantrell
@forthy42
This is a bit too facile for me to agree. There’s a wide range of judges out there, and that range includes the best and the worst of people.
I think it’s healthier to see the legislative and judicial system as participants in whole-society negotiations over who we are and what kind of society we want. Anything haggled out in court would be haggled out by other means elsewhere, often by the same people — and not necessarily in a better way.
Bernd Paysan R.I.P Natenom 🕯️
@inthehands But that's what they do. The interpreters of the law are not the humans I would trust with that task. Judges are a self-selected set of people who think that law is just, particularly the way they interpret it. They often also think the way power is distributed now is just.
These people are the last ones that should be allowed to do that life debugging of the law. They are even worse than the politicians that make the law in the first place (and that's awful enough).
Simon Tatham
@inthehands conversely, there's a case where people expect code to behave like law. Namely, the people who want encryption systems to have backdoors accessible via govt paperwork like a court order.
The point they (perhaps deliberately) never understand is: you can't make the _cryptosystem_ verify a court order is legit. A human has to check that. And then they do something to the code. Which the code would obey whether or not the human had checked the court order, or even had one to check.
Paul Cantrell
@resuna
I can’t even count the iterations of “make programming accessible to everyone by changing the syntax” that I’ve seen in my lifetime.
Resuna
Twas always thus.
"Part of the appeal of COBOL, I think, is that it makes a tantalizing false promise of allowing humans to give instructions to computers in something like human language, expecting the same kind of common-sense interpretation we expect of humans."
My first job was in COBOL, I was still in high school, an intern, I knew nothing about business, and I didn't understand what the code meant. They basically had to undo everything I had done.
Paul Cantrell
@EpiphanicSynchronicity
I’m glad you found it worthwhile!
Epiphanic Synchronicity
@inthehands This whole thread is brilliant, Paul.
GNU social JP is a social network, courtesy of GNU social JP管理人. It runs on GNU social, version 2.0.2-dev, available under the GNU Affero General Public License.
All GNU social JP content and data are available under the Creative Commons Attribution 3.0 license.