Paul Cantrell
Agreed with everything @kevinriggle wrote here. Another angle on this to try with people who simply do not understand what software engineering •is•:
AI can generate code fast. Often it’s correct. Often it’s not, but close. Often it’s totally wrong. Often it’s •close• to correct and even appears to work, but has subtle errors that must be corrected and may be hard to detect.
Paul Cantrell
All the above is also true (though perhaps in different proportions) of humans writing code! But here’s the big difference:
When humans write the code, those humans are •thinking• about the problem the whole time: understanding where those flaws might be hiding, playing out the implications of business assumptions, studying the problem up close.
When AI write the code, none of that happens. It’s a tradeoff: faster code generation at the cost of reduced understanding.
2/
Paul Cantrell
The effect of AI is to reduce the cost of •generating code• by a factor of X at the cost of increasing the cost of •thinking about the problem• by a factor of Y.
And yes, Y>1. A thing non-developers do not understand about code is that coding a solution is a deep way of understanding a problem — and conversely, using code that’s dropped in your lap greatly increases the amount of problem that must be understood.
3/
Paul Cantrell
Increase the cost of generating code by a factor of X; increase the cost of understanding by a factor of Y. How much bigger must X be than Y for that to pay off?
Check that OP again: if a software engs spend on average 1 hr/day writing code, and assuming (optimistically!) that they only work 8 hr days, then a napkin sketch of your AI-assisted cost of coding is:
1 / X + 7 * Y
That means even if X = ∞ (and it doesnt, but even if!!), then Y cannot exceed ~1.14.
Hey CXO, you want that bet?
4/
Paul Cantrell
This is a silly thumbnail-sized model, and it’s full of all kinds of holes.
Maybe devs straight-up waste 3 hours a day, so then payoff is Y < 1.25 instead! Maybe the effects are complex and nonlinear! Maybe this whole quantification effort is doomed!
Don’t take my math too seriously. I’m not actually setting up a useful predictive model here; I’m making a point.
5/
Paul Cantrell
Though my model is quantitatively silly, it does get at the heart of something all too real:
If you see the OP and think it means software development is on the cusp of being automatable because devs only spend ≤1/8 of their time actually typing code, you’d damn well better understand how they spend the other ≥7/8 of their time — and how your executive decisions, necessarily made from a position of ignorance* if you are an executive, impact that 7/8.
/end (with footnote)
Paul Cantrell
* Yes, executive decisions are •necessarily• made from a position of ignorance. The point of having these high-level roles isn’t (or at least should not be) amassing power, but rather having people looking at things at different zoom levels. Summary is at the heart of good management, along with the humility to know that you are seeing things in summary. If you know all the details, you’re insufficiently zoomed out. If you’re zoomed out, you have to remember how many details you don’t know.
Paul Cantrell
@rgarner @RuthMalan
Yup. All that.
And Brooks’s maxim that there is no silver bullet still stands undefeated.
Russell Garner
@inthehands @RuthMalan every dev wants a greenfield project. LLMs shade even greenfield projects brown.
But then it's not the devs that are asking for this* so much as a managerial class looking for the sort of silver bullet that brings down both pay and the amount of time dealing with a type of worker they find difficult.
*not the ones who are any good, anyway
Jeff Miller (orange hatband)
@inthehands I appreciate how your line of argument chimes with Fred Brooks' "No Silver Bullet", along the lines of essential complexity of the problem and the solution matching up, except the comparison here being the embedded understanding (hopefully) in code written specifically for the problem, versus the embedded unchecked assumptions in generated code.
Novel software libraries have a similar problem: easy to adopt, not necessarily easy to evaluate.
Jeff Miller (orange hatband)
@inthehands <3 NSB
Paul Cantrell
Yes, this from @sethrichards is what I’m talking about:
https://mas.to/@sethrichards/113642032055823958
I had a great moment with a student the other day dealing with some broken code. The proximate problem was a Java NullPointerException. The proximate solution was “that ivar isn’t initialized yet.”
BUT…
Paul Cantrell
…It wasn’t initialized because they weren’t thinking about the lifecycle of that object because they weren’t thinking about when things happen in the UI because they weren’t thinking about the sequence of the user’s interaction with the system because they weren’t thinking about how the software would actually get used or about what they actually •wanted• it to do when it worked.
The technical problem was really a design / lack of clarity problem. This happens •constantly• when writing code.
Paul Cantrell
A good point from @rgarner, looking at this through the lens of Brooks’s Law:
https://mastodon.social/@rgarner/113642040777621582
Paul Cantrell
@n_dimension
Please note the hypothetical downthread where I ask what happens if AI transcends human coding performance by a factor of infinity.
For your line of reasoning to work, this thing we call AI has to do something that LLMs simply do not do, do not approach doing, are not on a trajectory toward doing, and are fundamentally architecturally incapable of doing. It’s like saying “eventually if this car goes fast enough, it will be a time machine!”
𝕎𝕦𝕝𝕗𝕪
You are 100% correct...
...right now.
But consider this, upper limits of AI transcend human performance constraints.
One of the possible paths for AI development is you have all these multiple agents working in orchestra.
One is responsible for overall architecture.
One for environmental impact.
Etc etc.
Thought provoking post.
Thanks
Dr Andrew A. Adams #FBPE 🔶
@inthehands
Coding is teaching a really, really dumb student how to solve a problem.
Teaching something is the best way to understand it properly.
Raphael
@inthehands Oh, why hello, Amdahl!
Stu
@inthehands kickers are only on the field for a few minutes per game, fire them and make AI punt.
Paul Cantrell
@kevinriggle
Yeah, I’ve heard that thought too. It’s tantalizing nonsense. I could write about this at length, and maybe one day I will, but the very short version is that automation is not even remotely the same thing as abstraction.
Kevin Riggle
@inthehands The bet that a lot of these CXOs are making implicitly is that this will be like the transition from assembly to higher-level languages like C (I think most of them are too young and/or too disconnected to make it explicitly). And I'm not 100% sold on it but my 60% hunch is that it's not.
Paul Cantrell
@OmegaPolice
That’s it: Amdahl’s Law law except optimization actually creates large costs in the other parts of the system!
Paul Cantrell
@kevinriggle
That’s exactly the line of thought, yes. And the thing that makes abstractions useful, if they are useful, is that they make good decisions about what doesn’t matter, what can be standard, and what requires situation-specific thought. Those decisions simultaneously become productivity boost, safety, and a metaphor that is a tool for thought and communication.
Kevin Riggle
@inthehands Yes! Yes. This is it exactly.
One can imagine a version of these systems where all the "source code" is English-language text describing a software system, and the Makefile first runs that through an LLM to generate C or Python or whatever before handing it off to a regular complier, which would in some sense be more abstraction, but this is like keeping the .o files around and making the programmers debug the assembly with a hex editor.
Paul Cantrell
@kevinriggle
What happens when the semantics of your model are defined by probabilistic plagiarism, and may change every single time you use it? That might be good for something, I guess??? But it doesn’t remotely resemble what a high-level language does for assembly.
Paul Cantrell
@kevinriggle
Yeah, one of the troubles with the systems is that basically every metaphor we can think of for what they are is misleading.
Kevin Riggle
@inthehands Another bull-case argument about LLMs is that they are a form of autonomation (autonomy + automation), in the sense that the Toyota Production System uses it, the classic example being the automated loom which has a tension sensor and will stop if one of the warp yarns break. But we already have many such systems in software, made out of normal non-LLM parts, and also that's ... not really what's going on here, at least the way they're currently being used.
Paul Cantrell
@kevinriggle
Even if that gave semantically staple answers, which I’m not convinced it would, it’s still skips that all important step where there’s communication and reflection and consensus building.
I suppose there’s some help in approaches where and LLM generates plausible answers and then some semantically reliable verification checks that the results aren’t nonsense. But we’re really stretching it.
Kevin Riggle
@inthehands One could imagine using a fixed set of model weights and not retraining, using a fixed random seed, and keeping the model temperature relatively low. I'm imagining on some level basically the programming-language-generating version of Nvidia's DLSS tech here. But that's not what people are doing and I'm not convinced if we did that it would be useful
Eleanor Saitta
@kevinriggle
The primary job of a development team is the creation and maintenance of a shared mental model of what the software does and how it does it. Periodically, they change the code to implement changes to the mental model that have been agreed upon, or to correct places where the code does not match the model. An LLM cannot reason and does not have a theory of mind and as such cannot participate in the model process or meaningfully access that model — written documentation is at best a memory aid for the model — and thus cannot actually do anything that matters in the process. The executive class would prefer that other people in the org not be permitted to think, let alone paid for it, and therefore willfully confuses the output with the job.
@inthehands
Paul Cantrell
That’s all well said, and gets to what @jenniferplusplus was talking about here: https://jenniferplusplus.com/losing-the-imitation-game/
Shafik Yaghmour
What I am seeing from studies is pretty troubling e.g. https://hachyderm.io/@shafik/113391494588227091
Folks feel more productive but actual objective measures say otherwise.
It is never about lines of code produced that is a silly measure by itself.
It is really about how many errors make it into production. If using AI code generators means a lot more bad code makes into production you are making a very poor tradeoff.
Paul Cantrell
@lwriemen
This is an overly pedantic quibble, though I agree with the underlying sentiment that people rush into coding too fast without thinking.
Doing the work of filling the things left between the lines for the reader to infer above: coding something •while thinking• — assessing the results, letting the ideas talk back and surprise, treating design and implementation problems as prompts to think about goals and context — is a deep way of understanding a problem.
Old Fucking Punk
@inthehands My 30+ years as a dev disagree with the statement, "coding a solution is a deep way of understanding a problem", on semantics. Analyzing a solution is a deep way of understanding a problem. I've found that too many devs are in a rush to code and give short shrift to the analysis.
Rich Felker
@sroccaserra @inthehands @dymaxion @kevinriggle @jenniferplusplus This is absolutely what it is and what we've been telling folks it is for years. Of course they won't listen. See: my display name.
Sébastien Roccaserra 🐿️
@inthehands @dymaxion @kevinriggle @jenniferplusplus This is interesting. I have another concert: what if the goal of the GenAI narrative was mostly an accounting goal? Meaning, tech is / was worth investing in because the huge promise of future growth enables to boost the balance sheet & enables to pay nice dividends every year. So you have to regularly renew the narratives that justify promises of future growth. GenAI is just that, it doesn't even *need* to be a real productivity boost.
Paul Cantrell
@dalias @sroccaserra @dymaxion @kevinriggle @jenniferplusplus
A shockingly large portion of the world’s current problems trace back to an excess of investment money desperate for more things to invest in.
Jonathan Yu
@inthehands @dalias @sroccaserra @dymaxion @kevinriggle @jenniferplusplus I'm not sure if the problem is excess money or an inefficient market (lots of dumb ideas get lots of funding while lots of good ideas don't.) Or maybe the problem is that we have lots of money but we're insufficiently creative with respect to how we allocate it?
Probably, an issue is that people with control of capital chase returns for their own sake, instead of being a byproduct of doing something useful
Rich Felker
@jawnsy @inthehands @sroccaserra @dymaxion @kevinriggle @jenniferplusplus The problem is excess money combined with wanting obscene returns on it, and without driving down the value of other holdings. This necessitates perpetual scams.
Rich Felker
@jawnsy @inthehands @sroccaserra @dymaxion @kevinriggle @jenniferplusplus For example a prosocial business activity to do with obscene amounts of money would be building ultra affordable housing at scale and renting or selling it at minimal profit. At scale this would still be very profitable. But it would drive down value of the billionaire class's assets.
GNU social JP is a social network, courtesy of GNU social JP管理人. It runs on GNU social, version 2.0.2-dev, available under the GNU Affero General Public License.
All GNU social JP content and data are available under the Creative Commons Attribution 3.0 license.