Paul Cantrell
I once worked at a company that sold industry-specific core-business software to deep-pocketed corps who couldn’t / wouldn’t / shouldn’t roll their own. I got into a discuss with my manager about whether our products were essentially — my words — a hoax.
Me: “Look, our products are riddled with bugs and holes. They’re nearly impossible to deploy, manage, and maintain. They frequently don’t even work •at all• on the putative release date, and we sell the mop-up as expensive ‘consulting.’”
1/
Paul Cantrell
“How can it not be a hoax?!”
He said something that completely changed how I look at the workings of business:
“Paul, you are making the mistake of comparing our software to your ideal of what it •should• be. That’s not what these companies are doing. They’re comparing it to what they already have now. And what they have now is •terrible•.
2/
Paul Cantrell
He continued: “They’re doing this with Excel spreadsheets, or ancient mainframes, or in many cases still using pen and paper processes [this was the early 00s], and those processes are just wildly labor-intensive and error-ridden. They lose unimaginable amounts of money to this. For them to pay us a measly few million to get software that takes 18 months to get deployed and just barely working? That is a •huge• improvement for them.”
In short: our product sucked, but it wasn’t a hoax.
3/
Paul Cantrell
There’s a weird disconnect about gen AI between the MBA crowd and the tech crowd: either it’s the magical make-money sauce CEOs can just pour on everything, or it’s fake and it’s all a hoax.
A lot of that is just gullibility and hype at play, huge amounts of investor money and wishful thinking desperately hoping to find huge payoffs in whiz-bang tech.
But: companies do actually deploy gen AI, and it sucks, and they •don’t stop•. Why?!
4/
Paul Cantrell
I suspect that conversation long ago might shed some light on how companies are actually viewing gen AI right now. Behind all the flashy “iT cOuLD bE sKYnEt” nonsense, there’s something much more disappointingly cynical but rational: Gen AI sucks. They know it sucks. But in some cases, in some situations, viewed through certain bottom-line lenses, it sucks slightly less.
5/
Paul Cantrell
So Megacorp’s new AI customer support tool describes features that don’t exist, or tells people to eat nails and glue, or is just •wrong•.
Guess what? Their hapless, undertrained, poverty-wage, treated-like-dirt humans who used to handle all the support didn’t actually help people either. They demanded throughput so high and incentivized ticket closure so much that their support staff were already leading people on wild goose chases, cussing them out, and/or quitting on the spot.
6/
Paul Cantrell
Gen AI doesn’t cuss people out, doesn’t quit on the spot, and has extremely high throughput. It leads people on wild goose chases •far• more efficiently than the humans. And hell, sometimes, just by dumb luck, it’s actually right! Like…maybe more than half the time!
When your previous baseline is the self-made nightmare of late stage capitalism tech support, that is •amazing•.
7/
Paul Cantrell
And you can control it (sort of)! And it protects you from liability (maybe)! And all it takes is money and environmental disaster!
Run that thought process across other activites where corps are deploying gen AI.
I suspect a lot of us, despite living in this modern corporate hellscape, still fail to understand just how profoundly •broken• the operations of big businesses truly are, how much they function on fakery and deception and nonsense.
So gen AI is fake? So what. So is business.
8/
Jeff Miller (orange hatband)
@inthehands oh ouch you just hit me in (current job) better than Excel and (previous job) automated response substituting when it should be augmenting human support agents.
Paul Cantrell
I am hamming this up for cynical dramatic effect, but I do think there’s a serious thought here: so much activity within business delivers so little of actual value to the world that replacing slow human nonsense crap with fast automated nonsense crap seems like a win.
Trying to imagine the world through MBA goggles on, it seems perfectly rational.
When people consider gen AI, I ask them to ask themselves: “Does it matter if it’s wrong?” Often, the answer is “no.”
9/
Paul Cantrell
If you’ll indulge another industry story — sorry, this thread is going to get absurdly long — let me tell you about one of the worst clients I ever had:
Group of brothers. They’d made fuck-you money in marketing or something. They founded a startup with a human benefit angle, do some good for the world, yada yada.
Common now, but new-ish idea at the time: gamified online health & well-being platform that a company (or maybe insurer, whatever) offers to its employees.
10/
Paul Cantrell
The big brilliant idea at the heart of the product they were building? The Life Score: a number that quantifies your overall well-being, a number that you can try to raise by doing healthy activities.
How exactly was this number calculated? Eh, details.
11/
Paul Cantrell
They had this giant, elaborate business plan: the market opportunity, the connections, the moving parts — and in the middle of this giant world-domination scheme, a giant hole. Just black box (currently empty) labeled “magic number that makes people get healthier.”
The core feature of their product, the lynchpin that would make the entire thing actually useful, was just a giant-ass TBD.
12/
Paul Cantrell
I was hired to implement, but quickly realized they had no idea what they wanted me to build. Worse: they hadn't hired any of the people (like, say, a health actuary or a behavioral psychologist) who would be remotely qualified to help them figure it out. The architect of their giant system was a chemical engineer of some kind who was trying to get into tech. Lots of big ideas about what it would •look like•, but nobody in sight had a clue how this thing would actually •work•. Zero R&D.
13/
Paul Cantrell
No worries. Designers were cranking out UI! Marketers were…marketing! Turning the Life Score from vague founder notion to working system was a troublesome afterthought.
So…like a fool, I tried to help them suss it out. It turned out they •did• sort of have a notion:
1. Intake questionnaire about your lifestyle
2. Assign points to responses
3. System suggests healthy activities
4. Each activity adds points to your score if you do it
14/
Paul Cantrell
And then, like a •damn• fool, I pointed out to them the gaping chasm between (2) and (4). Think about it: at the start, the score measures (however dubiously) the state of your health. But after you do some activities, the score measures how any activities you did.
The score •changes meaning• after intake. And it's designed to go up.
And like an •utter• damn fool, I thought this was a flaw.
15/
Paul Cantrell
It was only after the whole contract crashed and burned (they were, it turns out, truly awful people) that I realized that my earnest data-conscious questions were threatening their whole model.
Their product was there to make the “healthy” line go up. Not to actually make people healthy, no! Just to make the line go up.
It was an offer of plausible deniability: for users, for their employers, for everyone. We can all •pretend• we’re getting healthier! Folks will pay good money for that.
16/
Paul Cantrell
Of •course• their whole business plan had a gaping hole at the center. That was the point! If that Life Score is •accurate•, it wrecks the whole thing.
Now, of course, there would be no Paul to ask them annoying questions about the integrity of their metrics. They’d just build it with gen AI.
17/
Paul Cantrell
Would gen AI actually be a good way to help people get healthy with this product? No. But that was never the goal.
Would gen AI have been a good option for these rich people trying to get richer by building a giant hoax box that lets a bunch of parties plausibly claim improved employee health regardless of reality? Hell yes.
18/
William Pietri
@inthehands Agreed, and there's another level of fakery here that interests me. I suspect a bunch of the corporate "AI" projects are just taking advantage of the hype wave to rebuild something that needed rebuilding. That key people know the "AI" benefit is zero, but it's the only way to get the rest of the project done.
Paul Cantrell
Again, my gen AI question: Does it matter if it’s wrong?
I mean, in some situations, yes…right? Like, say, vehicles? that can kill people?
Tesla’s out there selling these self-crashing cars that are •clearly• not ready for prime time, and trap people inside with their unopenable-after-accident doors and burn them alive. And they’re •still• selling crap-tons of those things.
If it doesn’t matter to •them•, how many biz situations are there where “fake and dangerous” is 100% acceptable?
19/
Paul Cantrell
Does it matter if it’s wrong?
In the nihilism of this current stage of capitalism, “no” sure looks like a winning bet.
/end
Paul Cantrell
@tehstu
That’s kind! My classroom lectures / discussions are much more cheerful: lots more “let’s be excellent to one another and make some cool things.”
Cedric Rossi
Very interesting! @pluralistic said that AI works for ”low-stakes low-value tasks” like ”political disinformation, spam, fraud, academic cheating, nonconsensual porn, dialog for video-game NPCs” but that ”none of them seem likely to generate enough revenue(…)to justify the billions spent(…)nor the trillions in valuation(…)” and that there are pbly no ”low-stakes, high-value tasks”
[https://pluralistic.net/2024/04/01/human-in-the-loop/#monkey-in-the-middle]
But maybe you’re right and most business is indeed low-stake, high-value🤔
Jeremy D. Miller
@inthehands You're giving me so many flashbacks -- and not the good time -- to being an internal architect in "Big IT" in the early 00's
Nicole Parsons
Extend this thought experiment to political campaign funding and tech billionaires.
Silicon Valley bought an election win for a set of GOP crooks because "innovation" has been redefined as:
1. Successful scams & frauds
2. Tax evasion
3. Corporate welfare & subsidies
4. Monopolies
5. Regulatory capture
6. Pollution & climate denial
7. Deregulation
Silicon Valley does not want saleable products that generate revenue.
They want Saudi cash. They want Russian oligarchs...
1/2
Nicole Parsons
2/2
...money laundering through their VC's and hedge funds.
They want their cut off Chinese IP theft & ubiquitous surveillance capabilities.
They want to preserve patriarchy & white supremacy, plus the wealth it generates for them.
https://whatever.scalzi.com/2024/02/21/the-big-idea-cory-doctorow-4/
https://www.wired.com/story/the-internet-con-cory-doctorow-book-excerpt/
https://doctorow.medium.com/https-pluralistic-net-2024-07-31-greater-fools-coinbased-7398817ecda5
Paul Cantrell
@Npars01
Yep, this is all at the heart of it. It’s the same thing that brought us the 2008 crash: too much investor money looking for returns that don’t exist. Then it was just investor wanting more mortgages to invest in than there were mortgages to reasonably offer. But this time, it’s broader, and I fear much worse.
Exandra
@inthehands thanks Paul, your posts helped set to rest some confusion I had about gen AI: how can people, knowing it is bad for their own businesses when it inevitably and continually fails and implicates them, still want to incorporate it into their products.
Your thread answers this question!
Paul Cantrell
@froge @zannesan
I mean, rich fucks, yes, but: it’s systemic. Large-scales forces create and reward their existence. If these people went away or became better, others would appear to take their place.
Zanne
@inthehands crying that this is the shithole world we’ve created for ourselves and we can’t make it stop.
CyberFrog
@zannesan@mastodon.social @inthehands@hachyderm.io you know we really could make it stop... it's just the rich fucks with infinitely more resources than every other human kinda don't want to stop it right now smh
rrdot
@inthehands off topic to this thread, but damn Paul, you've been posting some amazing and on point thoughts and stories the past few days. Thanks for sharing!
Paul Cantrell
@rrdot
Cheers! Screaming into the void may be futile, but I guess it’s nice when somebody enjoys the concert?!
🔏 Matthias Wiesmann
@inthehands well Goodhart’s law applies, even if it was a crappy metric to start with.
Paul Cantrell
@thias
Absolutely. Getting to the place where Goohart is the relevant problem was my foolish dream.
Óscar Morales Vivó
@inthehands or as I usually put it, the bullshit machine looks awful nice to the folks that have made it with bullshit.
Paul Cantrell
@peter
Yeah, the first example is the only one in the thread that's •not• bullshit. Point is that (1) if you say it’s terrible, you have to ask “As compared to what?,” and point of the later examples is (2) the true business goal isn’t always what you think it is.
If we want to understand the function of LLMs for corps, we have to work through those questions. And given the amount of work that’s •already• BS, I’m not necessarily shorting the LLMs even though they’re BS too.
Peter
@inthehands this is a great thread, and i think you're right: companies care about making money, not giving users a pleasant experience. but your original example has mediocre software saving companies millions. whereas LLMs are mediocre software that is basically a money pit, with no indication that they will ever become profitable.
Stewart Sims
@inthehands you'll have to forgive I've not read the full thread. But up to this point I'd like to share an alternative perspective. I work with businesses that still have those 'outdated' processes today and one of the reasons they haven't migrated to more sophisticated software yet is that it offers marginal benefits. When you dig into the cost of employing people to do general admin sometimes the cost-benefit of upgrading is very tight. Often the sales and marketing is the actual 'hoax'.
Paul Cantrell
@stew_sims
Yeah. For example: some of those ancient mainframe systems are just rock solid: somebody built it right in like 1972 in COBOL or whatever, and it’s far wiser to do hardware maint to keep the same code running than to attempt a rewrite!
Point is that in my first example, yes, the bad product really •was• better. “Compared to what?” and “Toward what goal?” are both questions that can have very surprising answers.
Paul Cantrell
@sennoma
sorry
Bill Hooker
@inthehands Jesus Christ, Paul.
BANG!
*thud*
Plsik (born in 320 ppm)
@inthehands I have to say that the level of my pessimism about our civilization has risen again after reading this thread. And that's a good thing. I'm almost never pessimistic enough when I look back in time. Good job, thanks.
Paul Cantrell
@plsik
sorry/welcome?
Paul Cantrell
@KatS
You’re agressively agreeing with me: the point is that, in that first example in my thread that you're replying to, our terrible product really •was• an improvement. Nobody was being swindled. The companies knew how bad our stuff was, and knew how much worse their current processes were. No hoax there.
—
Students are always astonished when I tell them that of course you release software w/bugs. “Ideally, a good team understands •what• bugs they're releasing.”
Kat
@inthehands That's an almost impressive perversion of a perfectly good argument.
Somebody had interviewed the CEO of a software development company (can't remember who or which; doesn't matter). In a time when "you must fix all bugs before release" was popular dogma, they asked whether he was comfortable with shipping releases that he knew contained bugs.
"Absolutely!" he said with a big smile.
Then he explained. Their customers got more value from having the software as it was, even with those bugs, than from not having it. Perfect, good, enemy, value of "now" vs "eventually, when it's perfect" and all that.
I think he was also enthusiastic about fixing those bugs to improve the quality over time. I'm taking that approach, in any case.
Paul Cantrell
@penguin42
Yup, definitely recognize “nobody’s actually gotten that far” from support — and I’ve been on both sides of that conversation!
penguin42
@inthehands Sounds like chip design/CAD software. I remember in a startup my boss installing a new Cadence component and hitting a problem with it, and getting the response from support that they hadn't had anyone get that far before.
Wolf480pl
@inthehands Also, regarding the customer support example:
Yes, it saves time for the company, and it was valuable human time that was being spent on pointless things.
But it makes the *customer* waste *more time* doing pointless things
Moreover, it creates an assymmetry, where the company can spend relatively little time to make the customer waste a lot of time.
So AI is a weapon.
And as such, it should be regulated.
Paul Cantrell
@wolf480pl
Someone once remarked that patients are the free labor of the US medical system, and that’s really stuck with me. And it’s not just medical where that pattern occurs.
Wolf480pl
@inthehands in a sane world, after you realized the whole health score was a hoax, you'd be able (or even required) to report them to some kind of institution that would send inspections and lawsuits their way.
We don't live in a sane world.
Paul Cantrell
@cammerman
You’re welcome??
Chris Ammerman
@inthehands This thread started out as incredibly deflating and ended up flatly horrifying.
Paul Cantrell
Because I’ve apparently driven some people to despair with this thread, some rays of hope:
First, note that point of my very first example is that the product was •not• a hoax. It really did make things better for real people. Sometimes it can be hard for perfectionists like myself to accept that better is •better•. Sometimes we do actually build things that matter, even if they kind of stink relative to some nonexistent Platonic ideal. Take the win, Paul!
Ah, but the rest of the thread…
A/1
Paul Cantrell
As many point out, there are systemic forces at play that create these situations where “fake and dangerous” is acceptable to a business, even if it harms actual humans. It’s not just that there are awful people; it’s that our system creates the awful people it needs.
Leaving aside grandiose utopian theories, a few concrete things that could help:
1. Giving antitrust real teeth
2. Repeat 1 for emphasis
3. Making industry less investor-driven
4. …which means reducing wealth inequality
A/2
Marsh Gardiner 💡🐝🔧
@inthehands spectacular thread. no notes.
Paul Cantrell
Expanding on 3 & 4 a bit: Right now, there’s a huge mass of concentrated wealth whose owners aren’t going to spend it, so they want to invest it. And they want returns. They want more returns than actually are out there to be found. So industry needs to •invent• those returns.
That means a company that will never and can never ever deliver anything of true value can still make it if in convinces investors it might grow.
“Does it matter if it’s wrong?” Not if it's convincing to investors!
A/3
Paul Cantrell
It’s much the same thing that created the 2008 mortgage crisis. Wealth needs a place to roost → mortgages look safe → more demand for mortgage investments than there are mortgages → people find ways to invent mortgages that shouldn’t exist, to attract investment.
It’s not just a hype bubble; it’s an “oversupply of investment dollars” bubble. Is there a word for that? I am not an economist.
@Npars01 replied with more on this here: https://mstdn.social/@Npars01/113604472161286632
A/4
Paul Cantrell
Weirdly, the old consumer-driven version of capitalism, where profit came from sales, now looks lovely compared to the current investor-driven version where profit comes from wooing the ultra-wealthy.
Sorry, Marxists in the replies, I am not optimistic about the inevitable market swing bringing about revolution and glorious utopia. It didn’t in 2008, and it won’t now either. But I do believe the pendulum will swing back, at least.
And we do have greater than zero power to help it swing.
A/5
Paul Cantrell
OK, I promised you optimism:
The world, including the business world, is •full• of good people doing good things. Yes, there are actually businesses that do manage to do good sometimes…but that's not where I’m going with this.
Even in the most useless, soul-sucking, useless, everything-is-fake business environment, there are people solving fun problems, forming collegial relationships, being kind to each other, finding joy in the mundane. And •that• is life’s real work.
A/6
Paul Cantrell
Yes, a whole lot of business activity is fake. It’s playing with sand castles and pretending that princesses are going to live in them. But hell, at least it’s people playing!
If like me it’s hard for you to keep the MBA goggles on, and you keep asking “What about real good for real people?!,” well, all the sand castles come and go, profits come and go, and the lives lived along the way matter more than any of it.
A/7
Paul Cantrell
I don’t want to downplay the harm our systems do. The world is •full• of preventable harm. It’s enraging, and it’s heartbreaking.
All I’m saying is this: don’t neglect the importance of the humans living their lives in the middle of it all. I’m concerned about the quality of the software, but in the end, I’m a lot more concerned about the lives of the people building and using it.
A/8
Paul Cantrell
Even in the fakest and most ridiculous places I’ve worked, including the ones upthread, I’ve seen people being beautiful, exemplary frigging human beings, caring for each other, living good lives that are bigger and better and so much more important than the boulder we’re rolling up the hill just to see it roll down.
•That• is the most important thing in the world. I’d like less waste along the way. I’d sure like less harm. But I recognize that we are very, very far from hopeless.
A/9
Paul Cantrell
@Mela
For all the problems I have with that show, which are many, it really did capture this specific kind of cynicism very well.
Mela Eckenfels
@inthehands *insert Southpark meme here*
anilmc
finger to the moon, man
Paul Cantrell
@grimalkina
What a kind and humane reply, Cat. Of course it would be you who infers the author’s emotional and psychological experience from their writing, who spots the human in the situation. I appreciate you so much for it.
Cat Hicks
@inthehands oh Paul I felt so so so much empathy reading this. Because everyone who is an evidence bringer and data questioner ends up having this shocking moment at some point. When we realize that it is *what we are good at doing* and *what is good to do* that has made people who *asked us to do that thing* turn on us....how I recognize it
CatSalad🐈🥗 (D.Burch) :blobcatrainbow:
@inthehands Nailed it, right there
[Gen AI] leads people on wild goose chases •far• more efficiently than the humans.
Nicole Parsons
One of the alternative "investments" being made is public corruption in broad daylight.
Using campaign finance & Citizens United, #KochNetwork #RockbridgeNetwork , #AtlasNetwork converted constitutional democracy into kleptocracy.
Elon Musk bought a president for a quarter billion bucks. A bargain. He now gets to enrich himself on the taxpayers dime now.
https://www.msnbc.com/top-stories/latest/elon-musk-trump-donations-2024-election-rcna183231
The Supreme Court has legalized bribery of elected officials, and relabeled it as "a gratuity".
Paul Cantrell
@landley
I only half agree: from the little I know of the book, he’s on to something, but his specific assessment of which jobs are BS and how to tell seems deeply flawed and facile to me.
A more accurate assessment is that BS is marbled through most of our jobs in varying proportions, and you can’t really label specific positions — much less specific types of jobs — as being BS across the board.
Rob Landley
@inthehands David Graber's "BS jobs" should be a high school textbook. Late Stage Capitalism is a religion, and his 2013 artivle was 95 theses nailed to its door.
Especially when you ponder the "real" jobs done entirely in support of BS: top floor of management consultants bringing in money, supported by IT, HR, accounts receivable, payroll, janitors, cafeteria staff, and their managers... And all the jobs with 2 hours of real work a week and 38 looking busy.
bouriquet
@Npars01 @inthehands It’s all going to come tumbling down and they’re blind to see it from their lofty towers.
Both the people and the planet just can’t take it anymore.
Paul Cantrell
@bouriquet @Npars01
Sooner or later, yes, it has to. My concern is who gets hurt along the way.
Nicole Parsons
Fossil fuel dollars fund fascist movements & corrupt crony capitalism.
https://www.nytimes.com/2024/11/20/us/politics/rockbridge-trump-vance-wiles.html
https://www.nytimes.com/2024/11/11/us/donald-trump-jr-firm.html
https://tribune.com.pk/story/2489637/silicon-valleys-secret-influence-on-2024-election-revealed
Linda Woodrow
@inthehands @Npars01 I think the word is secular stagnation - too much investment money in an economy with too little demand creating too little growth for "real" places to put it. We need a better word though. I think it's a key concept for our times - we just don't need any more "stuff" - and secular stagnation has no riz.
Deirdre Saoirse Moen
@inthehands I want to give an example of this.
Friend of mine is IT guy for a non-software firm that uses a major db vendor and uses another third-party database set of tables, etc. And they have addons from another company.
Neither of these products use boolean column types. So the first set uses "Y" or "N" as a bool, and the second was written in Iceland, so decided to use "J" and "N" instead.
This kind of thing is just trivially awful, but chews up just great wads of time.
GNU social JP is a social network, courtesy of GNU social JP管理人. It runs on GNU social, version 2.0.2-dev, available under the GNU Affero General Public License.
All GNU social JP content and data are available under the Creative Commons Attribution 3.0 license.