Conversation

Notices

1. Embed this notice
   Cory Doctorow (pluralistic@mamot.fr)'s status on Wednesday, 23-Aug-2023 23:18:45 JST Cory Doctorow

   It wasn't just Ottawa: #MicrosoftTravel published a whole *bushel* of absurd articles, including the notorious Ottawa guide recommending that tourists dine at the #Ottawa #FoodBank ("go on an empty stomach"):

   https://twitter.com/parismarx/status/1692233111260582161

   --

   If you'd like an essay-formatted version of this thread to read or share, here's a link to it on pluralistic.net, my surveillance-free, ad-free, tracker-free blog:

   https://pluralistic.net/2023/08/23/automation-blindness/#humans-in-the-loop

   1/

   • pettter and novatorine 🏴🏳️‍⚧️ repeated this.
   • Embed this notice
     Cory Doctorow (pluralistic@mamot.fr)'s status on Wednesday, 23-Aug-2023 23:18:54 JST Cory Doctorow

     in reply to

     • Paris Marx

     After @parismarx pointed out the Ottawa article, #BusinessInsider's #NathanMcAlone found several more howlers:

     https://www.businessinsider.com/microsoft-removes-embarrassing-offensive-ai-assisted-travel-articles-2023-8

     There was the article recommending that visitors to Montreal try "a hamburger" and went on to explain that a hamburger was a "sandwich comprised of a ground beef patty, a sliced bun of some kind, and toppings such as lettuce, tomato, cheese, etc" and that some of the best hamburgers in Montreal could be had at McDonald's.

     2/

   • Embed this notice
     pettter (pettter@mastodon.acc.umu.se)'s status on Wednesday, 23-Aug-2023 23:21:27 JST pettter

     in reply to

     @pluralistic I don't think that "ChatGPT spits out something hilariously wrong or nonsensical" is _that_ rare - I think in this case it's more a case of "I'm not getting paid enough to care about this, it's probably fine"

   • Embed this notice
     Cory Doctorow (pluralistic@mamot.fr)'s status on Wednesday, 23-Aug-2023 23:21:29 JST Cory Doctorow

     in reply to

     This deliberate misdirection actually reveals a deep truth about AI: that the story of AI being managed by a #HumanInTheLoop is a fantasy, because humans are neurologically incapable of maintaining vigilance in watching for rare occurrences.

     Our brains wire together neurons that we recruit when we practice a task. When we *don't* practice a task, the parts of our brain that we optimized for it get reused.

     6/

   • Embed this notice
     Cory Doctorow (pluralistic@mamot.fr)'s status on Wednesday, 23-Aug-2023 23:21:30 JST Cory Doctorow

     in reply to

     The unnamed Microsoft spokesdroid only *appears* to be claiming that this wasn't written by an #AI, but they're *actually* just saying that the AI that wrote it wasn't "unsupervised." It was a #SupervisedAI, overseen by a human. Who made an error. Thus: the problem was human error.

     5/

   • Embed this notice
     Cory Doctorow (pluralistic@mamot.fr)'s status on Wednesday, 23-Aug-2023 23:21:34 JST Cory Doctorow

     in reply to

     Microsoft insisted that this wasn't an issue of "#UnsupervisedAI," but rather "human error." On its face, this presents a head-scratcher: is Microsoft saying that a human being erroneously decided to recommend the dining at Ottawa's food bank?

     But a close parsing of the mealy-mouthed disclaimer reveals the truth.

     4/

   • Embed this notice
     Cory Doctorow (pluralistic@mamot.fr)'s status on Wednesday, 23-Aug-2023 23:21:35 JST Cory Doctorow

     in reply to

     For Anchorage, #Microsoft recommended trying the local delicacy known as "seafood," which it defined as "basically any form of sea life regarded as food by humans, prominently including fish and shellfish," going on to say, "seafood is a versatile ingredient, so it makes sense that we eat it worldwide."

     In Tokyo, visitors seeking "photo-worthy spots" were advised to "eat Wagyu beef."

     There were more.

     3/

   • Embed this notice
     Sabrina Web :privacypride: 📎 (sabrinaweb71@sociale.network)'s status on Thursday, 24-Aug-2023 05:03:59 JST Sabrina Web :privacypride: 📎

     in reply to

     • quinta :ubuntu:

     @pluralistic this definition of an LLM as a Plausible Sentence Generator is perfect (just like @quinta 's S.A.L.A.M.I.), I think I will borrow it

   • Embed this notice
     Cory Doctorow (pluralistic@mamot.fr)'s status on Thursday, 24-Aug-2023 05:04:00 JST Cory Doctorow

     in reply to

     Microsoft - like every corporation - is insatiably horny for firing workers. It spent the past three years cutting its writing staff to the bone, with the express intent of having AI fill its pages, with humans relegated to skimming the output of the #PlausibleSentenceGenerators and clicking "OK":

     https://www.businessinsider.com/microsoft-news-cuts-dozens-of-staffers-in-shift-to-ai-2020-5

     We know about the howlers and the clunkers that Microsoft published, but what about all the other travel articles that don't contain any (obvious) mistakes?
     15/

     novatorine 🏴🏳️‍⚧️ repeated this.
   • Embed this notice
     Cory Doctorow (pluralistic@mamot.fr)'s status on Thursday, 24-Aug-2023 05:04:03 JST Cory Doctorow

     in reply to

     As I wrote earlier: "There's no market for a machine-learning autopilot, or content moderation algorithm, or loan officer, if all it does is cough up a recommendation for a human to evaluate. Either that system will work so poorly that it gets thrown away, or it works so well that the inattentive human just button-mashes 'OK' every time a dialog box appears":

     https://pluralistic.net/2022/10/21/let-me-summarize/#i-read-the-abstract

     14/

   • Embed this notice
     Cory Doctorow (pluralistic@mamot.fr)'s status on Thursday, 24-Aug-2023 05:04:04 JST Cory Doctorow

     in reply to

     The "human in the loop" is a figleaf. The *whole point* of automation is to create a system that operates at superhuman scale - you don't buy an #LLM to write *one* Microsoft Travel article, you get it to write *a million* of them, to flood the zone, top the search engines, and dominate the space.

     13/

     novatorine 🏴🏳️‍⚧️ repeated this.
   • Embed this notice
     Cory Doctorow (pluralistic@mamot.fr)'s status on Thursday, 24-Aug-2023 05:04:05 JST Cory Doctorow

     in reply to

     But the problem is, the other 0.01% of the time, it's because criminals are waiting for you to click "OK" so they can steal all your money:

     https://finance.yahoo.com/news/ema-report-finds-nearly-80-130300983.html

     Automation blindness can't be automated away. From interpreting radiographic scans:

     https://healthitanalytics.com/news/ai-could-safely-automate-some-x-ray-interpretation

     to #AutonomousVehicles:

     https://newsroom.unsw.edu.au/news/science-tech/automated-vehicles-may-encourage-new-breed-distracted-drivers

     12/

   • Embed this notice
     Cory Doctorow (pluralistic@mamot.fr)'s status on Thursday, 24-Aug-2023 05:04:07 JST Cory Doctorow

     in reply to

     This is an inescapable, biological aspect of human cognition: we *can't* maintain vigilance for rare outcomes. This has long been understood in automation circles, where it is called "#AutomationBlindness" or "#AutomationInattention":

     https://pubmed.ncbi.nlm.nih.gov/29939767/

     Here's the thing: if nearly all of the time the machine does the right thing, the human "supervisor" who oversees it becomes *incapable* of spotting its error.

     10/

   • Embed this notice
     Cory Doctorow (pluralistic@mamot.fr)'s status on Thursday, 24-Aug-2023 05:04:07 JST Cory Doctorow

     in reply to

     The job of "review every machine decision and press the green button if it's correct" *inevitably* becomes "just press the green button," assuming that the machine is usually right.

     This is a *huge* problem. It's why people just click "OK" when they get a bad certificate error in their browsers. 99.99% of the time, the error was caused by someone forgetting to replace an expired certificate.

     11/

   • Embed this notice
     Cory Doctorow (pluralistic@mamot.fr)'s status on Thursday, 24-Aug-2023 05:04:08 JST Cory Doctorow

     in reply to

     In other words, they train *all day* to spot water bottles, and the only training they get in spotting knives, guns and bombs is in exercises, or the odd time someone forgets about the hand-cannon they shlep around in their day-pack. Of *course* they're excellent at spotting water bottles and shit at spotting weapons.

     9/

   • Embed this notice
     Cory Doctorow (pluralistic@mamot.fr)'s status on Thursday, 24-Aug-2023 05:04:10 JST Cory Doctorow

     in reply to

     TSA agents (not "officers," please - they're bureaucrats, not cops) spend *all day* spotting water bottles that we forget in our carry-ons, but almost *no one* tries to smuggle a weapons through a checkpoint - 99.999999% of the guns and knives they *do* seize are the result of flier forgetfulness, not a planned hijacking.

     8/

   • Embed this notice
     Cory Doctorow (pluralistic@mamot.fr)'s status on Thursday, 24-Aug-2023 05:04:13 JST Cory Doctorow

     in reply to

     Our brains are finite and so don't have the luxury of reserving precious cells for things we don't do.

     That's why the #TSA sucks so hard at its job - why they are the world's most skilled water-bottle-detecting X-ray readers, but consistently fail to spot the bombs and guns that red teams successfully smuggle past their checkpoints:

     https://www.nbcnews.com/news/us-news/investigation-breaches-us-airports-allowed-weapons-through-n367851

     7/

   • Embed this notice
     Mayobrot 🇵🇸 (mayobrot@zirk.us)'s status on Thursday, 24-Aug-2023 22:24:00 JST Mayobrot 🇵🇸

     in reply to

     @pluralistic
     [In a positive marketing voice] With ChatGPT, now *you* can become the next Steve Bannon!