Conversation

Notices

1. Embed this notice
   Cory Doctorow (pluralistic@mamot.fr)'s status on Saturday, 23-Dec-2023 13:41:22 JST Cory Doctorow

   • Medium

   This week on my podcast, I read my recent @medium column, "How To Think About #Scraping: In #privacy and #labor fights, copyright is a clumsy tool at best," which proposes ways to retain the benefits of scraping without the privacy and labor harms that sometimes accompany it:

   https://doctorow.medium.com/how-to-think-about-scraping-2db6f69a7e3d?sk=4a1d687171de1a3f3751433bffbb5a96

   --

   If you'd like an essay-formatted version of this thread to read or share, here's a link to it on pluralistic.net, my surveillance-free, ad-free, tracker-free blog:

   https://pluralistic.net/2023/09/25/deep-scrape/#steering-with-the-windshield-wipers

   1/

   • clacke likes this.
   • Embed this notice
     Cory Doctorow (pluralistic@mamot.fr)'s status on Saturday, 23-Dec-2023 13:41:12 JST Cory Doctorow

     in reply to

     Since 2017, the Archive has started ignoring robots.txt files for news services; whether or not the news site wants to be crawled, the Archive crawls it and makes copies of the different versions of the articles the site publishes. That's because news sites - even the so-called "paper of record" - have a nasty habit of making sweeping edits to published material without noting it.

     8/

     clacke likes this.
   • Embed this notice
     Cory Doctorow (pluralistic@mamot.fr)'s status on Saturday, 23-Dec-2023 13:41:13 JST Cory Doctorow

     in reply to

     For the most part, the Internet Archive limits its scraping to websites that permit it. The #RobotsExclusionProtocol (AKA #robots.txt) makes it easy for webmasters to tell different kinds of crawlers whether or not they are welcome. If your site has a robots.txt file that tells the Archive's crawler to buzz off, it'll go elsewhere.

     Mostly.

     7/

   • Embed this notice
     Cory Doctorow (pluralistic@mamot.fr)'s status on Saturday, 23-Dec-2023 13:41:15 JST Cory Doctorow

     in reply to

     • internetarchive

     But what about entities that explicitly object to being scraped? Sometimes, it's good to scrape them, too.

     Since 1996, the @internetarchive has scraped every website it could find, storing snapshots of every page it found in a giant, searchable database called #TheWaybackMachine. Many of us have used the Wayback Machine to retrieve some long-deleted text, sound, image or video from the internet's memory hole.

     6/

   • Embed this notice
     Cory Doctorow (pluralistic@mamot.fr)'s status on Saturday, 23-Dec-2023 13:41:16 JST Cory Doctorow

     in reply to

     The linguists who study this speech will go through #InstitutionalReviewBoard approvals to make sure that as they produce aggregate accounts of speech, they don't compromise the privacy or integrity of their subjects.

     Computational linguistics is an unalloyed good, and while the speakers whose words are scraped to produce the raw material that these scholars study, they probably wouldn't object, either.

     5/

   • Embed this notice
     Cory Doctorow (pluralistic@mamot.fr)'s status on Saturday, 23-Dec-2023 13:41:17 JST Cory Doctorow

     in reply to

     This ushered in a new era in linguistic study, one that is enthusiastically analyzing and codifying the rules of informal speech, the spread of vernacular, and the regional, racial and class markers of different kinds of speech:

     https://memex.craphound.com/2019/07/24/because-internet-the-new-linguistics-of-informal-english/

     The people whose speech is scraped and analyzed this way are often unreachable (anonymous or pseudonymous) or impractical to reach (because there's millions of them).

     4/

   • Embed this notice
     Cory Doctorow (pluralistic@mamot.fr)'s status on Saturday, 23-Dec-2023 13:41:19 JST Cory Doctorow

     in reply to

     The thing is, very few of us produce formal, written work, whereas we *all* engage in casual dialog. But then the internet came along, and for the first time, we had a species of mass-scale, informal dialog that also written, and which was born in machine-readable form.

     3/

   • Embed this notice
     Cory Doctorow (pluralistic@mamot.fr)'s status on Saturday, 23-Dec-2023 13:41:21 JST Cory Doctorow

     in reply to

     What are those benefits from scraping? Well, take #ComputationalLinguistics, a relatively new discipline that is producing the first accounts of how informal language works. Historically, linguists overstudied written language (because it was easy to analyze) and underanalyzed speech (because you had to record speakers and then get grad students to transcribe their dialog).

     2/

   • Embed this notice
     Cory Doctorow (pluralistic@mamot.fr)'s status on Saturday, 23-Dec-2023 13:41:25 JST Cory Doctorow

     in reply to

     I'm not talking about fixing a typo or a formatting error: I'm talking about making a *massive* change to a piece, one that completely reverses its meaning, and pretending that it was that way all along:

     https://medium.com/@brokenravioli/proof-that-the-new-york-times-isn-t-feeling-the-bern-c74e1109cdf6

     This happens *all the time*, with major news sites from all around the world:

     http://newsdiffs.org/examples/

     By scraping these sites and retaining the different versions of their article, the Archive both detects and prevents journalistic malpractice.

     9/

     clacke likes this.
   • Embed this notice
     Cory Doctorow (pluralistic@mamot.fr)'s status on Saturday, 23-Dec-2023 13:43:29 JST Cory Doctorow

     in reply to

     Like all self-declared "job creators," they constantly fantasize about destroying all of our jobs. Like all capitalists, they hate capitalism, and dream of earning rents from owning things, not from doing things.

     The work these AI tools sucks, but that doesn't mean our bosses won't try to fire us and replace us with them.

     22/

     clacke likes this.
   • Embed this notice
     Cory Doctorow (pluralistic@mamot.fr)'s status on Saturday, 23-Dec-2023 13:43:30 JST Cory Doctorow

     in reply to

     Scraping to produce LLMs isn't intrinsically harmful, and neither are LLMs. Admittedly, nonprofits using LLMs to build war crimes databases do not justify even 0.0001% of the valuations that AI hypesters ascribe to the field, but that's their problem.

     Scraping is good, sometimes - even when it's done against the wishes of the scraped, even when it harms their interests, and even when it's used to train an LLM.

     But.

     21/

     clacke repeated this.
   • Embed this notice
     Cory Doctorow (pluralistic@mamot.fr)'s status on Saturday, 23-Dec-2023 13:43:30 JST Cory Doctorow

     in reply to

     Scraping to violate peoples' privacy is *very* bad. Take #ClearviewAI, the grifty, sleazy #FacialRecognition company that scraped billions of photos in order to train a system that they sell to cops, corporations and authoritarian governments:

     https://pluralistic.net/2023/09/20/steal-your-face/#hoan-ton-that

     Likewise: scraping to alienate creative workers' labor is *very* bad. Creators' bosses are ferociously committed to firing us all and replacing us with "generative AI."

     22/

     clacke likes this.
   • Embed this notice
     Cory Doctorow (pluralistic@mamot.fr)'s status on Saturday, 23-Dec-2023 13:43:32 JST Cory Doctorow

     in reply to

     This is important work, and it was carried out at a scale and with a precision that would have been impossible without an LLM. As with all of HRDAG's work, this report and the subsequent testimony draw on cutting-edge statistical techniques and skilled science communication to bring technical rigor to some of the most important justice questions in our world.

     LLMs need large bodies of text to train them - text that, inevitably, is scraped.

     20/

   • Embed this notice
     Cory Doctorow (pluralistic@mamot.fr)'s status on Saturday, 23-Dec-2023 13:43:33 JST Cory Doctorow

     in reply to

     This work forms the core of ongoing Colombian Truth and Reconciliation proceedings, and has been instrumental in demonstrating that the majority of war crimes were carried out by right-wing militias who operated with the direction and knowledge of the richest, most powerful people in the country. It also showed that the majority of child soldier recruitment was carried out by these CIA-backed, US-funded militias.

     19

   • Embed this notice
     Cory Doctorow (pluralistic@mamot.fr)'s status on Saturday, 23-Dec-2023 13:43:34 JST Cory Doctorow

     in reply to

     Using a LLM, HRDAG was able to produce an analysis of each killing in each database, estimating the probability that it appeared in more than one database, and the probability that it was carried out by a right-wing militia, by government forces, or by FARC guerrillas.

     18/

   • Embed this notice
     Cory Doctorow (pluralistic@mamot.fr)'s status on Saturday, 23-Dec-2023 13:43:35 JST Cory Doctorow

     in reply to

     HRDAG's latest project is its most ambitious yet. Working with partner org #Dejusticia, they've just released the largest data-set in human rights history:

     https://hrdag.org/jep-cev-colombia/

     What's in that dataset? It's a merger and analysis of more than 100 databases of killings, #ChildSoldier recruitments and other crimes during the Colombian civil war.

     17/

   • Embed this notice
     Cory Doctorow (pluralistic@mamot.fr)'s status on Saturday, 23-Dec-2023 13:43:37 JST Cory Doctorow

     in reply to

     Take #HRDAG's work on #TruthAndReconciliation in #Colombia. The #HumanRightsDataAnalysisGroup is a tiny nonprofit that makes an outsized contribution to #HumanRights, by using statistical methods to reveal the full scope of the human rights crimes that take place in the shadows, from #EastTimor to #Serbia, #SouthAfrica to the USA:

     https://hrdag.org/

     16/

   • Embed this notice
     Cory Doctorow (pluralistic@mamot.fr)'s status on Saturday, 23-Dec-2023 13:43:38 JST Cory Doctorow

     in reply to

     We should scrape *all* of these looting bastards, even though it will harm their economic interests. We should scrape them *because* it will harm their economic interests. Scrape 'em and scrape 'em and scrape 'em.

     Now, it's one thing to scrape text for scholarly purposes, or for journalistic accountability, or to uncover criminal corporate conspiracies. But what about scraping to train a #LargeLanguageModel?

     Yes, there are socially beneficial - even vital - uses for #LLMs.

     15/

   • Embed this notice
     Cory Doctorow (pluralistic@mamot.fr)'s status on Saturday, 23-Dec-2023 13:43:39 JST Cory Doctorow

     in reply to

     Zechner's work comes at a crucial time, as grocers around the world turn the screws on both their suppliers and their customers, disguising their #greedflation as #inflation. In Canada, the grocery cartel - led by the guillotine-friendly hereditary grocery monopolilst #GalenWeston - pulled the most Les Mis-ass caper imaginable when they illegally conspired to rig the price of *bread*:

     https://en.wikipedia.org/wiki/Bread_price-fixing_in_Canada

     14/

   • Embed this notice
     Cory Doctorow (pluralistic@mamot.fr)'s status on Saturday, 23-Dec-2023 13:43:40 JST Cory Doctorow

     in reply to

     Soon, they were documenting ripoffs in #Germany and #Slovenia, too:

     https://mastodon.gamedev.place/@badlogic/111071485142332765

     Zechner's on a roll, but the grocery cartel could shut him down with a keystroke, simply by blocking his API access. If they do, Zechner *could* switch to scraping their sites - but only if he can be protected from legal liability for nonconsensually scraping commercially sensitive data in a way that undermines the profits of a powerful corporation.

     13/

   • Embed this notice
     Cory Doctorow (pluralistic@mamot.fr)'s status on Saturday, 23-Dec-2023 13:43:41 JST Cory Doctorow

     in reply to

     Zechner feared publishing his results at first. The companies whose thefts he'd discovered have enormous power and whole kennelsful of vicious attack-lawyers they can sic on him. But he eventually got the Austrian competition bureaucracy interested in his work, and they published a report that validated his claims and praised his work:

     https://mastodon.gamedev.place/@badlogic/111071673594791946

     Emboldened, Zechner open-sourced his monitoring tool, and attracted developers from other countries.

     12/

   • Embed this notice
     Cory Doctorow (pluralistic@mamot.fr)'s status on Saturday, 23-Dec-2023 13:43:42 JST Cory Doctorow

     in reply to

     • Mario Zechner

     Austrian technologist @badlogic used the #API from country's super-concentrated grocery giants to prove that they were colluding to rig prices. By assembling a longitudinal data-set, Zechner exposed the raft of dirty tricks the grocers used to rip off the people of Austria.

     From #shrinkflation to deceptive price-cycling that disguised price *hikes* as *discounts*:

     https://mastodon.gamedev.place/@badlogic/111071627182734180

     11/

   • Embed this notice
     Cory Doctorow (pluralistic@mamot.fr)'s status on Saturday, 23-Dec-2023 13:43:43 JST Cory Doctorow

     in reply to

     This is canonical #FairUse, the kind of copying that almost always involves overriding the objections of the site's proprietor. Not all adversarial scraping is good, but this sure is.

     There's an argument that scraping the news-sites without permission might piss them off, but it doesn't bring them any real harm. But even when scraping harms the scrapee, it is sometimes legitimate - and necessary.

     10/

   • Embed this notice
     Cory Doctorow (pluralistic@mamot.fr)'s status on Saturday, 23-Dec-2023 13:43:47 JST Cory Doctorow

     in reply to

     After all, prompting an LLM may produce bad wrting, but at least the LLM doesn't give you lip when you order to it give you "ET but the hero is a dog, and there's a love story in the second act and a big shootout in the climax." Studio execs already talk to screenwriters like they're LLMs.

     That's true of art directors, newspaper owners, and all the other job-destroyers who can't believe that creative workers want to have a say in the work they do - and worse, get paid for it.

     23/

     clacke likes this.