Soatok Dreamseeker
Honestly, the thing that will probably kill LLMs the hardest is someone writing a small language model that fits in JavaScript in a browser and hits comparable benchmarks.
Why bother with all those GPUs and energy usage if your Raspberri Pi could get comparable results?
Rich Felker
@soatok If you want it just to be able to use language, sure. But they want a vastly overfitted model that lossily compresses the volume of human writing and can spit back out obfuscated plagiarism of arbitrary parts.
Soatok Dreamseeker
Is this possible? I dunno. I'm not specialized in this.
But if I wanted to fuck the GenAI bubble over and had the relevant background experience? This is what I'd explore.
Rich Felker
@soatok Right but that's not all they want. They want it to generate obfuscated plagiarism of poetry. They want it to generate "copyright-free" copies of arbitrary FOSS programs, songs, etc. This inherently requires the largeness of the model because the plagiarism is buried in the overfitting.
Soatok Dreamseeker
@dalias One model per language.
Want it to generate C? Download the C model.
Want it to write bad poetry? Download the Vogon I mean English model.
Rich Felker
@soatok If you had to give it the things you wanted copied as explicit input, the plagiarism and copyright infringement would be obvious to users and courts. Making it gigantic ambient state obfuscated in the model is how they get away with it.
Rich Felker
@lritter @soatok "They" being the AI enthusiasts or people who feel like they're getting something of value from "AI". They almost surely don't frame what they want to themselves or others in terms of the plagiarism, but it'd be useless to them without that outcome.
LR
Rich Felker
@lritter @soatok Um, no, you're not being the "bearer of bad news". AI propagandism isn't news. An interpolation of existing works to cover up that it's essentially the same as someone else's work is plagiarism if a human does it too. This is why the early architects of free software always insisted on clean room reimplementations based on a specification someone else had worked out, not reading reverse engineered or leaked proprietary code then pretending they could forget it and write something equivalent.
LR
@dalias @soatok i don't mean to be the bearer of bad news but if these systems would only plagiarize it would certainly be easier to dismiss em. they also inter- and extrapolate, which, if a human had done it, would be counted as original work. of course that's not all that creative work is, and so it's quite limited.
overfitting is undesirable, because it turns a fuzzy database into a regular database - then it's not plagiarism, it's a straight up copyright violation.
GNU Too
Rich Felker
@gnu2 OK but they're not here to help you fuck copyright. They're here for a unidirectional exception for a billionaire fascist owned industry.
Soatok Dreamseeker
@lxo Killing it as in making any hope of a return on investment in all these datacenters impossible
Alexandre Oliva
Alexandre Oliva
Soatok Dreamseeker
@lxo You're completely misunderstanding what I'm suggesting, to the point that I question whether further discussion is even worthwhile
Alexandre Oliva
Soatok Dreamseeker
@lxo The way things have been going for years is this:
I'm focused on killing 1, which directly affects 4 and 5 in some way. I'm not offering a silver bullet for 2, 3, 6, or 7.
But if people with AI expertise were to choke out the centralization of this tech by obviating the big data center investments through "can run on a low-power device in your home", that won't be without impact.
Soatok Dreamseeker
@lxo In short, I'm suggesting that people who have the expertise I lack fight fire with fire.
Mark T. Tomczak
@soatok Incidentally, catching up on your blog:
If that doesn’t make you feel all warm and fuzzy, remember that many industries still use FTP to transfer encrypted ZIP files back and forth in 2026.
I worked as liaison between my company and a company they'd acquired for about nine months. The acquiree's entire business niche was something that really reframed my understanding of where I set the bar for technical literacy and competency in general industry; I just flat-out thought we were training more software engineers than we were, and I was wrong.
This company's business model was that they did high-touch data massaging between
Big online retail search engines like Amazon, Baidu, and Google Shopping
Manufacturers who (a) made really high-quality products in their niche and (b) had an IT team that would celebrate with a pizza party if they could successfully implement one new data pipeline end-to-end. Per quarter.
The job was literally "You FTP us your inventory in whatever format you were able to conduct enough black-arts rituals to get the internal tracking system that someone else built for you to spit out, and we'll turn it into the right formats for these online retail markets." The mechanics of the job were a lot of "Take this not-actually-compliant CSV file and build heuristics to guess which commas were column separators and which were a place where their system had left out quotation marks."
... the company grossed past $1 million annually.
Those FTP'd files aren't even encrypted ZIPs. Encryption is way outside the problem domain. 😉
Soatok Dreamseeker
@mark A lot of the value of slop comes in three buckets:
The actual value of language models that can run on a Raspberry Pi and produce useful results without being one or more of those three buckets is something that gets left off the table during these discussions because of how egregious those three are.
Whether that's an error or a strategic decision to focus on the societal-scale impact of those three is not my place to say.
Mark T. Tomczak
@soatok I've been quietly beating the drum for awhile now that a lot of the anti-AI rhetoric is medium- and long-term moot because the cat. Is. Out. Of. The. Bag.
Yes, OpenAI are fascists and a problem and down with them.
Smaller models already run on a Raspberry Pi, and there's no particular reason to believe at this time that the next iteration of the raw research won't make training or cross-training them better / faster / cheaper. Most of the anti-AI arguments I see don't stay relevant when it becomes "A thing you slap on a shelf PC and have running in your own closet," and I don't think a lot of people are talking about what that world looks like.
Kay Ohtie 🔜 FWA
@mark @soatok Used to work for a company in the armpit of healthcare, but one of the products we offered accepted input in one of a few formats with published standards.
_Industry leader_ software wasn't properly compliant with any of the standards, and I dreaded every I saw a call with a particular caller ID because I knew who'd be on the other end by first name and how she'd be saying we didn't process a file correctly... until we pointed out syntax errors every time.
It wasn't even consistent, and I was just on support, but it was wild to see _smaller_ vendors than a company who's name rhymes with Mick Esson get it right every time, and them simply throw their name around as if that meant it.
Also their software couldn't do public key auth for SFTP, only password, which we didn't permit for SFTP upload.
I'm still trying to figure out if they called all the other vendors out there with similar services to us or they just rolled their eyes and eventually coded around this vendor's lack-of-compliance, or said "fuck it" and started just taking CSV files.
GNU social JP is a social network, courtesy of GNU social JP管理人. It runs on GNU social, version 2.0.2-dev, available under the GNU Affero General Public License.
All GNU social JP content and data are available under the Creative Commons Attribution 3.0 license.