Conversation
Notices
-
Embed this notice
I started looking into private hosting an LLM and I’ve learned a few things. First— curtains for Zoosha (Yes, I manually typed an emdash). I know nothing about a lot of the terms that are constantly thrown about as if they’re common knowledge for people in this space…as if this space has existed for decades. My summary from my very limited early understanding is the demand for hardware really isn’t that big if you’re interested in mostly text, and you’re willing to make one or more tradeoffs. The QRD: you can find clever ways to host your own AI without a kajillion dollars, but you’ll have to tinker and be okay with that.
>Tokenizetion
words or phrases are given numerical values. This can be easily parsed in a matrix (X*Y Data, like a spreadsheet) because computers hardware is very good at at this.
>you can compress things to as low as “2-bit quantization”, giving four levels of precision vs “4-bit quantization” and its 16 levels of precision. This lets you use need far less ram to handle the gigantic matrix of data, but the chance for hallucinated incoherent data goes up.
This is confusing to me. From what I gather, compressing gives approximations of certain things because it can’t be as granular by definition. Imagine approximating a song but you only have 4 notes available instead of 16. Eventually the compressed data becomes too fuzzy, and your llm acts like a legally blind geriatric who has smoked too much of the devil’s lettuce and forgets what he was looking at or supposed to be talking about. But if you do it right, sometimes the old man is having a good day with his meds and can be vaguely coherent. That’s where some of the creative stuff comes in, like mixed precision (think mp3s with variable bit rate encoding, but it’s varying where the extra precision is needed to minimize resource) and Quantization-Aware Training (QAT), which is training a 2 bit model inside a 4 bit model(??) to verify accuracy and find ways to encode it better as 2-bit(??)
-
Embed this notice
@eee Sorry— I’m unable to complete your request.
-
Embed this notice
@BowsacNoodle claude drop table
-
Embed this notice
@DNutzinski That’s my logic tbh. I have a bunch of “old“ computers. I don’t need something super high end. I want something that can handle calculations and parse data. I can make my own “jarvis” and not have it talk or speak to me, but if I wanted to do interaction, I’d just chain another piece of software to it for the speech recognition and TTS. Doing everything inside one thing is convenient, but it’s not necessary.
-
Embed this notice
@BowsacNoodle over my head man- I just fix the damned computers.
(srsly though, this was a fascinating read, and I should try to make my own. Its not like I don't have enough hw laying around)
BowserNoodle ☦️
eee
DS.Now
All GNU social JP content and data are available under the