Conversation

Notices

1. Embed this notice
   kaia (kaia@brotka.st)'s status on Wednesday, 14-Feb-2024 00:38:06 JST kaia
   @4censord 24GB and I can run models that are pretty close to GPT-4. and have the advantage of being completely uncensored. with two 3090s I could load larger models.
   • Embed this notice
     Sexy Moon (moon@shitposter.club)'s status on Wednesday, 14-Feb-2024 00:39:23 JST Sexy Moon

     in reply to
     @kaia @4censord are you running into situations where you need that extra 24GB?
   • Embed this notice
     Sexy Moon (moon@shitposter.club)'s status on Wednesday, 14-Feb-2024 01:00:04 JST Sexy Moon

     in reply to

     • guizzy (in exile)
     @guizzy @4censord @kaia I was asking if she, specifically, was trying to do things and running out of VRAM. I am now considering a second card because of Mixtral.
     kaia likes this.
   • Embed this notice
     guizzy (in exile) (guizzy@shitposter.club)'s status on Wednesday, 14-Feb-2024 01:00:05 JST guizzy (in exile)

     in reply to

     • Sexy Moon
     @Moon @kaia @4censord To run 70B models or Mixtral entirely on GPU at higher bit precision (4, 5, 6 bit quants), especially with a large max context, you need more than 24gb.
     kaia likes this.
   • Embed this notice
     kaia (kaia@brotka.st)'s status on Wednesday, 14-Feb-2024 01:04:28 JST kaia

     in reply to

     • Sexy Moon
     • guizzy (in exile)
     @Moon @guizzy @4censord
     same issue with Mixtral. 7b models are nice though!
     
     I love that I can use it to help me with smutty stories, but it's probably not worth another $2k to make it even better
   • Embed this notice
     Sexy Moon (moon@shitposter.club)'s status on Wednesday, 14-Feb-2024 01:05:17 JST Sexy Moon

     in reply to

     • guizzy (in exile)
     @kaia @guizzy @4censord I am realizing I have to train my own models to get better results.
     kaia likes this.
   • Embed this notice
     kaia (kaia@brotka.st)'s status on Wednesday, 14-Feb-2024 01:09:05 JST kaia

     in reply to

     • Sexy Moon
     • guizzy (in exile)
     @Moon @guizzy @4censord
     yeah that's also a big factor for me: I should invest more time and effort to make it better, but eventually I'm not as interested in the technique, only the outcome :asuka_hmm:
     Sexy Moon likes this.
   • Embed this notice
     Sexy Moon (moon@shitposter.club)'s status on Wednesday, 14-Feb-2024 01:17:51 JST Sexy Moon

     in reply to

     • guizzy (in exile)
     @kaia @guizzy @4censord I'm inrterested I just don't have time!
     kaia likes this.
   • Embed this notice
     guizzy (in exile) (guizzy@shitposter.club)'s status on Wednesday, 14-Feb-2024 01:45:29 JST guizzy (in exile)

     in reply to

     • Sexy Moon
     @kaia @Moon @4censord Yeah, you can probably already get Mixtral to write to you at a leisurely reading speed, no? A second 24Gb would make it faster, but is it useful to be faster as a writing assistant?
     
     As a coding assistant, the answer would be yes.
     kaia likes this.