Conversation

Notices

1. Embed this notice
   Sexy Moon (moon@shitposter.club)'s status on Tuesday, 06-Feb-2024 09:30:35 JST Sexy Moon
   It is almost impossible to coerce llama.cpp into returning just the fucking data you asked for from the command prompt.
   • Haelwenn /элвэн/ :triskell: and kaia like this.
   • Embed this notice
     Sexy Moon (moon@shitposter.club)'s status on Tuesday, 06-Feb-2024 09:34:57 JST Sexy Moon

     in reply to
     duhhh just use the server
   • Embed this notice
     Sexy Moon (moon@shitposter.club)'s status on Tuesday, 06-Feb-2024 09:49:24 JST Sexy Moon

     in reply to

     • The Harblinger
     @harblinger It's still fucking up, but because it's too stupid to return actual JSON without shit in front of it like "example:" or "example output:" that's impossible to detect because it changes every time.
   • Embed this notice
     The Harblinger (harblinger@shitposter.club)'s status on Tuesday, 06-Feb-2024 09:49:25 JST The Harblinger

     in reply to
     @Moon I was going to say I read the server docs yesterday and it seems like The Way(tm) https://github.com/ggerganov/llama.cpp/blob/master/examples/server/README.md
   • Embed this notice
     Sexy Moon (moon@shitposter.club)'s status on Tuesday, 06-Feb-2024 09:50:15 JST Sexy Moon

     in reply to

     • The Harblinger
     @harblinger So I am right back to square one, trying to figure out how to strip all the shit text before valid JSON.
   • Embed this notice
     Sexy Moon (moon@shitposter.club)'s status on Tuesday, 06-Feb-2024 09:52:31 JST Sexy Moon

     in reply to

     • The Harblinger
     • mr64bit
     @mr64bit @harblinger Yeah I need to learn how to do that.
   • Embed this notice
     mr64bit (mr64bit@p.mr64.net)'s status on Tuesday, 06-Feb-2024 09:52:32 JST mr64bit

     in reply to

     • The Harblinger
     @Moon @harblinger Supposedly this is a thing, never used it though. (force the sampler to only select tokens which are valid grammer)
     https://github.com/ggerganov/llama.cpp/blob/master/grammars/README.md
   • Embed this notice
     Sexy Moon (moon@shitposter.club)'s status on Tuesday, 06-Feb-2024 09:54:41 JST Sexy Moon

     in reply to

     • Birdulon
     • The Harblinger
     @birdulon @harblinger I'm trying to do it in bourne shell using regular cli tools, if I can't do that then yes I'll just write a program in node.js to strip every character before "["
   • Embed this notice
     Birdulon (birdulon@shpposter.club)'s status on Tuesday, 06-Feb-2024 09:54:43 JST Birdulon

     in reply to

     • The Harblinger
     @Moon @harblinger uhhhh just try-catch parsing the text as json progressively stripping one whitespace-separated token from the start until it succeeds