@Moon @icedquinn Yeah, it's a problem with the field, it moves so fast that to stay relevant devs have to jump to the next feature immediately and never have time to clean up behind them; Oobabooga (the project, not the guy) is the most guilty of all here; it supports almost everything, at the cost of shitty UX and badly or sometimes completely undocumented features. My bad for pushing you towards it, it seems simple to me because I've been dealing with its quirks for the better part of a year.

If you want something end user or production ready, there's some options: lmstudio, ollama, vllm. But they won't support the bleeding edge feature. I don't think any of them support grammars.