@feld Local or hosted models makes no difference, the reality is that the models need to be tuned per task. They need to behave in ways that are not harmful to the user and allow the user to opt for the model to run slower or faster for given tasks.

If you are struggling with the model handling your context from loss of attention or heads? You should be able to increase its accuracy at cost of performance. Run fast when you can, but when it’s actively harmful you should be able to turn up the heat so to speak.