a few things you should pay attention to when you write your own LLM harness:

- get caching right, and better test it. If you get this wrong, you will burn through your tokens before you can even notice. easy way to fuck this up is to put something into the system prompt that changes with every turn, like a timestamp. conversations are APPEND ONLY.

- don't feed unlimited tool responses into the agent. if an agent gets the command wrong and suddenly gets 200kb injected straight into its context with nonsense, that doesn't help anyone. come up with ways to reduce this, or to let the agent ask for more.

- think about how an agent should react if you send him a message mid turn. there's a few ways to go about it, decide what you want.

- pay attention to image handling. all models support text, but not all support vision, even though it's extremely useful for any kind of debugging with a visual component.

- don't try to second guess the models too much. the harness should actually be more of a toolbox. current models are smart and capable. do not try to steer them too much in their choice of tools.

Just some things i noticed.