Embed Notice

Hear me out: I think applying RL on #LLMs and LMMs is misguided, and we can do much better.

Those #RL algorithms are unsuitable for this, and for example they cannot learn how their decisions affect the eventual rewards, but instead are just optimized to make the decisions based on Bellman optimization.

Instead we can simply condition the LLMs with the rewards. The rewards become the inputs to the model, not something external to it, so the model will learn the proper reward dynamics, instead of only being externally forced towards the rewards. The model can itself do the credit assignment optimally without fancy mathematical heuristics!

This isn't a new idea, it comes from goal-conditioned RL, and decision transformers.

We can simply run the reasoning trajectories, judge the outcomes, and then put the outcome tokens first to these trajectories before training them to the model in a batch.

https://arxiv.org/abs/2211.15657