Embed Notice

@hazlin The problem is that GPT systems lack the ability to reason in the sense that humans do. They're essentially a giant truth table from a logic perspective. You can store a certain degree of logic in one, but no more than that. Even if you trained an absurdly expensive GPT with an attention window that could fit millions of lines of code into it, it could still only store logical relationships up to the depth of the internal layers. Humans, on the other hand, can trace logical threads as deep as they have the patience for.

But all of that is fantasy. The cost of a GPT system roughly correlates to the size of the attention window times the depth of the internal layers. This is why OpenAI keeps shrinking the window size on the free version.