To benchmark GPT-4's coding ability, OpenAl evaluated it on problems from Codeforces, a website that hosts coding competitions. Surprisingly, Horace He pointed out that GPT-4 solved 10/10 pre-2021 problems and 0/10 recent problems in the easy category. The training data cutoff for GPT-4 is September 2021. This strongly suggests that the model is able to memorize solutions from its training set — or at least partly memorize them, enough that it can fill in what it can't recall.
https://stockroom.wandering.shop/media_attachments/files/110/104/159/238/443/320/original/997ffc5ef969fa62.png