Embed Notice

HTML Code

<blockquote style="position: relative; padding-left: 55px;"><section><a href="https://wandering.shop/users/janellecshane/statuses/110104164829618120">Janelle Shane (janellecshane@wandering.shop)'s status on Thursday, 30-Mar-2023 00:23:13 JST</a><a href="https://wandering.shop/@janellecshane" title="janellecshane@wandering.shop"><img src="https://gnusocial.jp/avatar/60812-48-20221208225749.webp" width="48" height="48" alt="Janelle Shane" style="position: absolute; left: 0; top: 0;">Janelle Shane</a></section><article><p>Remember seeing something about GPT-4 doing well on standardized tests? It turns out it may have memorized the answers.<br><a href="https://aisnakeoil.substack.com/p/gpt-4-and-professional-benchmarks" rel="nofollow noreferrer">https://aisnakeoil.substack.com/p/gpt-4-and-professional-benchmarks</a><br><a href="https://wandering.shop/tags/gpt4" rel="tag">#gpt4</a> <a href="https://wandering.shop/tags/AIHype" rel="tag">#AIHype</a> <a href="https://wandering.shop/tags/ThisIsWhyWeDontTestOnTheTrainingData" rel="tag">#ThisIsWhyWeDontTestOnTheTrainingData</a></p></article><footer><a rel="bookmark" href="https://gnusocial.jp/conversation/1359812#notice-2812106">In conversation</a><time datetime="2023-03-30T00:23:13+09:00" title="Thursday, 30-Mar-2023 00:23:13 JST">Thursday, 30-Mar-2023 00:23:13 JST</time> <span>from <span><a href="https://wandering.shop/@janellecshane/110104164829618120" rel="external" title="Sent from wandering.shop via ActivityPub">wandering.shop</a></span></span><a href="https://wandering.shop/@janellecshane/110104164829618120">permalink</a><h4>Attachments</h4><ol><li><label><a rel="external" href="https://gnusocial.jp/attachment/840435">To benchmark GPT-4's coding ability, OpenAl evaluated it on problems from Codeforces, a website that hosts coding competitions. Surprisingly, Horace He pointed out that GPT-4 solved 10/10 pre-2021 problems and 0/10 recent problems in the easy category. The training data cutoff for GPT-4 is September 2021. This strongly suggests that the model is able to memorize solutions from its training set — or at least partly memorize them, enough that it can fill in what it can't recall.</a></label><br><a href="https://stockroom.wandering.shop/media_attachments/files/110/104/159/238/443/320/original/997ffc5ef969fa62.png" rel="external">https://stockroom.wandering.shop/media_attachments/files/110/104/159/238/443/320/original/997ffc5ef969fa62.png</a></li><li><article><header><div>Domain not in remote thumbnail source whitelist: substackcdn.com</div><h5><a href="https://aisnakeoil.substack.com/p/gpt-4-and-professional-benchmarks">GPT-4 and professional benchmarks: the wrong answer to the wrong question</a></h5><div> from <span>Arvind Narayanan</span></div></header><div>OpenAI may have tested on the training data. Besides, human benchmarks are meaningless for bots.</div><footer></footer></article></li></ol></footer></blockquote>

Corresponding Notice

Embed this notice
Janelle Shane (janellecshane@wandering.shop)'s status on Thursday, 30-Mar-2023 00:23:13 JST Janelle Shane
Remember seeing something about GPT-4 doing well on standardized tests? It turns out it may have memorized the answers.
https://aisnakeoil.substack.com/p/gpt-4-and-professional-benchmarks
#gpt4 #AIHype #ThisIsWhyWeDontTestOnTheTrainingData
In conversationThursday, 30-Mar-2023 00:23:13 JST from wandering.shoppermalink
Attachments
1. To benchmark GPT-4's coding ability, OpenAl evaluated it on problems from Codeforces, a website that hosts coding competitions. Surprisingly, Horace He pointed out that GPT-4 solved 10/10 pre-2021 problems and 0/10 recent problems in the easy category. The training data cutoff for GPT-4 is September 2021. This strongly suggests that the model is able to memorize solutions from its training set — or at least partly memorize them, enough that it can fill in what it can't recall.
  https://stockroom.wandering.shop/media_attachments/files/110/104/159/238/443/320/original/997ffc5ef969fa62.png
2. Domain not in remote thumbnail source whitelist: substackcdn.com
  GPT-4 and professional benchmarks: the wrong answer to the wrong question
  from Arvind Narayanan
  OpenAI may have tested on the training data. Besides, human benchmarks are meaningless for bots.

Public

Embed Notice

HTML Code

Corresponding Notice