• Public

  • Public
  • Network
  • Groups
  • Featured
  • Popular
  • People

Audiogram with MAIHT3k logo, ep 44 title and subtitle “How o1’s ‘complex chains of thought’ are assessed with meaningless metrics -- and erase what thinking really is” EMILY M. BENDER: The funniest thing--so this is where they're testing it against the International Olympiad in Informatics problems, I guess. So they're doing some kind of supervised learning effectively over this specific task. And then they say, "This model competed in the 2024 IOI under the same conditions as the human contestants." But, "It had 10 hours to solve six challenging algorithmic problems and was allowed 50 submissions per problem." But the thing that cracked me up about this is, "With a relaxed submission constraint, we found that model performance improved significantly. When allowed 10,000 submissions per problem, the Model achieved a score above the gold medal threshold even without any test time selection strategy." ALEX HANNA: I too would like to have 10,000 submissions per every exam I am taking. EMILY M. BENDER: I would never want to be the one grading that exam. So SJayLett says "'Under the same conditions.' It had an itchy nose?" ALEX HANNA: Yeah, this whole thing is a bit bizarre. And it's sort of, the analogy of chain of thought to sort of steps of human reasoning is, you know, a further kind of devaluing of what it means for humans to reason and how humans do it. [Finished in next post]