Blog · ai · the-ai-times · video · · Trevor Heshka
Benchmarks: The Measurement Problem (The AI Times Ep. 036)
Why AI benchmarks are harder to trust than they look, and what the measurement problem means for comparing models.
Originally posted at youtu.be
Why AI benchmarks are harder to trust than they look, and what the measurement problem means for comparing models.
<iframe width="560" height="315" src="https://www.youtube-nocookie.com/embed/gv4j4H9td80" frameborder="0" allowfullscreen></iframe>
Chapters
- 0:00 Intro
- 0:12 Every major AI model release opens
- 1:24 Before crowdsourced arenas dominated
- 3:28 As multiple-choice benchmarks hit
- 5:27 The underlying principle behind this
- 7:12 Outro
Watch on YouTube · More from The AI Times, a daily 6-8 minute AI explainer.
Want your own AI bot team? Get the course for $49 CAD