The nonprofit AI-progress assessment organization, METR (Model Evaluation and Threat Research) released a graph recently that has become “an industrywide obsession,” according to the New York Times. But are there better ways to track AI progress? Here, the Derby Mill panel weighs in.

METR uses a task-completion time horizon to assess the progress of AI frontier systems. The “time horizon” is the duration that it would take a human expert to complete a set of diverse software development tasks.

METR’s 50% time horizon, then, is the “duration at which an agent is predicted to succeed” at a given software development task half the time.

According to METR, frontier models over the last year have increased their task-completion horizon by about an order of magnitude, from roughly 1.5 hours to about 15 hours (see graph).

In this episode, Derby Mill host Ajay Agrawal begins by challenging Rich Sutton, Suzanne Gildert and Niamh Gavin: Should we expect another order-of-magnitude increase over the next 12 months?

The discussion quickly turns into a debate over the authenticity of METR’s measurement. Does the METR graph, and its remarkable improvement, reflect an improvement in AI model intelligence?

“I think this stuff is totally bogus,” says reinforcement learning pioneer Rich Sutton. “You have to realize how easy it is to cook these numbers.”

Rich’s criticism prompts Ajay to challenge the panel: Is there a way to measure AI’s progress that is better than METR’s method? Niamh discusses her role in a fascinating initiative to assess AI models by how well they win elections in simulated environments. And what merit is there in Suzanne’s suggestion, to assess progress based on aggregate numbers of job postings for software developers?

FEATURING

Ajay Agrawal, co-founder and partner, Intrepid Growth Partners

Niamh Gavin, senior advisor, Intrepid Growth Partners, Applied AI scientist and CEO, Emergent Platforms

Suzanne Gildert, CEO, Nirvanic Consciousness Technologies, quantum physicist, co-founder of Sanctuary AI and Kindred

Richard Sutton, senior advisor, Intrepid Growth Partners, 2024 Turing Award recipient, pioneer of reinforcement learning and professor, University of Alberta

LINKS

METR’s task-completion time horizon model explained, including the various versions of the time horizon graph at 50% and 80% success rates. Scroll down for a great FAQ explaining the effort’s various points.

METR CEO Beth Barnes in conversation on 80,000 hours.

The NY Times article that observed how METR’s graph has become “an industrywide obsession.”

Derby Mill is created by the team at Intrepid Growth Partners and produced by Ghost Bureau.

DISCUSSION POINTS

00:00 Cold open

00:27 Task Horizon Graph Explained

06:01 Low Context vs. High Context Tasks

10:10 Critiquing Model Evaluation Metrics

15:53 Testing AI Capabilities

20:08 Alternative Measures

27:07 Do We Need Quantitative Benchmarks?

DISCLAIMER

The content of this podcast is for informational and educational purposes only and should not be construed as marketing, solicitation, or an offer to buy or sell any securities or investments. The opinions expressed in this video are those of the participants and do not necessarily reflect the views of Intrepid Growth Partners or its affiliates. Any discussion of specific companies, technologies, or industries is for illustrative purposes and does not constitute investment advice. Viewers are encouraged to consult with their own financial, legal, and tax advisors before making any investment decisions.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit insights.intrepidgp.com