Joel Becker | Reconciling Impressive AI Benchmark Performance

AI agents can now autonomously complete coding tasks that take humans with relevant expertise multiple hours with 50% reliability — and this capability is doubling every 4-7 months. Yet when METR ran an RCT around ~March 2025 (when frontier AIs had around a 1 hour time horizon), experienced open-source developers took 19% longer to complete tasks when they were allowed access to frontier AI tools relative to when AI tools were disallowed.

This talk will introduce both papers, attempt to reconcile apparently contrasting conclusions, and discuss research frontiers and directions coming out of the attempted reconciliation.

Link to time horizon paper

Link to developer productivity paper

Details:

Time: 12:00 pm - 1:00 pm PT

Location: Gates Computer Science Building, Room 119, 353 Jane Stanford Way, CA 94503

You can find additional event details, speaker bio, and information about upcoming seminars on our website.

Event Venue & Nearby Stays

Gates Computer Science, Room 119, 353 Serra Mall, Stanford, United States

Concerts, fests, parties, meetups - all the happenings, one place.

Get App