I’ve been trying to review and summarize Eliezer Yudkowksy’s recent dialogues on AI safety. Previously in sequence:  Yudkowsky Contra Ngo On Agents . Now we’re up to Yudkowsky contra Cotra on biological anchors, but before we get there we need to figure out what Cotra’s talking about and what’s going on. The  Open Philanthropy Project  (“Open Phil”) is a big effective altruist foundation interested in funding AI safety. It’s got $20 billion, probably the majority of money in the field, so its decisions matter a lot and it’s very invested in getting things right. In 2020, it asked senior researcher Ajeya Cotra to produce  a report on when human-level AI would arrive.  It says the resulting document is “informal” - but it’s 169 pages long and likely to affect millions of dollars in funding, which some might describe as making it  kind  of formal. The report finds a 10% chance of “transformative AI” by 2031, a 50% chance by 2052, and an almost 80% chance by 2100. Eliezer rejects their methodology and expects AI earlier (he doesn’t offer many numbers, but  here  he gives Bryan Caplan 50-50 odds on 2030, albeit  not totally seriously ). He made the case in his own very long essay,  Biology-Inspired AGI Timelines: The Trick That Never Works , sparking a bunch of arguments and counterarguments and even more long essays. Source: Crossposted from the Astral Codex Ten podcast.

Crossposted from the AI Alignment Forum . May contain more technical jargon than usual. I worked on my draft report on biological anchors for forecasting AI timelines mainly between ~May 2019 (three months after the release of GPT-2) and ~Jul 2020 (a month after the release of GPT-3), and posted it on LessWrong in Sep 2020 after an internal review process. At the time, my bottom line estimates from the bio anchors modeling exercise were : [1] Roughly ~15% probability of transformative AI by 2036 [2] (16 years from posting the report; 14 years from now). A median of ~2050 for transformative AI (30 years from posting, 28 years from now). These were roughly close to my all-things-considered probabilities at the time, as other salient analytical frames on timelines didn’t do much to push back on this view. (Though my subjective probabilities bounced around quite a lot around these values and if you’d asked me on different days and with different framings I’d have given meaningfully different numbers.) It’s been about two years since the bulk of the work on that report was completed, during which I’ve mainly been thinking about AI. In that time it feels like very short timelines have become a lot more common and salient on LessWrong and in at least some parts of the ML community. My personal timelines have also gotten considerably shorter over this period. I now expect something roughly like this:

- 1988 -Hans Moravec:  Behold my book Mind Children.  Within, I project that, in 2010 or thereabouts, we shall achieve strong AI.  I am not calling…

In this note I’ll summarize the bio-anchors report, describe my initial reactions to it, and take a closer look at two disagreements that I have with background assumptions used by (readers of) the report. This report attempts to forecast the year when the amount of compute required to train a transformative AI (TAI) model will first become available, as the year when a forecast for the amount of compute required to train TAI in a given year will intersect a forecast for the amount of compute that will be available for a training run of a single project in a given year. Original article: Narrated for the Effective Altruism Forum by TYPE III AUDIO . Share feedback on this narration .

on bio anchors“Bio Anchors” is a long document by Open Philanthropy researcher Ajeya Cotra. Many people have written more accessible summaries of this document. My favorite of these is Scott’s, and he...

tl;dr: The report underestimates the amount of compute used by evolution because it only looks at what it would take to simulate neurons, rather than…

Ajeya Cotra’s draft report on AI timelines is the most useful, comprehensive report about AI timelines I’ve seen so far. I think the report is a big…

Or: Big Timelines Crux OperationalizedWhat fun things could one build with +12 orders of magnitude of compute? By ‘fun’ I mean ‘powerful.’ This hypo…