Longtermists have argued that humanity should significantly increase its efforts to prevent catastrophes like nuclear wars, pandemics, and AI disasters. But one prominent longtermist argument overshoots this conclusion: the argument also implies that humanity should reduce the risk of existential catastrophe even at extreme cost to the present generation. This overshoot means that democratic governments cannot use the longtermist argument to guide their catastrophe policy. In this paper, we show that the case for preventing catastrophe does not depend on longtermism. Standard cost-benefit analysis implies that governments should spend much more on reducing catastrophic risk. We argue that a government catastrophe policy guided by cost-benefit analysis should be the goal of longtermists in the political sphere. This policy would be democratically acceptable, and it would reduce existential risk by almost as much as a strong longtermist policy. Original article: Narrated for the Effective Altruism Forum by TYPE III AUDIO . Share feedback on this narration .

The long-term future of intelligent life is currently unpredictable and undetermined. In the linked document, we argue that the invention of artificial general intelligence (AGI) could change this by making extreme types of lock-in technologically feasible. In particular, we argue that AGI would make it technologically feasible to (i) perfectly preserve nuanced specifications of a wide variety of values or goals far into the future, and (ii) develop AGI-based institutions that would (with high probability) competently pursue any such values for at least millions, and plausibly trillions, of years. The rest of this post contains the summary (6 pages), with links to relevant sections of the main document (40 pages) for readers who want more details. Original article: Narrated for the Effective Altruism Forum by TYPE III AUDIO . Share feedback on this narration .

By Nick Bostrom & Carl Shulman. Draft version 1.10. AIs with moral status and political rights? We'll need a modus vivendi , and it’s becoming urgent to figure out the parameters for that. This paper makes a load of specific claims that begin to stake out a position. Read the full paper: More episodes at: --- Outline: (00:00) Introduction (00:36) Disclaimer (01:07) Consciousness and metaphysics (06:48) Respecting AI interests (21:41) Security and stability (32:04) AI-empowered social organization (38:07) Satisfying multiple values (42:23) Mental malleability, persuasion, and lock-in (47:20) Epistemology (53:36) Status of existing AI systems (59:52) Recommendations regarding current practises and AI systems (01:07:08) Impact paths and modes of advocacy (01:11:11) Closing credits ---